Custom technical lineage JSON file

The version of the JSON architecture. Specify the value of 1.0, which is the only supported version.

This section contains tree definitions of data objects between which lineages can be defined. The data objects are systems, databases, schemas, tables, views, columns, dashboards and reports.

Each node of a tree contains the name, type and optionally children or leaves properties which form a hierarchy of data objects. You must define a node only once in this section. With the nested tree format, you can reuse the properties of one node for multiple children. For example, you can define a database once and use the children array to define multiple tables in the database.

Tip Usually, the structure you map is the following: system > database > schema > table > column. The system is optional, unless the useCollibraSystemName property is set to true in the lineage harvester configuration file. The Collibra Data Lineage can stitch these data objects to assets in Data Catalog. However, you can also map custom objects, for example dashboards and reports. Custom objects cannot be stitched to assets in Data Catalog.

This section contains the path from a source to a target and defines the transformation code or transformation references to be processed by the Collibra Data Lineage service.

The name of your data object. Specify this property with the system name, database name, schema name, table name, view name or column name.

The following rules apply when you specify this property:

• The names are case sensitive.
• The names of children and leaves can be identical if the children and leaves with the same names are in different parent nodes.

Note If you do not want to use the system or server name of your data source to match the System asset in Data Catalog, ensure that you do not add the system data object. The the Collibra system name setting for custom technical lineage is ignored. If you add the system data object in the JSON file, Collibra Data Lineage always uses the full path including the system (system > database > schema > table > column) for stitching, regardless of whether Collibra system name is set to True or False.

The type of your data object. You can specify one of the following options: system, database, schema, table, view, column, dashboard or report.

Note If you do not want to use the system or server name of your data source to match the System asset in Data Catalog, ensure that you do not add the system data object. The the Collibra system name setting for custom technical lineage is ignored. If you add the system data object in the JSON file, Collibra Data Lineage always uses the full path including the system (system > database > schema > table > column) for stitching, regardless of whether Collibra system name is set to True or False.

The sub-objects that have a hierarchical relation to the defined data object.

Each child can contain children properties, except for the penultimate child. The penultimate children property must contain the leaves property. The leaves property cannot contain a children property.

For example, you can use the children property to define a table and use the leaves properties to define columns that have a relation to the table node.

Each child and leave have the name and type properties and the optional catalog_fullname, catalog_domain_id, catalog_asset_type_name and catalog_asset_type_uuid properties.

The hierarchical path to the source data object. This data object is defined as a leaf in the tree section.

This property represents where the data comes from for a transformation.

The hierarchical path to the target data object. This data object is defined as a leaf in the tree section.

This property represents where the data flows to.

An ordered array of data object names. This array is required to define the sub-objects of the src_path and trg_path properties.

Specify the array with the data object names that start from the top of the tree section and finish at a leaf node.

This example shows data objects that can be stitched: system > database > schema > table > column.

This example shows data objects that cannot be stitched: dashboard > report > column.

Note If you do not want to use the system or server name of your data source to match the System asset in Data Catalog, ensure that you do not add the system data object. The the Collibra system name setting for custom technical lineage is ignored. If you add the system data object in the JSON file, Collibra Data Lineage always uses the full path including the system (system > database > schema > table > column) for stitching, regardless of whether Collibra system name is set to True or False.

The mapping name. This property specifies a name for the transformation code.

The transformation code, which determines how the technical lineage is constructed.

The transformation code can be a descriptive string or a SQL statement that manipulates data.

This property contains the name of the mapping reference to the transformation code in source code files. This property also contains the position and length of the transformation code to be highlighted in the technical lineage graph.

The name of the source code file that contains the transformation code. The transformation code can be a SQL statement, code that manipulates data or a descriptive string.

The source code file must be in the Shared Storage connection folder.

The unique descriptor of a part of transformation code in a source code file that is in the Shared Storage connection folder.

A source code file can contain different parts of transformation code that represent different data flows. This property indicates the referenced data flow.

The value of this property is the same as the value of the mapping_refs property in the codebase_files section.

The positions indicate a string of the transformation code in a source code file to be highlighted in the bottom part of the Collibra technical lineage graph. The whole lines that include the transformation code are highlighted.

The string must be a subset of the string of the transformation code that is defined by the pos_start and pos_len properties of the mapping_refs property in the codebase_files section.

The start position of the string of the transformation code to be highlighted. The start position is in characters, not bytes.

The value must be equal to or greater than the value of the pos_start property of the mapping_refs property in the codebase_files section.

The file path to source code files that contain the transformation code. The transformation code can be a SQL statement or code that manipulates data.

The source code file must be in the Shared Storage connection folder.

The mapping of the transformation code and the position of the transformation code that is shown in the bottom part of the technical lineage graph.

This property defines a string of the transformation code in the source code file to be shown in the technical lineage graph. The string must include the string that is defined by the pos_start and pos_len properties of the mapping property in the lineage section.

The unique descriptor of a part of transformation code in a source code file that is in the Shared Storage connection folder.

A source code file can contain different parts of transformation code that represent different data flows. This property indicates the referenced data flow.

The value must match the value of the mapping property in the lineage section.

The start position of the string of the transformation code. The start position is in characters, not bytes.

Specify a value in the following range:

• Equal to or greater than 0.
• Less than or equal to the value of the pos_start property in the mapping property in the lineage section.