Prepare the Data Catalog physical data layer for technical lineage

Important 

In Collibra 2024.02, we've launched a new user interface (UI) in beta for Collibra Data Intelligence Platform! You can learn more about this latest UI in the UI overview.

Use the following options to see the documentation in the latest UI or in the previous, classic UI:

Important This topic does not apply if you register a data source via Edge because in that case, Collibra automatically creates the system > database > schema > table > column hierarchy.

To stitch data objects in your data sources to their corresponding assets in Collibra Data Intelligence Platform, the full names of the data objects and assets must match exactly. The full names are constructed according to the full path of the data objects in your data source:

(system name) > database name > schema name > table name > column name

However, when you register a data source via Jobserver or by running the lineage harvester, only assets of the following asset types are created in Data Catalog:

  • Schema
  • Table
  • Column

Therefore, you have to create a Database asset and create a relation between it and the relevant Schema asset, to construct the full path hierarchy required for full name matching. If you set the useCollibraSystemName property to true in your lineage harvester configuration file, you also need to create a System asset and create a relation between it and the Database asset. We refer to this as preparing the Data Catalog physical data layer.

For more information, see Automatic stitching for technical lineage.

Prerequisites

  • You have a global role with the Catalog global permission, for example, Catalog Author.
  • You have a resource role with the following resource permissions on the Schema community if you use a Jobserver and on the Database community if you use Edge.
    • Asset > add
    • Attribute > add
    • Domain > add
    • Attachment > add

Additional prerequisites for JDBC data source types

If you are working with a JDBC data source type, you also need to meet the following prerequisites:

  • You have the permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
    • Schemas
    • Tables
    • Columns
  • You have set up the JDBC driver of your source data, for example MySQL.
  • You have registered a data source.
    Tip The full name of your Schema asset must match the exact name of the schema (including for case-sensitivity) in the data source that you register in the configuration file.
    If you use Jobservers in Collibra Console and there is no available Jobserver, the Register data source actions will be grayed out in the global create menu in Collibra.

Steps

  1. Create a System asset:
    Important This is only required if you set the useCollibraSystemName property to true in your lineage harvester configuration file.
    Tip The full name of the System asset must match (including for case-sensitivity) the exact name of the system of the data source that you register in the configuration file.
    1. Open the product for which you want to create an asset, for example, Business Glossary.
    2. On the main toolbar, click .
      The Create dialog box appears.
    3. On the Assets tab, click Database.
      The Create Asset dialog box appears.
    4. Enter the required information.
      FieldDescription
      Asset Type

      The asset type of the asset that you are creating.

      Domain

      The domain to which the asset will belong.

      Tip Ensure that the domain type of the selected domain is assigned to the selected asset type.

      Asset Name

      A name to identify the asset. Do not include the > symbol in the name. If an asset name contains the > symbol, Collibra Data Lineage fails to stitch the ingested data objects to the asset.

      Tip 

      You can simultaneously create multiple assets.
      To do so, after typing the name, press Enter, and then type the next name. Depending on the settings, asset names may need to be unique in their domain. If you enter a name that already exists, it appears in the strike-through style.

    5. Click Create.
      A message stating that one or more assets are created appears in the upper-right corner of the page.
  2. Create a Database asset:
    Tip The full name of your Database asset must match (including for case-sensitivity) the exact name of the database or project, in case of Google BigQuery, that you register in the configuration file. The names are case-sensitive.
    1. Open the product for which you want to create an asset, for example, Business Glossary.
    2. On the main toolbar, click .
      The Create dialog box appears.
    3. On the Assets tab, click Database.
      The Create Asset dialog box appears.
    4. Enter the required information.
      FieldDescription
      Asset Type

      The asset type of the asset that you are creating.

      Domain

      The domain to which the asset will belong.

      Tip Ensure that the domain type of the selected domain is assigned to the selected asset type.

      Asset Name

      A name to identify the asset. Do not include the > symbol in the name. If an asset name contains the > symbol, Collibra Data Lineage fails to stitch the ingested data objects to the asset.

      Tip 

      You can simultaneously create multiple assets.
      To do so, after typing the name, press Enter, and then type the next name. Depending on the settings, asset names may need to be unique in their domain. If you enter a name that already exists, it appears in the strike-through style.

    5. Click Create.
      A message stating that one or more assets are created appears in the upper-right corner of the page.
  3. Create a relation between the System asset and the Database asset using the "Technology Asset groups / is grouped by Technology Asset" relation type.
    Important This step is only relevant if you created a System asset, in step 1.

    1. Open the asset page of the System asset.
    2. In the tab pane, click Add characteristic.
      The Add a Characteristic dialog box appears.
    3. Click Relations.
    4. Search for and click groups Technology Asset.
      The Add groups Technology Asset dialog box appears.
    5. Enter the required information.
      OptionDescription
      Assets

      The name of the Database asset.

      Filter suggested assets by organization

      Option to filter the suggestions based on selected communities and domains.

      If this option is selected, the organization tree appears. You can then filter and select domains and communities.

      Start dateOptionally enter the date on which the relation between the assets becomes applicable.
      End dateOptionally enter the date on which the relation between the assets is no longer applicable. Leave this field empty to create a permanent relation.
    6. Click Save.

    1. Open the asset page of the System asset.
    2. Do one of the following:

        1. If a Groups Technology Asset relation already exists on the asset page:

          1. Click +.

            An asset picker opens.
          2. Search for and select the Database asset that you created in the previous step.
          3. Click outside of the asset picker to create the relation.

        2. If a Groups Technology Asset relation does not yet exist on the asset page:

          1. Turn off the Hide empty values option, if it is turned on.
          2. Scroll down to the Groups Technology Asset relation type, and click in the empty field.

            An asset picker opens.
          3. Search for and select the Database asset that you created in the previous step.
          4. Click outside of the asset picker to create the relation.

  4. Create a relation between the Database asset and the relevant Schema asset using the "Technology Asset has / belongs to Schema" relation type.

    1. In the tab pane, click Add characteristic.
      The Add a Characteristic dialog box appears.
    2. Click Relations.
    3. Search for and click has Schema.
      The Add has Schema dialog box appears.
    4. Enter the required information.
      OptionDescription
      Assets

      The name of the Schema asset.

      Filter suggested assets by organization

      Option to filter the suggestions based on selected communities and domains.

      If this option is selected, the organization tree appears. You can then filter and select domains and communities.

      Start dateOptionally enter the date on which the relation between the assets becomes applicable.
      End dateOptionally enter the date on which the relation between the assets is no longer applicable. Leave this field empty to create a permanent relation.
    5. Click Save.

    1. Open the asset page of the Database asset.
    2. Do one of the following:

        1. If a Has Schema relation already exists on the asset page:

          1. Click +.

            An asset picker opens.
          2. Search for and select the relevant Schema asset.
          3. Click outside of the asset picker to create the relation.

        2. If a Has Schema relation does not yet exist on the asset page:

          1. Turn off the Hide empty values option, if it is turned on.
          2. Scroll down to the Has Schema relation type, and click in the empty field.

            An asset picker opens.
          3. Search for and select the relevant Schema asset.
          4. Click outside of the asset picker to create the relation.

What's next?

If you haven't created a configuration file yet, you are now required to create it.

If you created the configuration file and prepared the physical data layer, you can run the lineage harvester to start the technical lineage process.

When the technical lineage process is finished and you have the required permissions, you can go to the asset page of a Table or Column asset from the data source that you added in the configuration file and visualize the technical lineage. At the same time, new relations of the type "Data Element targets / sources Data Element" between assets in Data Catalog are created.