Technical lineage admin options

Important The features described in this topic will be available as of January 26, 2025.

This topic:

• Describes the following data lineage admin options:
  • List sources
  • Ignore sources
  • Analyze files
  • Sync
• Provides guidance on how to create the required Technical Lineage Admin Edge connection, add the Technical Lineage Admin Edge capability, and start the job.

Until recently, these options were available only via the CLI lineage harvester. For reference, go to Lineage harvesting app command options and arguments.

In this topic

• Technical lineage admin options
• Admin option descriptions
• Prerequisites
• Steps

Admin option descriptions

Option	Description
List sources	Use this option to list all of the data sources that will be used to create a technical lineage. The results include the following details for each data source: Source ID <ID of data source> (from edge: false|true) (useSystemName: false|true). Source ID <ID of data source> The source ID of your data source. from edge: false|true Indicates that the data source is ingested via Edge. useSystemName: false|true Indicates whether Collibra Data Lineage uses the system or server name of the data source to match the System asset in Data Catalog. If the value is true, the system or server name of the data source is used. If the value is false, the system or server name of the data source is not used. The value of useSystemName is taken from the Collibra system name field for the data source on Edge. Example Source ID 1redshift (from edge: true) (useSystemName: false) indicates that the data source with the 1redshift source ID was ingested, and that the system name of the data source is not used to match the System asset in Data Catalog. Note No data sources are synchronized when you run this option.
Ignore sources	Use this option to exclude specified data sources from those that will be used to create the technical lineage. The next time you synchronize a technical lineage capability via Edge, the specified data source is ignored. Note No data sources are synchronized when you run this option. For information on deleting a technical lineage, go to Delete the of a data source on Edge.
Analyze files	Use this option to analyze specified batches (ZIP files) of metadata on the Collibra Data Lineage service instance. Note No data sources are synchronized when you run this option.
Sync	Use this option to trigger the synchronization of the metadata, for all data sources. More specifically, a new Main batch (of metadata) is created from the latest batches on the Collibra Data Lineage service instance, for all data sources, and synchronized with the corresponding assets in Data Catalog. The Sources tab page shows the transformation details or source code that was analyzed and the results of the analysis. Important  For information on the importance of the Processing Level setting and the Analyze option in the technical lineage Edge capabilities for your respective data sources: Refer to the step "Add a technical lineage capability to your Edge site" for any data source, in the topic Create a technical lineage via Edge. Read Tips for successful lineage synchronization.

Prerequisites

• You have created an Edge connection and added an Edge capability for each of the data sources you want to include in the technical lineage. For complete information, go to Create a technical lineage via Edge.

Steps

1. Create a Technical Lineage Admin connection.

   For CollibraData Lineage to connect to and retrieve metadata from the Collibra Data Lineage service instance, you have to create an Technical Lineage Admin connection.

   Prerequisites

   • You have a global role that has the System administration global permission.
   • You have a global role that has the Manage Edge sites global permission.
   • You have a global role that has the Manage connections and capabilities global permission.

   Steps

   1. Open an Edge site.
      1. On the main toolbar, click , and then click Settings.
         The Collibra settings page opens.
      2. In the tab pane, click Edge.
         The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the Edge site overview, click the name of an Edge site.
         The Edge site page appears.
   2. Click Create Connection.
      The Connection settings page opens.
   3. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
   4. Select the Technical Lineage Admin connection.
   5. Enter the connection information.

      Field	Description	Required
      Connection settings	This section contains the general settings of your connection.
      Name	A name for the Edge connection.	Yes
      Description	A description of the connection.	No

      Field	Description	Required
      Name	A name for the Edge connection.	Yes
      Description	A description of the connection.	No

   6. Click Create.
2. Add a Technical Lineage Admin capability to your Edge site.

   Complete the following steps to add the technical lineage capability to your Edge site.

   Prerequisites

   • You have a global role that has the System administration global permission.
   • You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

   Steps

   1. Open an Edge site.
      1. On the main toolbar, click , and then click Settings.
         The Collibra settings page opens.
      2. In the tab pane, click Edge.
         The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the table, click the name of the Edge site whose status is Healthy.
         The Edge site page opens.
   2. Select the relevant capability template: Technical Lineage Admin.
   3. Enter the required information.
      

      Field	Description	Required?
      Capability	This section contains general information about the capability.
      Name	The name of the Edge capability.	Yes
      Description	The description of the Edge capability.	Yes
      Admin connection	The name of the Edge connection you created in the previous step.	Yes
      Property	This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property. You can use this field to set the HTTP timeout duration by adding the httpTimeout property:  httpTimeout Name Description Type Encryption Example value httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. Text Not encrypted (plain text) 15 Type Value Type Name Example value Text Plaintext httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. 15 Warning If you are a Collibra Cloud for Government customer, this field is required to connect to a Collibra Data Lineage service instance: Properties for Collibra Cloud for Government customers Name Description Type Encryption Example value techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Text Not encrypted (plain text) https://techlin-gcp-dev.collibra.com techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. Text To be encrypted by Edge management server Value will be stored on Edge Type Value Type Name Value Text Plaintext techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Example: https://techlin-gcp-dev.collibra.com Text Secret techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. <your-techlin-key> See an example	Yes for US government customers.
      Logging	This section contains general information about logging.
      Debug	An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (beta). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files. Select one of the following values: True Enables logging of the JDBC job. False Disables logging of the JDBC job. This is the default value.	No

      Field	Description	Required?
      Name	The name of the Edge capability.	Yes
      Description	The description of the Edge capability.	Yes
      Admin connection	The name of the Edge connection you created in the previous step.	Yes
      Property	This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property. You can use this field to set the HTTP timeout duration by adding the httpTimeout property:  httpTimeout Name Description Type Encryption Example value httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. Text Not encrypted (plain text) 15 Type Value Type Name Example value Text Plaintext httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. 15 Warning If you are a Collibra Cloud for Government customer, this field is required to connect to a Collibra Data Lineage service instance: Properties for Collibra Cloud for Government customers Name Description Type Encryption Example value techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Text Not encrypted (plain text) https://techlin-gcp-dev.collibra.com techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. Text To be encrypted by Edge management server Value will be stored on Edge Type Value Type Name Value Text Plaintext techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Example: https://techlin-gcp-dev.collibra.com Text Secret techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. <your-techlin-key> See an example	Yes for US government customers.
      Debug	An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (beta). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files. Select one of the following values: True Enables logging of the JDBC job. False Disables logging of the JDBC job. This is the default value.	No
      Log level	An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.	No

   4. Click Save.
      The capability is added to the Edge site.
      The fields become read-only.
   5. Click Create.
      The capability is added to the Edge site.
      The fields become read-only.
3. Configure one of the admin options and run the job.

   In this step, you configure one of the admin options and then manually trigger the job.

   You can only choose one option at a time. After you click Synchronize to run the job, you can configure another option. You don't have to create a new Edge connection or add another capability.

   Prerequisites

   • A global role that has the following global permission:
     • Catalog, for example Catalog Author
     • View Edge connections and capabilities
   • A resource role with Configure external system resource permission, for example Owner.

   Steps

   1. On the main toolbar, click , and then click Catalog.
      The Data Catalog Home opens.
   2. In the Register with Edge section of the Create dialog box, click Register a data source.
      The Register content page opens.
   3. In the tab bar, click Integrations.
      The Integrations page opens.
   4. Click the Integration Configuration tab.
      
   5. Find the connection that you used when you added the technical lineage capability, and click the link in the Data sources/CapabilitiesCapabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
   6. On the Configuration SectionSynchronize Configuration section, click Edit Configuration.
   7. In the Admin command drop-down list, select the option you want to run.
      
   8. Enter the required information.
      

      If you selected...	Then...
      List sources	Click the green Refresh button, as shown in the following image. Note  After you add the Technical Lineage Admin capability to your Edge site, as described in the step "Add a Technical Lineage Admin capability to your Edge site", the green Refresh button appears only after the next data refresh, the schedule of which is configured in the Integration configuration data refresh schedule setting in Console. The results are shown in the Sources field. Note  You don't need to click Save; it serves no purpose in this context.
      Ignore sources	In the Sources drop-down list, select the source or sources you want excluded from the technical lineage. Click Save. In the Configuration SectionSynchronize Configuration section, click Synchronize nowSynchronize. To view confirmation of the ignored source or sources: Open the Activities list. In the row containing the job, click Result. The Synchronization Results dialog box appears.
      Analyze files	In the File ID field, specify the file ID of the ZIP file on the Collibra Data Lineage service instance that you want analyzed. Enter only one file ID per field. To specify more than one file, click Add File ID. Click Save. In the Configuration SectionSynchronize Configuration section, click Synchronize nowSynchronize. To view confirmation of the analyzed files: Open the Activities list. In the row containing the job, click Result. The Synchronization Results dialog box appears.
      Sync	Click Save. In the Configuration SectionSynchronize Configuration section, click Synchronize nowSynchronize. A single synchronization job is triggered for all configured data sources. To view the ingestion report: Open the Activities list. In the row containing the synchronization job, click Result. The synchronization report summary appears.