Technical lineage admin options

Note To use AI Governance, the latest Collibra user interface must be enabled for your Collibra environment.

Until recently, the options described in this topic were available only via the CLI lineage harvester. For reference, go to Lineage harvesting app command options and arguments.

In this topic

• Technical lineage admin options
• Admin options and descriptions
• Configure and run an admin job

Admin options and descriptions

Option	Description
List sources	Use this option to list all of the data sources that will be used to create a technical lineage. The results include the following details for each data source: Source ID <ID of data source> (from edge: false|true) (useSystemName: false|true). Source ID <ID of data source> The source ID of your data source. from edge: false|true Indicates that the data source is ingested via Edge. useSystemName: false|true Indicates whether Collibra Data Lineage uses the system or server name of the data source to match the System asset in Data Catalog. If the value is true, the system or server name of the data source is used. If the value is false, the system or server name of the data source is not used. The value of useSystemName is taken from the Collibra system name field for the data source on Edge. Example Source ID 1redshift (from edge: true) (useSystemName: false) indicates that the data source with the 1redshift source ID was ingested, and that the system name of the data source is not used to match the System asset in Data Catalog. Note No data sources are synchronized when you run this option.
Ignore sources	Use this option to exclude specified data sources from those that will be used to create the technical lineage. The next time you synchronize a technical lineage capability via Edge, the specified data source is ignored. Note No data sources are synchronized when you run this option. Important For each source that you want to ignore, ensure that the Active checkbox in the respective technical lineageEdge capability is cleared. For more information, go to Delete the of a data source on Edge.
Analyze files	Use this option to analyze specified batches (ZIP files) of metadata on the Collibra Data Lineage service instance. Tip When using the Load option in the Processing Level setting in your Edge capacity, the job result includes a link to download a ZIP file with the metadata. The link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you've inspected the metadata, use the "Analyze files" option to send the ZIP file for analysis. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID. No data sources are synchronized when you run this option.
Sync	Use this option to trigger the synchronization of the metadata, for all data sources. When you use this option, a new Main batch of metadata is created from the latest batches on the Collibra Data Lineage service instance, for all data sources, and synchronized with the corresponding assets in Data Catalog. The Sources tab page shows the transformation details or source code that was analyzed and the results of the analysis. Important  For information on the importance of the Processing Level setting and the Analyze option in the technical lineage Edge capabilities for your respective data sources: Refer to the step "Add a technical lineage capability to your Edge site" for any data source, in the topic Create a technical lineage via Edge. Read Tips for successful lineage synchronization.

Configure and run an admin job

Prerequisites

• You have created an Edge connection and added an Edge capability for each of the data sources you want to include in the technical lineage. For complete information, go to Create a technical lineage via Edge.

Steps

1. Create a Technical Lineage Admin connection.

   For CollibraData Lineage to connect to and retrieve metadata from the Collibra Data Lineage service instance, you have to create an Technical Lineage Admin connection.

   Prerequisites

   • You have a global role that has the System administration global permission.
   • You have a global role that has the Manage Edge sites global permission.
   • You have a global role that has the Manage connections and capabilities global permission.

   Steps

   1. Open an Edge site.
      1. On the main toolbar, click → Settings.
         The Collibra settings page opens.
      2. In the tab pane, click Edge.
         The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the Edge site overview, click the name of an Edge site.
         The Edge site page appears.
   2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
   3. Select the Technical Lineage Admin connection.
   4. Enter the connection information.
      

   Field	Description	Required
   Name	A name for the Edge connection.	Yes
   Description	A description of the connection.	No
   Authentication Type	The authentication method you use to connect to Collibra Data Lineage: Basic Authentication If you choose this method, ignore the rest of the fields. OAuth Important OAuth authentication will be made available in the coming weeks.	Yes
   Client ID	Your client ID for OAuth authentication. Important OAuth authentication will be made available in the coming weeks.
   Client Secret	Your client secret for OAuth authentication. Important OAuth authentication will be made available in the coming weeks.	.

   5. Click Create.
2. Add a Technical Lineage Admin capability to your Edge site.

   Complete the following steps to add the technical lineage capability to your Edge site.

   Prerequisites

   • You have a global role that has the System administration global permission.
   • You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

   Steps

   1. Open an Edge site.
      1. On the main toolbar, click → Settings.
         The Collibra settings page opens.
      2. In the tab pane, click Edge.
         The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the table, click the name of the Edge site whose status is Healthy.
         The Edge site page opens.
   2. Select the relevant capability template: Technical Lineage Admin.
   3. Enter the required information.
      

      Field	Description	Required?
      Name	The name of the Edge capability.	Yes
      Description	The description of the Edge capability.	Yes
      Admin connection	The name of the Edge connection you created in the previous step.	Yes
      Property	This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property. You can use this field to set the HTTP timeout duration by adding the httpTimeout property:  httpTimeout Name Description Type Encryption Example value httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. Text Not encrypted (plain text) 15 Type Value Type Name Example value Text Plaintext httpTimeout Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15. 15 Warning If you are a Collibra Cloud for Government customer, this field is required to connect to a Collibra Data Lineage service instance: Properties for Collibra Cloud for Government customers Name Description Type Encryption Example value techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Text Not encrypted (plain text) https://techlin-gcp-dev.collibra.com techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. Text To be encrypted by Edge management server Value will be stored on Edge Type Value Type Name Value Text Plaintext techlinHost This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com. Example: https://techlin-gcp-dev.collibra.com Text Secret techlinKey This is the unique API key to connect to a Collibra Data Lineage service instance. Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Customer Success Manager. <your-techlin-key> See an example	Yes for US government customers.
      Debug	An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (beta). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files. Select one of the following values: True Enables logging of the JDBC job. False Disables logging of the JDBC job. This is the default value.	No
      Log level	An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.	No

   4. Click Create.
      The capability is added to the Edge site.
      The fields become read-only.
3. Configure one of the admin options and run the job.

   In this step, you configure one of the admin options and then manually trigger the job.

   You can only choose one option at a time. After you click Synchronize to run the job, you can configure another option. You don't have to create a new Edge connection or add another capability.

   Important 

   To use the List sources option or the Ignore sources option, you must first:

   1. Schedule a data refresh in the Integration configuration data refresh schedule setting in Collibra Console.
   2. Restart Collibra.

   

   If you don't, an error is shown on the Integration Configuration tab.

   Prerequisites

   • A global role that has the following global permission:
     • Catalog, for example Catalog Author
     • View Edge connections and capabilities
   • A resource role with Configure external system resource permission, for example Owner.

   Steps

   1. On the main toolbar, click → Catalog.
      The Data Catalog Home opens.
   2. In the tab bar, click Integrations.
      The Integrations page opens.
   3. Click the Integration Configuration tab.
      
   4. Find the connection that you used when you added the technical lineage capability, and click the link in the Data sources/CapabilitiesCapabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
   5. On the Synchronize Configuration section, click Edit Configuration.
   6. In the Admin command drop-down list, select the option you want to run.
      
   7. Enter the required information.
      

      If you selected...	Then...
      List sources	Important  To use the List sources option, you must first: Schedule a data refresh in the Integration configuration data refresh schedule setting in Collibra Console. Restart Collibra. If you don't, an error is shown on the Integration Configuration tab. After the data is refreshed, the green Refresh button that you need click is shown on the Integration Configuration tab. Click the green Refresh button, as shown in the following image. The results are shown in the Sources field. Note  You don't need to click Save; it serves no purpose in this context.
      Ignore sources	Important  To use the Ignore sources option, you must first: Schedule a data refresh in the Integration configuration data refresh schedule setting in Collibra Console. Restart Collibra. If you don't, an error is shown on the Integration Configuration tab. Important For each source that you want to ignore, ensure that the Active checkbox in the respective technical lineageEdge capability is cleared. For more information, go to Delete the of a data source on Edge. In the Sources drop-down list, select the source or sources you want excluded from the technical lineage. Click Save. In the Synchronize Configuration section, click Synchronize. To view confirmation of the ignored source or sources: Open the Activities list. In the row containing the job, click Result. The Synchronization Results dialog box appears.
      Analyze files	In the File ID field, specify the file ID of the ZIP file on the Collibra Data Lineage service instance that you want analyzed. Enter only one file ID per field. To specify more than one file, click Add File ID. Tip When using the Load option in the Processing Level setting in your Edge capacity, the job result includes a link to download a ZIP file with the metadata. The link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. In this example, the file ID that you have to specify is: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. Click Save. In the Synchronize Configuration section, click Synchronize. To view confirmation of the analyzed files: Open the Activities list. In the row containing the job, click Result. The Synchronization Results dialog box appears.
      Sync	Click Save. In the Synchronize Configuration section, click Synchronize. A single synchronization job is triggered for all configured data sources. To view the ingestion report: Open the Activities list. In the row containing the synchronization job, click Result. The synchronization report summary appears.