Delete the technical lineage of a data source

You can delete the technical lineage of a data source if you no longer want to see it in the technical lineage graph. You can do so via Edge or the CLI lineage harvester.

Via Edge

For more information about the Technical Lineage Admin Edge connection and capability mentioned in this procedure, go to Technical lineage admin options.

    1.  Open an Edge site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Collibra settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the Edge site overview, click the name of the Edge site to which you added the technical lineage capability for the data source.
        The Edge site page appears.
    2. In the Capabilities section, locate and click the technical lineage capability for the data source you want to delete.
    3. Clear the Active checkbox.
    4. Click Save.
      The capability is updated.

  2. For CollibraData Lineage to connect to and retrieve metadata from the Collibra Data Lineage service instance, you have to create an Technical Lineage Admin connection.

    Prerequisites

    Steps

    1. Open an Edge site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Collibra settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the Edge site overview, click the name of an Edge site.
        The Edge site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select the Technical Lineage Admin connection.
    4. Enter the connection information.
      5. Field Description Required
      Name

      A name for the Edge connection.

      		 Yes
      Description

      A description of the connection.

      		 No

      Authentication Type

      The authentication method you use to connect to Collibra Data Lineage:

      • Basic Authentication
        If you choose this method, ignore the rest of the fields.
      • OAuth

      If you choose this method, you must use the following fields to provide a client ID, client secret, and impersonated user.

      Important OAuth authentication is not yet available for Collibra Cloud for Government customers.

      		 Yes

      Client ID

      Your client ID for OAuth authentication.

      1. In Collibra Settings, click OAuth Applications.
      2. Click Register New Application.
        The Register New Application dialog box appears.
      3. Enter the following information:
        1. For the Application Type, select Platform.
        2. Provide a name for the application.
        3. In the Integration Type drop-down list, select Technical Lineage.
      4. Click Register.
      5. Copy and safely store the Client ID and Client Secret.

        Important This is the only time you are able to see the client secret.

      For complete information, go to OAuth Applications.

      Yes

      Client Secret

      Your client secret for OAuth authentication.

      1. In Collibra Settings, click OAuth Applications.
      2. Click Register New Application.
        The Register New Application dialog box appears.
      3. Enter the following information:
        1. For the Application Type, select Platform.
        2. Provide a name for the application.
        3. In the Integration Type drop-down list, select Technical Lineage.
      4. Click Register.
      5. Copy and safely store the Client ID and Client Secret.

        Important This is the only time you are able to see the client secret.

      For complete information, go to OAuth Applications.

      Yes

      .
    5. Click Create.

  3. Complete the following steps to add the technical lineage capability to your Edge site.

    Prerequisites

    Steps

    1. Open an Edge site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Collibra settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of the Edge sites.
      3. In the table, click the name of the Edge site whose status is Healthy.
        The Edge site page opens.
    2. Select the relevant capability template: Technical Lineage Admin.
    3. Enter the required information.
      FieldDescriptionRequired?

      Name

      The name of the Edge capability.

      Yes

      Description

      The description of the Edge capability.

      Yes

      Admin connection

      The name of the Edge connection you created in the previous step.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      NameDescriptionTypeEncryptionExample value
      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		TextNot encrypted (plain text)15
      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Cloud for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      NameDescriptionTypeEncryptionExample value
      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com.

      		TextNot encrypted (plain text)https://techlin-gcp-dev.collibra.com
      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		TextTo be encrypted by Edge management serverValue will be stored on Edge
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-gcp-eu.collibra.com.

      		Example: https://techlin-gcp-dev.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (beta). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

    4. Click Create.
      The capability is added to the Edge site.
      The fields become read-only.
  4. Important 

    To use the Ignore sources option:

    1. Your metadata must be refreshed. You can either wait for the next scheduled synchronization to run, or you can edit the Integration configuration data refresh schedule setting in Collibra Console so that the refresh is done sooner.
    2. If you edit the Integration configuration data refresh schedule setting, you must restart Collibra.

    If you don't refresh your metadata (and restart Collibra, if necessary), an error is shown on the Integration Configuration tab.

    1. On the main toolbar, click Products icon → Catalog.
      The Data Catalog Home opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Data sources/CapabilitiesCapabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronize Configuration section, click Edit Configuration.
    6. In the Admin command drop-down list, select Ignore sources.
    7. In the Sources drop-down list, select the source or sources you want excluded from the technical lineage.
    8. Click Save.
    9. In the Synchronize Configuration section, click Synchronize.
      When synchronization is complete, the technical lineage of the data source is deleted.
    1. Open the Activities list.
    2. In the row containing the job, click Result.
      The Synchronization Results dialog box appears.

Via the CLI lineage harvester

Note If you use the CLI lineage harvester, you need to ensure that least one data source is configured in your lineage harvester configuration file.

    1. Optional: To determine the data source that you want to exclude from the Technical lineage, enter the list-sources command:

      • For Windows: .\bin\lineage-harvester.bat list-sources
      • For other operating systems: ./bin/lineage-harvester list-sources
      All data sources that were used to create the technical lineage are listed. The list also includes the source ID of each data source. You can use the list to identify the data source to be excluded.
    2. In the lineage harvester folder, open your lineage harvester configuration file.
    3. Delete the section with connection properties of the data source.
    4. Save the configuration file.
    5. Start the lineage harvester in the console and run the following command to ignore the data source:
      • For Windows: .\bin\lineage-harvester.bat ignore-source <source_ID>, where <source_id> is the ID of the data source that you want to ignore.
      • For other operating systems: ./bin/lineage-harvester ignore-source <source_ID>, where <source_id> is the ID of the data source that you want to ignore.
      The data source is excluded from the list of data sources that are used to create the technical lineage.
    6. Synchronize the technical lineage by running any of the following commands:
      • The sync command:
        • For Windows: .\bin\lineage-harvester.bat sync
        • For other operating systems: ./bin/lineage-harvester sync
      • The full-sync command:
        • For Windows: .\bin\lineage-harvester.bat full-sync
        • For other operating systems: ./bin/lineage-harvester full-sync

      For more information, go to Lineage harvesting app command options and arguments.

    7. When prompted, enter the password to connect to your Collibra Platform and data sources in the configuration file.
      8. The lineage harvester uploads the metadata of the remaining data sources in the configuration file to the Collibra Data Lineage service.
      The Collibra Data Lineage service synchronizes the technical lineage and removes the deleted data source from the technical lineage graph.

  2. You can check the progress of the technical lineage creation in Activities in your Collibra Platform environment. The Results field indicates how many relations were imported into Data Catalog. Go to the status page to see the log files of the SQL analysis.

    If the lineage harvester log shows an error message or the harvesting process fails, you can use the technical lineage common errors and issues in Collibra Support Portal to fix the error.