Migrate the technical lineage of a data source

You can use the lineage harvester and technical lineage via Edge together. You can migrate a data source from lineage harvester to technical lineage via Edge, and also from technical lineage via Edge to the lineage harvester.

Prerequisites and permissions

  • A global role that has the following global permission:
    • The Catalog, for example Catalog Author
    • View Edge connections and capabilities
  • A resource role with Configure external system resource permission, for example Owner.
  • The permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
    • Schemas
    • Tables
    • Columns
  • The lineage harvester pre-release version 2023.04 or later.
    Note Use the lineage harvester pre-release versions only in non-production environments.

Steps

  • Migrate to technical lineage via Edge
  • Migrate to the lineage harvester

  1. Open the lineage harvester configuration file in the config folder of your lineage harvester.

  2. For the data source that you want to move to Edge, remove the section of the data source from the lineage harvester configuration file and save the configuration file.
  3. Start the lineage harvester in the console and run the following command to ignore the data source. This step is optional if you use the same source ID for the data source on Edge.
    • For Windows: .\bin\lineage-harvester.bat ignore-source <source_ID>.
    • For other operating systems: ./bin/lineage-harvester ignore-source <source_ID>.
    Note <source_id> is the ID of the data source that you want to ignore. You can specify only one source ID, and the source ID must not contain any spaces. If your source ID includes spaces, you can use the lineage harvester pre-release version 2023.04-0-4 or newer as a workaround. With this version, you can enclose the source ID with spaces in double or single quotation marks, for example ignore-source "Source A".
    The data source is excluded from the list of data sources that are used to create the technical lineage.
  4. On Edge, add the technical lineage capability for the data source with the same configurations, for example, the same source ID.
  5. Synchronize the technical lineage.
    When the synchronization completes, the technical lineage is created for the data source.

  1. Open an Edge site.

    1. On the main menu, click , and then click Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Edge sites overview appears.
    3. In the Edge site overview, click the name of the Edge site where you created the technical lineage capability for the data source.
      The Edge site page appears.

  2. In the Capabilities section, locate and click the technical lineage capability for the data source.
    The technical lineage capability page opens.
  3. Clear the Active check box.
  4. Click Save.
    The capability is updated.
  5. Synchronize the technical lineage. If you added a synchronization schedule for the technical lineage capability, ensure that you delete the schedule.
    When the synchronization completes, the technical lineage of the data source is deleted.

  6. Open the lineage harvester configuration file in the config folder of your lineage harvester.

  7. Specify the properties in the lineage harvester configuration file for the data source with the same configurations of the capability, for example, the same source ID, and save the configuration file.

  8. Run the lineage harvester.
    When the lineage harvester finishes processing, the technical lineage is created for the data source.

For the overall steps to create technical lineage, go to Creating a Technical lineage via the lineage harvester or Creating technical lineage via Edge (beta).

What's next?

View the technical lineage graph.

You can check the progress of the technical lineage creation in Activities in your Collibra Data Intelligence Cloud environment. The Results field indicates how many relations were imported into Data Catalog. Go to the status page to see the log files of the SQL analysis.

If the lineage harvester log shows an error message or the harvesting process fails, you can use the technical lineage troubleshooting guide or Collibra Support Portal to fix the error.

For technical lineage via Edge, if the synchronization fails or completes with an error message, you can use the technical lineage via Edge troubleshooting guide or Collibra Support Portal to fix the error.