Migrate the technical lineage of a data source

You can use the lineage harvester and technical lineage via Edge together. You can migrate a data source from lineage harvester to technical lineage via Edge, and also from technical lineage via Edge to the lineage harvester.

Prerequisites and permissions

A global role that has the following global permission:
- The Catalog, for example Catalog Author
- View Edge connections and capabilities
A resource role with Configure external system resource permission, for example Owner.
The permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
- Schemas
- Tables
- Columns
The lineage harvester version 2023.04 newer.

Steps

Migrate to technical lineage via Edge
Migrate to the lineage harvester

Open the lineage harvester configuration file in the config folder of your lineage harvester.
For the data source that you want to move to Edge, remove the section of the data source from the lineage harvester configuration file and save the configuration file.
If needed, start the lineage harvester in the console and run the following command to ignore the data source.
- For Windows: .\bin\lineage-harvester.bat ignore-source <source_ID>, where <source_id> is the ID of the data source that you want to ignore.
- For other operating systems: ./bin/lineage-harvester ignore-source <source_ID>, where <source_id> is the ID of the data source that you want to ignore.
The data source is excluded from the list of data sources that are used to create the technical lineage.
Important This step is required only in the following cases:
- If you use a different source ID for the data source on Edge.
- If you are migrating SAP HANA data sources from the lineage harvester to Edge, regardless of the source IDs you use.
  When you created technical lineage for SAP HANA by using the lineage harvester, different sources IDs were required if you used the hana and hana-cviews, or hana and hana-cviews-v2 dialects. However, in the Technical Lineage for SAP HANA capability, you can use one source ID for both SQL based and calculated views input. Technical lineage via Edge adds suffixes to the source ID automatically and internally. When you synchronize the Technical Lineage for SAP HANA capability, an error occurs if the source IDs from the lineage harvester exist for the same data source.
On Edge, add the technical lineage capability for the data source with the same configurations, for example, the same source ID.
Synchronize the technical lineage.
When the synchronization completes, the technical lineage is created for the data source.

Open an Edge site.
1. On the main toolbar, click , and then click Settings.
  The Collibra settings page opens.
2. In the tab pane, click Edge.
  The Sites tab opens and shows a table with an overview of the Edge sites.
3. In the Edge site overview, click the name of the Edge site where you created the technical lineage capability for the data source.
  The Edge site page appears.
In the Capabilities section, locate and click the technical lineage capability for the data source.
The technical lineage capability page opens.
Clear the Active check box.
Click Save.
The capability is updated.
Synchronize the technical lineage. If you added a synchronization schedule for the technical lineage capability, ensure that you delete the schedule.
When the synchronization completes, the technical lineage of the data source is deleted.
Open the lineage harvester configuration file in the config folder of your lineage harvester.
Specify the properties in the lineage harvester configuration file for the data source with the same configurations of the capability, for example, the same source ID, and save the configuration file.
Run the lineage harvester.
When the lineage harvester finishes processing, the technical lineage is created for the data source.

For the overall steps to create technical lineage, go to Create a technical lineage via the lineage harvester or Create a technical lineage via Edge.

What's next?

View the technical lineage graph.

You can check the progress of the technical lineage creation in Activities in your Collibra Data Intelligence Platform environment. The Results field indicates how many relations were imported into Data Catalog. Go to the status page to see the log files of the SQL analysis.

If the lineage harvester log shows an error message or the harvesting process fails, you can use the technical lineage common errors and issues in Collibra Support Portal to fix the error.

For technical lineage via Edge, if the synchronization fails or completes with an error message, you can use the technical lineage via Edge common error messages in Support Portal to fix the error.