Manage technical lineage ingestion

You can create a customized SQL file to manage which data objects, for example columns and tables, are ingested in the technical lineage. In the SQL file, you can exclude data objects or change queries that are used to extract data from the database. You specify:

• Which data objects you want to visualize in the technical lineage.
• Between which columns you want to create new relations of the type "Data Element targets / sources Data Element" in Data Catalog.

Note When creating technical lineage for JDBC data sources via a "JDBC connection", lineage is not generated for tables created with CREATE TABLE AS SELECT statements. With JDBC connection method, SQL statements that create tables are not harvested; therefore, lineage can't be generated. If, however, you connect via "folder connection" (lineage harvester) or "Shared Storage connection" (Edge), these SQL statements are analyzed and lineage can be generated.

Steps

1. Open the lineage harvester folder.
2. Go to the sql folder and open the folder of the data source type of which you want to exclude tables or schemas or change queries.
3. Create a copy of the file you want to edit.
4. Rename the copy to [original name]-custom.sql.
   Example You want to change the file columns.sql, so you name the copy of this file and rename it to columns-custom.sql.
5. Delete or edit the content of the new SQL file to include or exclude specific tables or schemas or change specific queries in the file.
6. Save the new SQL file.
   The lineage harvester uses the new file and ignores the old one.