Lineage harvesting app command options and arguments

Uploads all of the metadata from the data sources mentioned in your configuration file to the Collibra Data Lineage service, where the metadata is then processed and uploaded to Data Catalog.

After you enter this command, the lineage harvester starts synchronization processing and displays the total number of data sources that are being ingested.

Synchronization processing ends with an error in the following situations:

• The lineage harvester does not find any data sources,
• The useSystemName value is not the same for all data sources. The value of useSystemName is based on the following settings:
  • The useCollibraSystemName property in the lineage harvester configuration file for different data sources.
  • The Collibra system name setting on Edge.

Uploads only the metadata from a specified data source. For example, full-sync -s "myOracleDataSource". The specified data source must be mentioned in your configuration file.

This command allows you to process data from a newly added data source or to refresh a data source in the configuration file, without refreshing the other data sources. This reduces the time you need to upload your data sources, since you only upload specific ones without affecting the others. If you want to process multiple data sources, add -s "ID of another data source" per data source to the command.

You can concurrently synchronize as many sources as you like. On the Collibra Data Lineage service instance, the sources are processed sequentially.

Note You can use this argument multiple times to include multiple data sources.

Uploads a technical lineage without stitching the data objects in your technical lineage to the corresponding Column and Table assets in Data Catalog.

Note As a result, you won't see the technical lineage of a specific Table or Column asset, but you can still see and browse the full technical lineage.

Whereas full-sync ingests metadata onto the Collibra Data Lineage service, processes the metadata and syncs it with assets in Data Catalog, the sync command only performs this last part: it syncs the metadata—as it exists on the Collibra Data Lineage service—and your assets in Data Catalog.

After you enter this command, the lineage harvester starts synchronization processing and displays the total number of data sources that are being ingested.

Synchronization processing ends with an error in the following situations:

• The lineage harvester does not find any data sources,
• The useSystemName value is not the same for all data sources. The value of useSystemName is based on the following settings:
  • The useCollibraSystemName property in the lineage harvester configuration file for different data sources.
  • The Collibra system name setting on Edge.

Tip See the following example for advice on how to use the sync command to add a new data source without re-harvesting all data sources.

Example

Let's say you've run bin/lineage-harvester full-sync, to upload from all data sources, process the metadata and sync with Data Catalog. You then decide that you want to add a new data source, but not harvest all data sources again.

1. Reference the new data source in the lineage harvester configuration file. Let's say that the new data source has the ID "MyNewSource".
2. Run bin/lineage-harvester load-sources -s MyNewSource, to load the new data source and create the ZIP file.
3. Run bin/lineage-harvester analyze ${zip_file_from_step_2}, to analyze the new data source on the Collibra Data Lineage service.
4. Run bin/lineage-harvester sync, to sync all of the data sources referenced in your configuration file and Data Catalog.

Syncs only the metadata on the Collibra Data Lineage service, from a specified data source. For example, sync -s "myOracleDataSource". The specified data source must be mentioned in your configuration file.

This command allows you to sync data from one data source without refreshing the other data sources. You must have previously uploaded the metadata to the Collibra Data Lineage service.

Warning Only the sources you specify are synced. This means that any previously ingested metadata from non-specified sources, in Data Catalog, is deleted, along with its existing technical lineage. If this is not your intention, consider using full-sync -s. With full-sync -s, all sources are synced, regardless of which sources are specified by the -s command. Therefore, any previously ingested metadata from non-specified data sources remains, as do the respective technical lineages.

Note You can use this argument multiple times to include multiple data sources.

Analyzes a specified batch (ZIP file) of metadata on the Collibra Data Lineage service instance.

The Sources tab page shows the transformation details or source code that was analyzed and the results of the analysis.

Downloads all your data sources in a separate ZIP file, per data source, to the lineage harvester output folder.

Downloads only the data source with a specific ID. For example, load-sources -s "myOracleDataSource".

Note You can use this argument multiple times to include multiple data sources.

Lists all of the data sources that will be used to create a technical lineage. When you enter this command, up to 500 data sources are listed per page by default.

The list includes the following details for each data source: Source ID <ID of data source> (from edge: false|true) (useSystemName: false|true).

Source ID <ID of data source>

The source ID of your data source.

from edge: false|true

Indicates whether the data source is ingested by using technical lineage via Edge. If the value is true, the data source is ingested by using technical lineage via Edge. If the value is false, the data source is ingested by using the lineage harvester.

useSystemName: false|true

Indicates whether Collibra Data Lineage uses the system or server name of the data source to match the System asset in Data Catalog. If the value is true, the system or server name of the data source is used. If the value is false, the system or server name of the data source is not used.

The value of useSystemName is based on the following settings:

• The useCollibraSystemName property in the lineage harvester configuration file for the data source.
• The Collibra system name setting for the data source on Edge.

Specifies the page to be displayed. The value of <page number> must be greater than 0. This option is optional.

For example, if you enter list-sources -p 2, page 2 is displayed with a default page size of 500 data sources listed. If there are less than 500 data sources in total, an error message is issued.

Specifies the number of data sources to be listed on one page. The value of <number of data sources> must be in the range 0 - 500. This option is optional.

For example, if you enter list-sources -s 40, default page 1 is displayed with 40 data sources listed. If there are 80 data sources in total, you see the Displaying page 1 of 2 message and a list of 40 data sources.

If you enter list-sources -p 3 -s 20, page 3 is displayed with 20 data sources listed. If there are 80 data sources, you see the Displaying page 3 of 4 message and a list of 20 data sources.

Lists all data sources. The data sources are not formatted in pages. If you enter this option with the -p and -s options, this option overrides the -p and -s options.

For example, if you enter list-sources -p 3 -s 20 --all, all data sources are listed.

Ignores the specified data source from the list of data sources that will be used to create the technical lineage, where <source_id> is the ID of the data source that you want to ignore. When you create the technical lineage again by entering the sync command or synchronizing a technical lineage capability via Edge, the specified data source is ignored.

You can specify only one source ID at a time. If your source ID includes spaces, enclose the source ID in double or single quotation marks, for example ignore-source "Source A".

You can use this command to delete the technical lineage of a data source by using the lineage harvester. For details, go to Delete the technical lineage of a data source if you use the lineage harvester and Delete the technical lineage of a data source on Edge for technical lineage via Edge.

Provides passwords of your Collibra Data Intelligence Platform instance and the data sources in your configuration file to the lineage harvester without storing the passwords in the lineage harvester folder.

You can replace cat passwords.json by a string generated by your password manager.

Checks the connectivity to the Collibra Data Lineage service instance and to Data Catalog. The logs will also show the IP addresses of the Collibra Data Lineage service instances that you have to allow.

This command is mostly used for troubleshooting purposes.

Shows an overview of all supported command options and arguments that you can use in the lineage harvester.

Shows the version of the lineage harvester that you are using.