Lineage harvesting app command options and arguments

After creating a configuration file, you can use the lineage harvester to perform specific actions with the data sources that are defined in your configuration file.

Tip If you run the lineage harvester in command line, you will see an overview of possible command options and arguments that you can use. If the lineage harvester process fails, you can use the technical lineage troubleshooting guide to fix your issue.

Typical command options and arguments

The following table shows the most commonly used command options and arguments.

Note Although we do not recommend it, you can use more than one lineage harvester connected to a single Collibra Data Intelligence Cloud instance, if you want to separately process data sources on different servers. In this case, all lineage harvesters must share the same configuration file. You then determine which data sources are relevant when you run the full-sync command. If the configuration files are not identical, then one harvester will be deleting data harvested by the other harvester.

Command	Description
`full-sync`	Uploads all of the metadata from the data sources mentioned in your configuration file to the Collibra Data Lineage service, where the metadata is then processed and uploaded to Data Catalog.
`-s "<ID of data source>"`	Uploads only the metadata from a specified data source. For example, `full-sync -s "myOracleDataSource"`. The specified data source must be mentioned in your configuration file. This command allows you to process data from a newly added data source or to refresh a data source in the configuration file, without refreshing the other data sources. This reduces the time you need to upload your data sources, since you only upload specific ones without affecting the others. If you want to process multiple data sources, add `-s "ID of another data source"` per data source to the command. Note You can use this argument multiple times to include multiple data sources.
`--no-matching`	Uploads a technical lineage without stitching the data objects in your technical lineage to the corresponding Column and Table assets in Data Catalog. Note As a result, you won't see the technical lineage of a specific Table or Column asset, but you can still see and browse the full technical lineage.
`sync`	Whereas `full-sync` ingests metadata onto the Collibra Data Lineage service, processes the metadata and syncs it with assets in Data Catalog, the `sync` command only performs this last part: it syncs the metadata—as it exists on the Collibra Data Lineage service—and your assets in Data Catalog. Tip See the following example for advice on how to use the `sync` command to add a new data source without re-harvesting all data sources. Example Let's say you've run `bin/lineage-harvester full-sync`, to upload from all data sources, process the metadata and sync with Data Catalog. You then decide that you want to add a new data source, but not harvest all data sources again. Reference the new data source in the lineage harvester configuration file. Let's say that the new data source has the ID "MyNewSource". Run `bin/lineage-harvester load-sources -s MyNewSource`, to load the new data source and create the ZIP file. Run `bin/lineage-harvester analyze ${zip_file_from_step_2}`, to analyze the new data source on the Collibra Data Lineage service. Run `bin/lineage-harvester sync`, to sync all of the data sources referenced in your configuration file and Data Catalog.
`-s "<ID of data source>"`	Syncs only the metadata on the Collibra Data Lineage service, from a specified data source. For example, `sync -s "myOracleDataSource"`. The specified data source must be mentioned in your configuration file. This command allows you to sync data from one data source without refreshing the other data sources. You must have previously uploaded the metadata to the Collibra Data Lineage service. Warning Only the sources you specify are synced. This means that any previously ingested metadata from non-specified sources, in Data Catalog, is deleted, along with its existing technical lineage. If this is not your intention, consider using `full-sync -s`. With `full-sync -s`, all sources are synced, regardless of which sources are specified by the `-s` command. Therefore, any previously ingested metadata from non-specified data sources remains, as do the respective technical lineages. Note You can use this argument multiple times to include multiple data sources.
`load-sources`	Downloads all your data sources in a separate ZIP file, per data source, to the lineage harvester output folder.
`-s <ID of data source>`	Downloads only the data source with a specific ID. For example, `load-sources -s "myOracleDataSource"`. Note You can use this argument multiple times to include multiple data sources.
`cat passwords.json \| ./bin/lineage-harvester <command-like-full-sync> --passwords-stdin`	Provides passwords of your Collibra Data Intelligence Cloud instance and the data sources in your configuration file to the lineage harvester without storing the passwords in the lineage harvester folder. You can replace `cat passwords.json` by a string generated by your password manager.
`test-connection`	Checks the connectivity to the Collibra Data Lineage service instance and to Data Catalog. The logs will also show the IP addresses of the Collibra Data Lineage service instances that you have to whitelist. This command is mostly used for troubleshooting purposes.
`--help`	Shows an overview of all supported command options and arguments that you can use in the lineage harvester.
`--version`	Shows the version of the lineage harvester that you are using.
`-Dlineage-harvester.log.dir=path/to/log/dir`	Determine the path of the log file.