You can use Task Scheduler on Windows or Crontab on Mac and Linux to make the lineage harvester run scheduled jobs at specific times, dates or intervals. In a scheduled job, the lineage harvester uploads data source information to the Collibra Data Intelligence Platform and Data Catalog automatically creates new relations of the type "Data Element sources / targets Data Element"
- Between data objects in your data source and their corresponding assets in Data Catalog, including the asset that you create when preparing the Data Catalog physical data layer.
- If you are integrating a BI tool, between ingested assets from BI sources and Data Catalog assets from registered data sources.
You can run one scheduled job for each data source that is listed in the same configuration file.
Note If you provide the passwords to your Collibra environment and/or to your individual data sources via stdin, you have to use the correct command.
Example You created a configuration file with two data sources. Data source A can run a scheduled job each day at 11 pm, while data source B can run a scheduled job every two days at 6 am.