Upgrade the lineage harvester

Each new lineage harvester adds features and enhancements to the previous version. We highly recommend that you always use the newest lineage harvester available.

If you have created a technical lineage using an older lineage harvester, you can easily upgrade to the newest lineage harvester and reuse your configuration file.

Tip For a list of differences between lineage harvester versions, see the lineage harvester change log.

Steps

  1. Download the newest lineage harvester from the Collibra Downloads page. To log in to the Collibra Downloads page, use your Collibra.com username and password.
  2. Install the lineage harvester and a new lineage harvester folder is created.
  3. Copy all files from your config folder in the old lineage harvester folder to the config folder in the new lineage harvester folder.
    All files, including the pwd.conf and lineage-harvester.conf files, are in the config folder in the new lineage harvester folder.
  4. In the config folder, open the lineage-harvester.conf file to check if there are other auxiliary files to be moved to the new lineage harvester folder. If needed, copy those files from the old lineage harvester folder to the new lineage harvester folder.

    Those files can be the custom technical lineage JSON file, the Informatica Intelligent Cloud Services <source ID> configuration file, the Matillion <source ID> configuration file, and so on.

  5. If you have customized SQL files that end with -custom.sql in the sql folder in the old lineage harvester folder, complete the following steps:
    1. Compare the original SQL files before customization with the SQL files in the new lineage harvester folder. For example, if you have a customized SQL file named access_history-custom.sql, compare the access_history.sql file in the old lineage harvester folder with the access_history.sql file in the new lineage harvester folder.
    2. Take any of the following actions:
      • If the SQL files are identical, copy the customized SQL files from the old lineage harvester folder to the new lineage harvester folder.
      • If the SQL files are not the same, complete the following steps:
        1. Create new SQL files that end with -custom.sql in the new lineage harvester folder based on the SQL files in the new lineage harvester folder.
        2. Review the customizations in the customized SQL files in the old lineage harvester folder, and make the same customizations to the newly created customized SQL files in the new lineage harvester folder.
      3. Example Take the access_history-custom.sql file as an example, and the customization in the access_history-custom.sql file was to change the database.schema from SNOWFLAKE.ACCOUNT_USAGE to MYDB.ACCOUNT_USAGE.
      1. Compare the following files:
        • lineage-harvester-OLD/sql/snowflake/access_history.sql
        • lineage-harvester-NEW/sql/snowflake/access_history.sql
      2. Take any of the following steps:
        • If the access_history.sql files are identical, copy the access_history-custom.sql file from lineage-harvester-OLD/sql/snowflake to the lineage-harvester-NEW/sql/snowflake directory.
        • If the access_history.sql files are not the same, complete the following steps:
          1. Create an access_history-custom.sql file in the lineage-harvester-NEW/sql/snowflake directory by copying the content of the lineage-harvester-NEW/sql/snowflake/access_history.sql file to the new access_history-custom.sql file.
          2. Customize the new access_history-custom.sql file by changing the database.schema from SNOWFLAKE.ACCOUNT_USAGE to MYDB.ACCOUNT_USAGE.
      Note Beginning with the lineage harvester version 2023.02, the SQL file that was named access_history_lineage_query_text.sql has been renamed to access_history.sql.
  6. Use the full-sync command to synchronize all data sources in your configuration file.
    The lineage harvester synchronizes your data sources on the Collibra Data Lineage service and refreshes your technical lineage.

What's next

You can check the progress of the technical lineage creation in Activities in your Collibra Platform environment. The Results field indicates how many relations were imported into Data Catalog. Go to the status page to see the log files of the SQL analysis.

If the lineage harvester log shows an error message or the harvesting process fails, you can use the technical lineage common errors and issues in Collibra Support Portal to fix the error.