Create a technical lineage via the lineage harvester

Requirements and restrictions

• In the configuration file, you must use UTF-8 or ISO-8859-1 characters, with the exception of SQL files, which can only be UTF-8 encoded.
• Comments in the lineage harvester configuration file are not supported.
• Technical lineage supports the username and password authentication method for the custom technical lineage.

Format

{
	"general" : {
		"catalog" : {
			"url" : "",
			"username" : "",
		},
		"useCollibraSystemName" : false|true
	},
	"sources" : [ {
		"type" : "ExternalDirectory",
		"id" : "",
		"dirType" : "custom-lineage",
		"collibraSystemName" : "",
		"path" : "",
		"deleteRawMetadataAfterProcessing": false|true
	} ]
}

Example

{
	"general" : {
		"catalog" : {
			"url" : "https://companydomain.collibra.com",
			"username" : "my-Collibra-username",
		},
		"useCollibraSystemName" : false
	},
	"sources" : [{
           "id": "MyCustomLineage",
			"type": "ExternalDirectory",
			"dirType": "custom-lineage",
			"path”: “/path/to/custom-lineage/dir/",
			"collibraSystemName": "MySystemName"
			}
			]
}

Before you begin

If you use a proxy server, connect to the proxy server. For more information, go to Connecting to a proxy server.

Requirements and permissions

• Collibra Data Intelligence Platform.
• You have purchased Collibra Data Lineage.
• A global role with the following global permissions:
  • Catalog, for example Catalog Author
  • Data Stewardship Manager
  • Manage all resources
  • System administration
  • Technical lineage
• A resource role with the following resource permissions on the community level in which you created the domain:
  • Asset: add
  • Attribute: add
  • Domain: add
  • Attachment: add

Steps

1. Start the lineage harvester by entering the  full-sync command.
   • To process data from all data sources in the configuration file:
     For windows:

     .\bin\lineage-harvester.bat full-sync

     For other operating systems:

     ./bin/lineage-harvester full-sync

   • To process data from specific data sources in the configuration file:
     For windows:

     .\bin\lineage-harvester.bat full-sync -s "ID of the data source"

     For other operating systems:

     ./bin/lineage-harvester full-sync -s "ID of the data source"

   Tip For more information and command options, go to Lineage harvesting app command options and arguments.
   Note 

   If you have Snowflake data sources in your lineage harvester configuration file, set the JAVA_OPTS environment variable first. For example, to process data from all data sources including the Snowflake data sources, take the following steps:

   On Windows

   1. Enter one of the following commands:
      • If you use OpenJDK 16:

        set JAVA_OPTS="-Djdk.module.illegalAccess=permit"

      • If you use OpenJDK 17 or higher:

        set JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"

   2. In the same command line, enter the following command:

      .\bin\lineage-harvester.bat full-sync

   Note The set command is specific to the Windows Command Shell. The command is different if you are using PowerShell.

   On Linux

   Enter the following command:

   • If you use OpenJDK 16:

     JAVA_OPTS="-Djdk.module.illegalAccess=permit" ./bin/lineage-harvester full-sync

   • If you use OpenJDK 17 or higher:

     JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED" ./bin/lineage-harvester full-sync

2. When prompted, enter the passwords to connect to Collibra and your data sources. Do one of the following:
   • Enter the passwords in the console.
     The passwords are encrypted and stored in /config/pwd.conf.
   • Provide the passwords via command line.
     The passwords are stored locally and not in your lineage harvester folder.
3. If you are creating technical lineage for dbt Cloud and prompted to enter your API token, enter the token value for the service token that you specified for the tokenName property in the lineage harvester configuration file for dbt Cloud.

Requirements and restrictions

• In the configuration file, you must use UTF-8 or ISO-8859-1 characters, with the exception of SQL files, which can only be UTF-8 encoded.
• Comments in the lineage harvester configuration file are not supported.
• Technical lineage supports the username and password authentication method for the custom technical lineage.

Format

{
	"general" : {
		"catalog" : {
			"url" : "",
			"username" : "",
		},
		"useCollibraSystemName" : false|true
	},
	"sources" : [ {
		"type" : "ExternalDirectory",
		"id" : "",
		"dirType" : "custom-lineage",
		"collibraSystemName" : "",
		"path" : "",
		"deleteRawMetadataAfterProcessing": false|true
	} ]
}

Example

{
	"general" : {
		"catalog" : {
			"url" : "https://companydomain.collibra.com",
			"username" : "my-Collibra-username",
		},
		"useCollibraSystemName" : false
	},
	"sources" : [{
           "id": "MyCustomLineage",
			"type": "ExternalDirectory",
			"dirType": "custom-lineage",
			"path”: “/path/to/custom-lineage/dir/",
			"collibraSystemName": "MySystemName"
			}
			]
}

Before you begin

If you use a proxy server, connect to the proxy server. For more information, go to Connecting to a proxy server.

Requirements and permissions

• Collibra Data Intelligence Platform.
• You have purchased Collibra Data Lineage.
• A global role with the following global permissions:
  • Catalog, for example Catalog Author
  • Data Stewardship Manager
  • Manage all resources
  • System administration
  • Technical lineage
• A resource role with the following resource permissions on the community level in which you created the domain:
  • Asset: add
  • Attribute: add
  • Domain: add
  • Attachment: add

Steps

1. Start the lineage harvester by entering the  full-sync command.
   • To process data from all data sources in the configuration file:
     For windows:

     .\bin\lineage-harvester.bat full-sync

     For other operating systems:

     ./bin/lineage-harvester full-sync

   • To process data from specific data sources in the configuration file:
     For windows:

     .\bin\lineage-harvester.bat full-sync -s "ID of the data source"

     For other operating systems:

     ./bin/lineage-harvester full-sync -s "ID of the data source"

   Tip For more information and command options, go to Lineage harvesting app command options and arguments.
   Note 

   If you have Snowflake data sources in your lineage harvester configuration file, set the JAVA_OPTS environment variable first. For example, to process data from all data sources including the Snowflake data sources, take the following steps:

   On Windows

   1. Enter one of the following commands:
      • If you use OpenJDK 16:

        set JAVA_OPTS="-Djdk.module.illegalAccess=permit"

      • If you use OpenJDK 17 or higher:

        set JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"

   2. In the same command line, enter the following command:

      .\bin\lineage-harvester.bat full-sync

   Note The set command is specific to the Windows Command Shell. The command is different if you are using PowerShell.

   On Linux

   Enter the following command:

   • If you use OpenJDK 16:

     JAVA_OPTS="-Djdk.module.illegalAccess=permit" ./bin/lineage-harvester full-sync

   • If you use OpenJDK 17 or higher:

     JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED" ./bin/lineage-harvester full-sync

2. When prompted, enter the passwords to connect to Collibra and your data sources. Do one of the following:
   • Enter the passwords in the console.
     The passwords are encrypted and stored in /config/pwd.conf.
   • Provide the passwords via command line.
     The passwords are stored locally and not in your lineage harvester folder.
3. If you are creating technical lineage for dbt Cloud and prompted to enter your API token, enter the token value for the service token that you specified for the tokenName property in the lineage harvester configuration file for dbt Cloud.