Prepare Matillion <source ID> configuration file

You use the lineage harvester configuration file to access Matillion data objects. The lineage harvester processes the data objects to create a technical lineage. However, if the useCollibraSystemName property in the lineage harvester configuration file is set to true, you also have to provide a <source ID> configuration file to define the system name for all sources and targets in the Matillion integration.

This is useful if you have multiple databases with the same name and want to distinguish between them in the technical lineage harvester by specifying the system or server specific to each.

Note To preserve stitching, you need a System asset in Data Catalog of the same name of each system or server you specify in your <source ID> configuration file.

Prerequisites

  • The useCollibraSystemName in the lineage harvester configuration file is set to true.
  • You have Admin permission on all objects that you want to harvest.

Steps

  1. Create a new JSON configuration file in the lineage harvester config folder.
  2. Give the JSON file the same name as the value of the Id property in the lineage harvester configuration file.
    Example If the value of the id property in the lineage harvester configuration file is matillion-source-1, then the name of your JSON file should be matillion-source-1.conf.
    Important Your JSON file must have the file extension .conf.
  3. For each Matillion connection, you can add the following content to the JSON file:

    Property

    		DescriptionMandatory?
    collibraSystemNamesThis section contains the system information for Matillion.Yes
    sources

    Use this section to define the system names of all sources in the Matillion job.

    		Yes
    jobName
    		The Matillion job name.Yes
    collibraSystemName

    The name of the Matillion source system or server.

    		Yes
    targets
    		Use this section to define the system names of all targets in the Matillion job.Yes
    jobName
    		The Matillion job name.Yes
    collibraSystemName
    		The name of the Matillion target system or server.Yes

    Copy code

    {
	"collibraSystemNames":
		{
			"sources":[
				{
					"jobName":"<name of job>",
					"collibraSystemName":"<name>"
				},
				{
					"jobName":"<name of job>",
					"collibraSystemName":"<name>"
				}
			],
			"targets":[
				{
					"jobName":"<name of job>",
					"collibraSystemName":"<name>"
				},
				{
					"jobName":"<name of jobv",
					"collibraSystemName":"<name>"
				}
			]
		}
}

    Tip Click Copy code to copy the example to your clipboard.

    		 
  4. Save the configuration file.

Supported source system dialects

The dialect property in the lineage harvester configuration file refers to the target system. The following table shows the supported source system dialects.

Matillion source dialect Technical lineage source dialect
Amazon Redshift redshift
Redshift redshift
IBM DB2 db2
IBM DB2 for i db2
Microsoft SQL Server mssql
MySQL mysql
Netezza netezza
Oracle oracle
PostgreSQL postgres
SAPHana hana
Snowflake snowflake
SQL Server (Microsoft Driver) mssql
Sybase ASE sybase
Teradata teradata