Prepare a <source ID> configuration file

The lineage harvester uses a lineage harvester configuration file to collect the Azure Data Factory data objects. It then sends the metadata to the Collibra Data Lineage service instance.

{
     "found_dbname=databasename1;found_hostname=server-name.onmicrosoft.com;found_schema=schema1": {
         "dbname": "mssql-database-name",
         "schema": "mssql-schema-name",
         "dialect": "mssql",
         "collibraSystemName": "mssql-system-name"
     },
     "found_dbname=datafactory_linkedservice;found_hostname=*": {
         "dbname": "linkedservice-dbname",
         "schema": "linkedservice-schema",
         "collibraSystemName": "linkedservice-system-name"
     }
}	

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-adf, the name of your JSON file must be my-adf.conf.
3. For each database in Azure Data Factory, add the following content to the JSON file:

   Property	Description	Mandatory?
   found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name> | found_dbname=<datafactory_name>_<linkedservice_name>;found_hostname=*	The information of the supported data sources in Azure Data Factory to be collected by Collibra Data Lineage. You can specify any of the following values for the found_dbname property: A database name. And then you can specify the following properties: found_hostname=<server name>, where <server name> is the name of the server that the database is running on. found_schema=<schema name>, where <schema name> is the name of the schema. This property is optional. The combination of <datafactory_name>_<linkedservice_name>, where <datafactory_name> is a data factory name and <linkedservice_name> is a linked service name. If you use this combination, specify * for the found_hostname property. Tip  You can use wildcards to capture multiple connection string combinations: Show me the supported wildcards Pattern Description * Matches everything. ? Matches any single character. [seq] Matches any character in "seq". [!seq] Matches any character not in "seq".	Yes
   dbname	The name of the database asset in Data Catalog. Specify this property with the database name that you created when you prepared the Data Catalog physical data layer.	No
   schema	The name of the schema asset in Data Catalog. Specify this property with the schema name that you created when you registered the data source. If the Collibra Data Lineage fails to find the schema that you specify, it uses the default schema.	No
   dialect	If you specify a database name for the found_dbname property, select one of the following dialects. If you specify a linked service name for the found_dbname property, ignore this property. Click here for a list of dialects of supported data sources in Azure Data Factory. azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for a IBM Db2 data source. generic, for any data source. greenplum, for a Greenplum data source. hana, for a SAP HANA data source. hive, for a Hive data source. impala, for a Impala data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for an PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source. vertica, for a Vertica data source.	No
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. If you don't specify a value for this property, DEFAULT is shown in the technical lineage. How do I configure this property if I have two databases with the same name? Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows. "found_dbname=databasename1;found_hostname=*;found_schema=schema1": { "dbname": "Customers", "schema": "mssql-schema-name", "dialect": "mssql", "collibraSystemName": "Customers-Europe" }, "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": { "dbname": "Customers", "schema": "oracle-schema-name", "dialect": "oracle", "collibraSystemName": "Customers-USA" }, Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.	No

4. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the DataStage data objects. It then sends the metadata to the Collibra Data Lineage service instance.

{
  "OdbcDataSources": {
    "oracle-data-source": {
      "dbname": "my-oracle-database",
      "schema": "my-oracle-schema",
      "dialect": "oracle",
      "collibraSystemName": "my-system"
    },
    "mssql-data-source": {
      "dbname": "my-mssql-database",
      "schema": "my-mssql-schema",
      "dialect": "mssql",
      "collibraSystemName": "my-system"
    }
  },
  "NonOdbcConnectors": {
    
    "admin@database-name": {
      "dbname": "my-netezza-database",
      "schema": "my-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    },
    "admin@second-database-name": {
      "dbname": "my-second-netezza-database",
      "schema": "my-second-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    }
  },   
  "jobs": [
    "my_job_1",
    "my_job_2"
  ],
  "jobParameters": [
    {
      "name": "job_parameter_name_1",
      "value": "job_parameter_value_1"
    },
    {
      "name": "job_parameter_name_2",
      "value": "job_parameter_value_2"
    }
  ]
}

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-datastage, the name of your JSON file must be my-datastage.conf.
3. For each database in DataStage, add the required content to the JSON file.

   Property	Description
   OdbcDataSources	Open Database Connectivity data sources in IBM InfoSphere DataStage for which you want to create a technical lineage.
   <data-source-name>	The ODBC data source name that you use in your DataStage projects. This section contains the properties to translate the database, schema and dialect.
   dbname	The name of your database, to which the ODBC data source connection refers.
   schema	The name of your schema, to which the ODBC data source connection refers.
   dialect	The dialect of the referenced database. See the list of allowed values. You can enter one of the following values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for an SAP HANA data source. hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source. hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source. Important  To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. This property is optional.
   NonOdbcConnectors	Other data source connectors in IBM InfoSphere DataStage for which you want to create a technical lineage. For example, DB2, Oracle or Netezza. Note This section is optional.
   <data-source-connector-ID>	The data source username and database of the connector that you use in your DataStage projects. This usually looks like for example admin@database-name. The combination of the username and database name should be unique. The following section contains the properties to translate the database, schema and dialect.
   dbname	The name of your database, to which the data source connection refers.
   schema	The name of your schema, to which the data source connection refers.
   dialect	The dialect of the referenced database. See the list of allowed values. You can enter one of the following values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for an SAP HANA data source. hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source. hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source. Important  To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. This property is optional.
   Jobs	The jobs that you want the lineage harvester to collect and process to create the technical lineage. This section is optional. The following rules apply when you specify this section: Specify jobs that are executed so that the technical lineage graph does not include any job parameters with undefined values. Specify only the first and parent jobs in a sequence of executed jobs. The lineage harvester automatically collects all jobs that are called by the parent jobs. For example, if you have the a sequence of jobs that include job1, job2, job3, job4, and job5, where job1 calls job2, job2 calls job3, job3 calls job5, and job4 calls job3. Specify only job1 and job4, and the lineage harvester collects and processes all five jobs based on the sequence. If you do not specify this section, the lineage harvester collects all jobs, but without proper sequencing. Therefore, some inherited parameters might not be parsed.
   JobParameters	The runtime parameters that are not in the DSX and ENV files. You can specify multiple job parameters.
   name	The name of the job parameter.
   value	The value of the job parameter.

4. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the dbt data objects. It then sends the metadata to Collibra Data Lineage service for processing. By default, the lineage harvester downloads all accounts that are accessible with the API token that you provided in the lineage harvester configuration file. For each account, the lineage harvester downloads all jobs and the resulting dbt models for each job. You can use this <source ID> configuration file to reduce the amount of data objects to be downloaded and enhance the lineage harvester performance in the following ways:

• Filter the projects and jobs to be downloaded. Include projects and jobs to be downloaded by specifying the filter property.
• Specify different Collibra system names for different projects by specifying the collibraSystemNames property .
• Map a materialization as a view instead of a table by specifying the materializedMapping property.

{
	"collibraSystemNames":{
	    "projects":[
		{"project_id":"654321","collibraSystemName":"SystemName"}
	    ]
	},
	"filter":{
	    "jobIds":[1234],
	    "projectIds":[654321]
	},
	"materializedMapping":{
	    "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
	}
}	

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-dbt, the name of your JSON file must be my-dbt.conf.
3. For each database in dbt, add the following content to the JSON file:

   Property	Description	Required?
   collibraSystemNames	You can use this section to specify the Collibra System Name for each project.	No
   projects	This section contains the project names and the Collibra system names.	No
   project_id	Your project ID. You can find the project ID in the dbt URL right after projects. For example, if your dbt URL is https://cloud.getdbt.com/develop/54321/projects/12345 , your project_id is 12345.	No
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. See an example. In this code example, the project with the 12345 project ID is stitched to the systemname1 System asset in Data Catalog. { "collibraSystemNames":{ "projects":[ {"project_id":"12345","collibraSystemName":"systemname1"} ] }, }	No
   filter	You can use this section to include projects and jobs to be downloaded. Collibra Data Lineage downloads and processes only the specified jobs and projects. See an example. In this code example, the job with the 1234 job ID and the projects with the 98 and 5678 project IDs are downloaded. { "filter": { "jobIds": [ 1234 ], "projectIds": [ 98, 5678 ] } }	No
   jobIds	The job IDs of the jobs that you want to include. Specify an integer. Do not specify a string. To get your job ID, in your dbt, select Deploy and then Jobs. Select a job and you can find your job ID in the URL. For example, if your URL is cloud.getdbt.com/deploy/65432/projects/23456/jobs/123456, 123456 is your job ID.	No
   projectIds	The account IDs of the accounts that you want to include. Specify an integer. Do not specify a string. To get your account ID, in dbt, click the gear icon in the upper right, select Account Settings and find your account ID in the URL. For example, if your URL is cloud.getdbt.com/settings/accounts/65432, 65432 is your account ID.	No
   materializedMapping	Indicates how materializations in dbt are mapped. If you do not specify this property, CollibraData Lineage maps materializations to tables by default. You can change the mapping of a materialization to view. In the following example, the ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES materialization is mapped to a view. "materializedMapping":{ "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW" }	No

4. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the Informatica PowerCenter data objects. It then sends the metadata to the Collibra Data Lineage service instance.

{
	"connectionDefinitions": {
		"oracle_source": {
			"dbname": "oracle-source-database-name1",
			"schema": "my Oracle source schema",
			"dialect": "oracle"
		},
		"oracle_target": {
			"dbname": "oracle-target-database-name2",
			"schema": "my other oracle target schema",
			"dialect": "oracle"
		}
	},
	"collibraSystemNames": {
		"databases": [
			{
				"dbname": "oracle-source-database-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"dbname": "oracle-target-database-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		],
		"connections": [
			{
				"connectionName": "oracle-connection-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"connectionName": "oracle-connection-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		]
	}
}	

Steps

1. Create a new JSON file in the lineage harvesterconfig folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-infa, the name of your JSON file must be my-infa.conf.
3. For each database, add the required content to the JSON file.

   If certain properties are not specified in the source ID file, an analyze error called CONFIGURATION is displayed in the transformations table on the Sources tab page when the technical lineage is created. The unspecified properties are marked as UNDEFINED in the analyze error. For more information about the analyze errors, go to Analyze errors and possible solutions in Technical lineage Sources tab page.

   Property	Description
   connectionDefinitions	This section contains the connection properties to a source in Informatica PowerCenter.
   <connectionName>	The type of your source or target data source. This section contains the connection properties to a source or target in Informatica PowerCenter. Note Define a connection in the JSON file only once; specifically, define a data source with the <connectionName> property specified only once in the JSON file. If you define a connection multiple times, unexpected lineage and stitching issues might occur.
   dbname	The name of your source or target database. When you specify the dbname and schema properties, Collibra Data Lineage can stitch the data objects to the assets in Data Catalog. If the properties are not specified, the data objects are not stitched.
   schema	The name of your source or target schema. When you specify the dbname and schema properties, Collibra Data Lineage can stitch the data objects to the assets in Data Catalog. If the properties are not specified, the data objects are not stitched.
   dialect	The dialect of the referenced database. If you specify a dialect for a database, the value overrides the dialect that you specify in the lineage harvester configuration file for this database. For any databases that do not have a dialect specified in the source ID file, the dialect that you specify in the lineage harvester configuration file is used as a global dialect. See the list of allowed values. You can enter one of the following values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for an SAP HANA data source. hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source. hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source. Important  To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.
   collibraSystemNames	This section contains the system or server name that is specified in your database and referenced in your connection. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. The following rules apply when you specify the collibrasystemname properties in this file and the lineage harvester configuration file: If you specify this property for a database or connection, the value of this property overrides the value in the lineage harvester configuration file for the database or connection. For any databases or connections that do not have a Collibra system name specified in the source ID file, the value of the collibrasystemname property in the lineage harvester configuration file is used as a global value.
   databases	This section contains the database information. This is required to connect directly to the system or server of the database.
   dbname	The name of the database. The database name is the same as the name you entered in the <connectionName> section.
   collibraSystemName	The system or server name of the database.
   connections	This section contains the connection information. This is required to reference to the system or server of the connection.
   connectionName	The name of the connection.
   collibraSystemName	The system or server name of the connection.

   Important If you are using variables in Informatica PowerCenter, add the value of the variable instead of the name in the connection definitionsJSON file. For example, if the parameter file contains $DBConnection_dwh=DWH_EXPORT, add the following connection definitions to the JSON file:

   {
   	"DWH_EXPORT":
   
   		{ "dbname": "DWH", "schema": "DBO" }
   }

   ©

4. Save the <source ID> configuration file.

You use the lineage harvester configuration file to access Informatica Intelligent Cloud Services Data Integration data objects. The lineage harvester processes the data objects to create a technical lineage. You also have to prepare a specific <source ID> configuration file that defines the Intelligent Cloud Services system name.

Important You must prepare a <source ID> configuration file regardless of whether the useCollibraSystemName property in your lineage harvester configuration files is set to true or false.

Prerequisites

You have Admin permission on all objects that you want to harvest.

{
	"collibraSystemNames": {
		"connections": [
			{
				"connectionName": "DG_con_standby_cmdm_clientors",
				"collibraSystemName": "PUBLIC"
			},
			{
				"connectionName": "DG_con_dev_dg_dgiauser_su",
				"collibraSystemName": "PUBLIC"
			}
		]
	},
	"connectionDefinitions": [
		{
			"connectionName": "DG_con_standby_cmdm_clientors",
			"databaseName": "main",
			"schemaName": "dbo",
			"dialect": "oracle"
		},
		{
			"connectionName": "DG_con_dev_dg_dgiauser_su",
			"databaseName": "main",
			"schemaName": "dbo",
			"dialect": "oracle"
		}
	]
}	

Steps

1. Create a new JSON configuration file in the lineage harvesterconfig folder.

   If you have a data source with a large size for an Informatica Intelligent Cloud Services connection, consider creating more than one JSON file for the data source. Each JSON file must have a unique name. The contents in the JSON files are the same. In this way, you can avoid errors that might occur when the lineage harvester ingests metadata from one source with a large size.

2. Give the JSON file the same name as the value of the Id property in the lineage harvester configuration file.
   Example If the value of the Id property in your lineage harvester configuration file is iics-source-1, then the name of your JSON file should be iics-source-1.conf.
Important Your JSON file must have the file extension .conf.
3. For each Informatica Intelligent Cloud Services connection, you can add the following content to the JSON file:

   Property	Description	Required?
   collibraSystemNames	This section contains the system information for Informatica Intelligent Cloud Services.
   connections	This section contains the system connection information. This is required to reference to the system or server of the connection.
   connectionName	The name of the connection. The name must match the System asset name in Data Catalog for stitching.	Yes
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.	No
   connectionDefinitions	This section contains the database, schema and dialect information for each connection in Informatica Intelligent Cloud Services. Note You can add connection information for each connection in the connections section.
   connectionName	The name of the connection. The name must match with the name in a connection name in the connections section. This property is required.	Yes
   databaseName	The name of your database. The name must match the Database asset name in Data Catalog for stitching.	Yes
   schemaName	The name of your schema. The name must match the Schema asset name in Data Catalog for stitching.	Yes
   dialect	The dialect of the connection. Specify this property for Collibra Data Lineage to properly extract and parse queries that are related to this connection. You can enter one of the following values: bigquery db2 hana hive greenplum mssql mysql netezza oracle postgres redshift snowflake spark teradata	No

4. Save the configuration file.

The lineage harvester uses the lineage harvester configuration file to collect the Looker data objects and send them to the Collibra Data Lineage service instance.

The <source ID> configuration file allows you to:

• Filter on the Looker folders from which you want to ingest metadata.
• If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in Looker.
  Collibra Data Lineage uses the system names to match the structure of databases in Looker to assets in Data Catalog.

{
   "Connections":{
      "connection-object1":{
         "dialect":"mssql",
         "schema":"mssql-schema-name",
         "dbname":"mssql-database-name",
         "collibraSystemName":"mssql-system-name"
      },
      "connection-object2":{
         "dialect":"oracle",
         "schema":"oracle-schema-name",
         "dbname":"oracle-database-name",
         "collibraSystemName":"oracle-system-name"
      }
   },
   "filters":[
      {
         "domainId":"605fa8ae-f8c6-4261-938b-8326e2806f3d",
         "description":"Databricks_folder",
         "folderIds":[
            "abc-123",
            "def-456"
         ]
      },
      {
         "domainId":"245bc5a4-4c30-44b5-8356-ddbe708b56d6",
         "description":"personal",
         "folderIds":[
            "hij-789*",
            "jkl-101112"
         ]
      }
   ]
}	

Steps

1. Create a new JSON file in the lineage harvesterconfig folder.
2. Give the JSON file the same name as the value of the Id property in the lineage harvester configuration file.
   Example The value of the Id property in the lineage harvester configuration file is looker-source-1. As a result, the name of your JSON file should be looker-source-1.conf.
   Important Your JSON file must have the file extension .conf.
3. For each database in Looker, add the following content to the JSON file:

   Property	Description	Mandatory?
   Connections	This section contains all Looker connections for which you want to create a technical lineage.	Yes
   <connection name>	The name of a connection object in Looker.	Yes
   dialect	The dialect of the supported data source in Looker.	No
   schema	The name of the default schema of a supported data source in Looker. If the lineage harvester fails to find a specific schema, it uses the default schema.	No
   dbname	The name of the database of a supported data source in Looker.	No
   collibraSystemName	The system or server name of a database. If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT". How do I configure this property if I have two databases with the same name? Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows. "connection-object1": { "dialect": "mssql", "schema": "mssql-schema-name", "dbname": "Customers", "collibraSystemName": "Customers-Europe" }, "connection-object2": { "dialect": "oracle", "schema": "oracle-schema-name", "dbname": "Customers", "collibraSystemName": "Customers-USA" }	Yes
   filters	Optionally, use this section to specify the Looker folders from which you want to ingest metadata. Note You can filter on Looker folders, but not on Looker data sets. That's because Looker data sets are linked directly to the server, instead of a folder, as shown in the Looker metadata overview. Looker data sets are ingested in the default domain, regardless of any filtering. Let’s say, for example, you filter on folder B. A Looker Folder asset is created in the specified domain in Collibra, and all of the metadata in folder B is ingested. If folder B has a parent folder A, then a Looker Folder asset is created (in the domain specified for folder B) to preserve the hierarchy, but no metadata from folder A is ingested. You can specify more than one Looker folder for ingestion into a single domain in Collibra. Warning If you don't want to filter on Looker Folders, you must completely remove this filters section. Tip There are significant benefits to filtering by folder ID. For information, see the filters > folderIds property description. Tip  You can use wildcards to capture multiple connection string combinations: Show me the supported wildcards Pattern Description * Matches everything. ? Matches any single character. [seq] Matches any character in "seq". [!seq] Matches any character not in "seq".	No
   domainId	The unique resource ID of the domain (or domains), in Collibra, in which you want to ingest data objects from one or more Looker Folders. Tip You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.
   description	Any description, as you see fit.
   folderNames	The name (or names) of the Looker Folders from which you want to ingest. Note You must specify either a folder name, a folder ID, or both.
   folderIds	The ID (or IDs) of the Looker Folder you want to ingest. Note You must specify either a folder ID, a folder name, or both. Tip If you filter by folder ID, filtering is carried out via the API, instead of on the Collibra Data Lineage service instances. When you filter by folder ID, the lineage harvester accesses only the folders you specify via this property, and sends only that metadata to the Collibra Data Lineage service instance for processing and ingestion in Data Catalog. Conversely, if you filter by folder name (via the folderNames property), metadata from all Looker folders is sent to the Collibra Data Lineage service instance. Only then is filtering applied. What are the advantages? Faster integration testing, as you can filter on a single folder. Enhanced data security and privacy by not harvesting folders that contain sensitive information. Improved processing times by not including folders dedicated to, for example, development and testing. This is especially beneficial for organizations with a lot of data in Looker.

4. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the Matillion data objects. It then sends the metadata to the Collibra Data Lineage service instance.

{
    "found_dbname=dbtest;found_hostname=test": {
        "collibraSystemName": "mssql-system-name"
    },
    "found_dbname=testsid;found_hostname=*": {
        "dbname": "oracle-database-name",
        "schema": "oracle-schema-name",
        "collibraSystemName": "oracle-system-name"
    }	
}

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-matillion, the name of your JSON file must be my-matillion.conf.
3. Add the required content to the JSON file.

   Property	Description	Mandatory?
   found_dbname=<database name>;found_hostname=<server name>	The information of the supported data sources in Matillion to be collected by Collibra Data Lineage. <database name> The database name in Matillion. <server name> The name of the server that the database is running on. You can specify found_hostname=* to include all servers. Note Define a connection in the connection definitions only once; specifically, define a data source with the found_dbname and found_hostname properties specified only once in the connection definitions. If you define a connection multiple times, unexpected lineage and stitching issues might occur. Tip  You can use wildcards to capture multiple connection string combinations: Show me the supported wildcards Pattern Description * Matches everything. ? Matches any single character. [seq] Matches any character in "seq". [!seq] Matches any character not in "seq".	Yes
   dbname	The name of the database asset in Data Catalog. Specify this property with the database name that you created when you prepared the Data Catalog physical data layer. If you leave this property blank, the database is stitched to the database of DEFAULT in Data Catalog.	No
   schema	The name of the schema asset in Data Catalog. Specify this property with the schema name that you created when you registered the data source. If you leave this property blank, the schema is stitched to the schema of DEFAULT in Data Catalog.	No
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog. If you leave this property blank, the system is stitched to the system of DEFAULT in Data Catalog. If you are missing lineage or your lineage objects aren’t stitching to Catalog assets in Data Catalog as you expect, ensure this property is specified properly. How do I configure this property if I have two databases with the same name? For example, if you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows. { "found_dbname=dbtest;found_hostname=test": { "dbname": "Customers", "collibraSystemName": "Customers-Europe" }, "found_dbname=testsid;found_hostname=*": { "dbname": "Customers", "schema": "oracle-schema-name", "collibraSystemName": "Customers-USA" } } Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.	No

4. Save the <source ID> configuration file.

The lineage harvester uses the configuration file to connect to MicroStrategy. You must also prepare a MicroStrategy <source ID> configuration file to:

• Specify the default domain, meaning the domain in Collibra in which the corresponding assets of MicroStrategy metadata will be ingested if domain mapping is not configured.
  
  Note If you do configure domain mapping, the default domain is still the destination domain of the MicroStrategy Server asset.
• Optionally, specify from which MicroStrategy projects you want to ingest metadata, and into which domains you want to ingest the corresponding assets.
• Optionally, configure data source mapping, to map the name of a data source returned by the lineage harvester to the true name of the data source.
  
  Note Mapping doesn't work for custom SQL.

Tip "<source ID>" refers to the value of the Id property in the lineage harvester configuration file.

{
	"default_domain_id": "66ab7911-d84a-43d6-b9f7-40130jd84ds98",
	"filters": [
		{
			"domainId": "3ae946ec-9030-46b8-ad86-a217b8f99bab",
			"projectIDs": ["3FF7CF7D4C5A2AEBF14D8B9802698B27"]
		},
		{
			"domainId": "21cfb3ae-0616-478e-9060-114130540be5",
			"projectNames": ["Human Resources Analysis Module"]
		}
	],
    "datasourceMapping": [
	 {
	     "found_datasource": "REDSHIFT",
	     "found_project": "*",
	     "mapping": {
		 	"dbname": "RD_pearl",
		  	"schema": "Default_North",
		  	"dialect": "spark",
		  	"collibraSystemName": "TV_dev"
	     }
	 }
    ]
}			

Steps

1. Create a new JSON file in the lineage harvesterconfig folder.
2. Give the JSON file the same name as the value of the Id property in the lineage harvesterconfiguration file.
   Example If the value of the Id property in the lineage harvester configuration file is mstr-source-1, then the name of your JSON file should be mstr-source-1.conf.
   Important Your JSON file must have the file extension .conf.

Property	Description	Mandatory
default_domain_id	The domain in which you want the corresponding assets of MicroStrategy metadata to be ingested. Note If you configure filtering, only the MicroStrategy Server asset is ingested into this default domain.	Yes
filters	This section allows you to specify: From which MicroStrategy projects you want to harvest metadata. Into which domains in Collibra you want to ingest the corresponding assets. If you don't want to filter on projects, don't include this section in your <source ID> configuration file.	No
domainId	The unique resource ID of the domain (or domains) in Collibra in which you want to ingest the MicroStrategy assets. Tip If you use a filters section, you must include the domainId property in the section. If, by chance, you want to filter on certain projects, but you want to ingest all assets into the default domain, then the value of the domainId property must match the value of the default_domain_id property. Show me an example "default_domain_id": "1234567890", "filters": [ { "domainId": "1234567890", "projectNames": ["MicroStrategy Tutorial","Testing_MSTR"] }, How do I find a domain reference ID? Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.	No
projectIds	The IDs of the MicroStrategy projects from which you want to ingest metadata.	No
projectNames	The project names of the MicroStrategy projects from which you want to ingest metadata.	No
datasourceMapping	This optional section allows you to configure data source mapping. Include this section only if you need to differentiate between multiple data sources that have the same name. Note Mapping doesn't work for custom SQL.	No
found_datasource	The name of the data source that was returned by the lineage harvester, as shown in the technical lineage. Note The data source name is case-sensitive.	Yes
found_project	The name of the project in which the data source information resides. You can specify an asterisk (*) to search for data source information across all projects.	Yes
mapping	Use this section to map the data source name that was returned by the lineage harvester to the true name of the data source. Example You have a Redshift data source named "RD_pearl", but the lineage harvester has returned the name "Redshift_connection". You can configure the datasourceMapping section as follows: { "datasourceMapping": [ { "found_datasource": "REDSHIFT", "found_project": "*", "mapping": { "dbname": "RD_pearl", "collibraSystemName": "TV_dev" } } ] }	Yes
dbname	The name of the database to which you want to map the found data source.	Yes
schema	The name of the schema in MicroStrategy.	No
dialect	The dialect of the data source in MicroStrategy.	No
collibraSystemName	The system or server name of a database. If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT". If you set the useCollibraSystemName property to false in your lineage harvester configuration file, leave this property empty as follows: "collibraSystemName": "". How do I configure this property if I have two databases with the same name? Let's assume that you have a data source named Customers. You use this data source connection in two different projects, Project_A and Project_B, but they are actually two different databases. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-North and Customers-South. You can then configure this property as follows. "datasourceMapping": [ { "found_datasource": "Customers", "found_project": "Project_A", "mapping": { "dbname": "Customers", "collibraSystemName": "Customers_North" } }, { "found_datasource": "Customers", "found_project": "Project_B", "mapping": { "dbname": "Customers", "collibraSystemName": "Customers_South" } } ] Warning The values of this property must exactly match the name of your System asset in Collibra.	Yes

3. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the Power BI data objects. It then sends the metadata to the Collibra Data Lineage service instances.

The <source ID> configuration file allows you to:

• Map the names of the server, database and schema that were collected by the lineage harvester to their true names.
  
  Note Mapping doesn't work for custom SQL.
• Configure workspace filtering.
  
  Tip We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.
• If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in Power BI. Collibra Data Lineage uses the system names to match the structure of databases in Power BI to assets in Data Catalog.

{
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "mssql-database-name",
		"schema": "mssql-schema-name",
		"dialect": "mssql",
		"collibraSystemName": "mssql-system-name"
	},
	"found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
		"dbname": "oracle-database-name",
		"schema": "oracle-schema-name",
		"dialect": "oracle",
		"collibraSystemName": "oracle-system-name"
	},
	"filters":[ 
		{
		"domainId": "<domain-ref-id>",
		"description": "FirstFilter",
		"workspaceNames": ["*"],
		"excludeWorkspaceIds": ["workspaceC", "workspaceD"]
		},
		{
		"domainId": "<domain-ref-id>",
		"description": "SecondFilter",
		"workspaceNames": ["workspace3", "workspace4"],
		"capacityIds": ["id1","id2"]
		}
	]
}	

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Give the JSON file the same name as the value of the sourceId property in the lineage harvester configuration file.
   Example The value of the sourceId property in the lineage harvester configuration file is power-bi-source-1. Therefore, the name of your JSON file should be power-bi-source-1.conf.
   Important Your JSON file must have the file extension .conf.
3. For each database in Power BI, add the following content to the JSON file:
   

Property	Description	Mandatory?
found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>	The database information of supported data sources in Power BI that is typically collected by the lineage harvester. Specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema). You then use the child properties to map the names collected by the lineage harvester to the true names. Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements. Important The keys that you specify must be unique. Show me an example of a duplicate key that would result in an error The following configuration would result in an error, becausee the key found_dbname=databasename1;found_hostname=*;found_schema=schema1 is specified twice. { "found_dbname=databasename1;found_hostname=*;found_schema=schema1": { "dbname": "mssql-database-name", "schema": "mssql-schema-name", "dialect": "mssql", "collibraSystemName": "mssql-system-name" }, "found_dbname=databasename1;found_hostname=*;found_schema=schema1": { "dbname": "oracle-database-name", "schema": "oracle-schema-name", "dialect": "oracle", "collibraSystemName": "oracle-system-name" }, "filters":[ { "domainId": "<domain-ref-id>", "description": "FirstFilter", "workspaceNames": ["*"], "excludeWorkspaceIds": ["workspaceC", "workspaceD"] }, { "domainId": "<domain-ref-id>", "description": "SecondFilter", "workspaceNames": ["workspace3", "workspace4"], "capacityIds": ["id1","id2"] } ] } Tip  You can use wildcards to capture multiple connection string combinations: Show me the supported wildcards Pattern Description * Matches everything. ? Matches any single character. [seq] Matches any character in "seq". [!seq] Matches any character not in "seq".	Yes
dbname	The name of the database of a supported data source in Power BI.	No
schema	The name of the schema of a supported data source in Power BI. If the lineage harvester fails to find a specific schema, it uses the schema you specify in this property. Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements.	No
dialect	The dialect of the supported data source in Power BI. Click here for a list of dialects of supported data sources in Power BI. azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. mssql, for a Microsoft SQL Server data source. oracle, for an Oracle data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. sybase, for a Sybase data source.	No
collibraSystemName	The system or server name of a database. If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT". How do I configure this property if I have two databases with the same name? Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows. "found_dbname=databasename1;found_hostname=*;found_schema=schema1": { "dbname": "Customers", "schema": "mssql-schema-name", "dialect": "mssql", "collibraSystemName": "Customers-Europe" }, "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": { "dbname": "Customers", "schema": "oracle-schema-name", "dialect": "oracle", "collibraSystemName": "Customers-USA" }, Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra. Important If you are using a <source ID> configuration file for the purpose of providing the true system name of an ODBC database in Power BI, you are not required to: Set the useCollibraSystemName property in the lineage harvester configuration file to true. Specify a Collibra system name in the <source ID> configuration file. However, if the useCollibraSystemName property is set to true in the lineage harvester configuration file, then you must specify a Collibra system name in the <source ID> configuration file.	Yes (unless you are using the <source ID> file to provide the true system names of ODBC databases in Power BI.)
filters	This section allows you to specify the Power BI workspaces from which you want to ingest metadata. The filters work as "workspace AND workspace AND capacity AND capacity", meaning that if you specify a capacity, all of the workspaces in that capacity are also ingested. Warning If you don't want to specify the Power BI workspaces from which to ingest, you must completely remove this filters section. Tip  You can use wildcards to capture multiple connection string combinations: Show me the supported wildcards Pattern Description * Matches everything. ? Matches any single character. [seq] Matches any character in "seq". [!seq] Matches any character not in "seq".	No
domainId	The unique resource ID of the domain (or domains), in Collibra Data Intelligence Platform, in which you want to ingest the Power BI assets. Tip You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.	Yes
description	Any description, as you see fit.	Yes
workspaceNames	The names of Power BI workspaces from which you want to ingest metadata. Important Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name "Sale and Marketing [automobiles]" should be formatted as follows: Sale and Marketing [[]automobiles[]]	No
workspaceIds	The IDs of Power BI workspaces from which you want to ingest metadata. Tip We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.	No
capacityNames	The names of capacities on which you want to filter.	No
capacityIds	The IDs of capacities on which you want to filter. Warning Any letters in a capacity ID must be in upper case.	No
excludeWorkspaceNames	The names of Power BI workspaces that you want to exclude from the ingestion job. This is useful if you want to exclude, for example, dedicated development and testing workspaces. Note The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content. For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.	No
excludeWorkspaceIds	The IDs of Power BI workspaces that you want to exclude from the ingestion job. This is useful if you want to exclude, for example, dedicated development and testing workspaces. For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.	No

4. Save the <source ID> configuration file.

When you create technical lineage for Snowflake by using the SQL-API ingestion method, you can create a <source ID> configuration file to configure the metadata that Collibra Data Lineage collects

{
     "displaySampleQueries": true|false
     "analyzeTemporaryTables": true|false
}	

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-adf, the name of your JSON file must be my-adf.conf.
3. For each database in Snowflake, add the following content to the JSON file:

   Property	Description	Required?
   displaySampleQueries	Indicates whether to display transformations with a question mark (?) or with actual values from queries in the Source code pane in the technical lineage graph. For example, you can choose to display WHERE amount < 100 or WHERE amount < ?. Specify one of the following values: true Actual values from queries are displayed. false A question mark (?) is displayed. This is the default value.	No
   analyzeTemporaryTables	Indicates whether to parse the CREATE TEMPORARY TABLE statement in the ingested queries. Specify one of the following values:  true Collibra Data Lineage examines the queries and parses the CREATE TEMPORARY TABLE statement when the following conditions are met: The query starts with the CREATE TEMPORARY TABLE statement. Collibra Data Lineage did not encounter the CREATE TEMPORARY TABLE statement before this query. false Collibra Data Lineage does not examine or parse the CREATE TEMPORARY TABLE statement in the ingested queries. This is the default value.	No

4. Save the <source ID> configuration file.

The lineage harvester uses the lineage harvester configuration file to collect the SQL Server Reporting Services (SSRS) and Power BI Report Server (PBRS) data objects and send them to the Collibra Data Lineage service.

The <source ID> configuration file allows you to:

• If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in SSRS and PBRS.
  
• Provide additional information about databases in SSRS and PBRS, which is necessary if the databases do not contain all information to process the SQL source code correctly.

{
	"DataSources": {
		"Redshift": {
			"dbname": "redshift-database-name",
			"schema": "redshift-schema-name",
			"dialect": "redshift",
			"collibraSystemName": "redshift-system-name"
		},
		"Oracle": {
			"dbname": "oracle-database-name",
			"schema": "oracle-schema-name",
			"dialect": "oracle",
			"collibraSystemName": "oracle-system-name"
		}
	},
	"CustomDataSources": {
		"/path to report/custom data souce name": {
			"dbname": "mssql-database-name",
			"dialect": "mssql"
		}
	}
}	

Steps

1. Create a new JSON file in the lineage harvesterconfig folder.
2. Give the JSON file the same name as the value of the Id property in the lineage harvester configuration file.
   Example The value of the Id property in the lineage harvester configuration file is ssrs-source-1. As a result, the name of your JSON file should be ssrs-source-1.conf.
   
   Important Your JSON file must have the file extension .conf.

3. For each database in SSRS and PBRS, add the following content to the JSON file:

   Property	Description	Required?
   DataSources	This section contains all connections for which you want to create a technical lineage. The DataSources section refers to shared data sources in SSRS and PBRS. For more information about shared data sources, see the Microsoft documentation.	Yes
   <data source type>	The path of a connection object in SSRS and PBRS.	Yes
   dbname	The name of the database of a supported data source in SSRS and PBRS.	No
   schema	The name of the default schema of a supported data source in SSRS and PBRS.	No
   dialect	The dialect of the supported data source in SSRS and PBRS.	No
   collibraSystemName	The system or server name of the database. If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT". How do I configure this property if I have two databases with the same name? Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows. "Redshift": { "dbname": "Customer", "schema": "redshift-schema-name", "dialect": "redshift", "collibraSystemName": "Customers-Europe" }, "Oracle": { "dbname": "Customer", "schema": "oracle-schema-name", "dialect": "oracle", "collibraSystemName": "Customers-USA" }	Yes
   CustomDataSources	You can use custom data processing extensions that are used to support embedded data sources of which the data source definition is specified locally in a report or embedded data set. The CustomDataSources section refers to embedded data sources in SSRS and PBRS. For more information about embedded data sources, see the Microsoft documentation.	No
   <path to report>/<custom data source name>	The full path to the report and the custom data source name. You can use wildcards to match multiple folders, reports or data sets. The connection information is this section is used to add missing information or to overwrite parsed information.	No
   dbname	The name of the database of a custom data source in SSRS and PBRS..	No
   schema	The name of the schema of a custom data source in power. If you don't provide the schema name, the default schema is used.	No
   dialect	The dialect of the custom data source in SSRS and PBRS.. Click for possible values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for a SAP Hana data source. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.	No

4. Save the <source ID> configuration file.

The lineage harvester uses a lineage harvester configuration file to collect the SQL Server Integration Services data objects. It then sends the metadata to the Collibra Data Lineage service instance.

{
  "DataSources": {
    "dhb-sql-prod": {
      "dialect": "mssql",
      "collibraSystemName": "my-system-name"
    }
},
{
  "ConnStringRegExTranslation": {

    "Data Source=dhb-sql-prod;Initial Catalog=SFG_repl_staging;Provider=SQLNCLI11;Integrated Security=SSPI.*": {
      "dbname": "DATAHUB",
      "schema": "DBO",
      "dialect": "mssql",
      "collibraSystemName" : "WAREHOUSE"
    },

    "Server=sb-dhub;User ID=SYS_USER;Initial Catalog=STAGEDB;Port=6306.*": {
      "dbname": "STAGEDB",
      "schema": "STAGE_OWNER",
      "dialect": "sybase",
      "collibraSystemName" : ""
    }

  }
}

Steps

1. Create a new JSON file in the lineage harvester config folder.
2. Name the JSON file as <sourceId>.conf, where <sourceId> is the same as the value of the sourceId property in the lineage harvester configuration file and the file extension must be .conf.
   Example If the value of the sourceId property in the lineage harvester configuration file is my-ssis, the name of your JSON file must be my-ssis.conf.
3. For each database, add the required content to the JSON file.

   Property	Description	Required?
   DataSources	The parent element that contains the connection definitions of your data sources in SQL Server Integration Services. If you specify the properties in this section and also the ConnStringRegExTranslation property for a data source, the connection definitions in the ConnStringRegExTranslation property takes precedence.	No
   DataSourceName	The name of your data source.	No
   dialect	The dialect of the referenced database. See the list of allowed values. You can enter one of the following values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for an SAP HANA data source. hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source. hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source. Important  To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.	No
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.	No
   ConnStringRegExTranslation	The parent element that opens the connection definitions. If you specify this property and also the properties in the DataSources section for a data source, the connection definitions in this property takes precedence.	No
   <regular expression>	A regular expression that must match one or more connection strings. Note  Important considerations: By default, the regular expression is not case sensitive. As a consequence, a regular expression can match with connection strings containing uppercase characters or lowercase characters. The connection string is part of the SSIS connection manager. SSIS connection managers are included in an SSIS package files (DTSX) or in connection manager files (CONMGR). Example  Regular expression: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.* Explanation: The first section, up to .*, is a literal, but not case-sensitive, match of the characters. The dot (.) can match any single character. The asterisk (*) means zero or more of the previous, in this case any character. Match: Any connection string that starts with Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306. Example: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306;Persist Security Info=True;Auto Translate=False;.	No
   dbname	The name of your database, to which the data source connection refers.	No
   schema	The name of your schema, to which the regular expression refers.	No
   dialect	The dialect of the referenced database. See the list of allowed values. You can enter one of the following values: azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. db2, for an IBM DB2 data source. hana, for an SAP HANA data source. hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source. hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source. Important  To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect. hive, for a HiveQL data source. greenplum, for a Greenplum data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. redshift, for an Amazon Redshift data source. snowflake, for a Snowflake data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.	No
   collibraSystemName	The system or server name of the data source. Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.	No

4. Save the <source ID> configuration file.

The lineage harvester uses the configuration file to connect to Tableau. You are not required to create a <source ID> configuration file, but you need one if you want to:

• Define your Tableau operating model.
• Provide additional information about databases and files in Tableau. For example, you can define the system name of files and connectors in Tableau.
• Use the hostnameMapping property to map the database, schema or system names that were returned by the Tableau APIs to the actual names of the assets in Data Catalog. For complete information, go to Tableau hostname, schema, and system name mapping.
  
  Note Mapping doesn't work for custom SQL.
• Define in which domains in Collibra you want to ingest assets from your Tableau sites and projects. See the domainMapping and filters properties.

Tip "<source ID>" refers to the value of the Id property in the lineage harvester configuration file.

{
  "collibraSystemNames": {										
    "files": [
      {"filePath": "C:\\ProgramData\\Tableau\\Tableau Server\\data\\files\\sample.xls",
        "collibraSystemName": "sample-files"
      }
    ],
    "connectors": [
      {
        "connectorUrl": "tableau-server-connector-url.com",
        "collibraSystemName": "Oracle-connector"
      }
    ],
    "cloudFiles": [
      {
        "name": "file-name",
        "collibraSystemName": "FILE"
      }
    ]
},
  "hostnameMapping": {
	"found_dbname=*;found_hostname=abc.net;found_schema=DEFAULT": {
		"dbname": "CData",
		"schema": "Jan_1_2022",
		"dialect": "spark",
		"collibraSystemName": "TV_testing"
	}
},
  "filters": {
	 "sites":{
	   "site_name":"domain_id"
	 },
        "projects":{
          "site_name2 > project_name2": "domain-reference-id2",
          "site_name3 > project_name3 > subproject_name": "domain-reference-id2"
	 }
   }	
}			

{
  "collibraSystemNames": {										
    "files": [
      {"filePath": "C:\\ProgramData\\Tableau\\Tableau Server\\data\\files\\sample.xls",
        "collibraSystemName": "sample-files"
      }
    ],
    "connectors": [
      {
        "connectorUrl": "tableau-server-connector-url.com",
        "collibraSystemName": "Oracle-connector"
      }
    ],
    "cloudFiles": [
      {
        "name": "file-name",
        "collibraSystemName": "FILE"
      }
    ]
  },
  "hostnameMapping": {
	 "found_dbname=Test;found_hostname=*;found_schema=DEFAULT": {
		"dbname": "CData",
		"schema": "Jan_1_2022",
		"dialect": "spark",
		"collibraSystemName": "TV_testing"
	 }
  },
  "domainMapping": {
	 "<Site-1>": "reference-id-of-Domain-1",
	 "<Site-1> > <Project-Default>": "reference-id-of-Domain-2"
  }	
}				

Steps

1. Create a new JSON file in the lineage harvesterconfig folder.
2. Give the JSON file the same name as the value of the Id property in the lineage harvesterconfiguration file.
   Example If the value of the Id property in the lineage harvester configuration file is tableau-source-1, then the name of your JSON file should be tableau-source-1.conf.
   Important Your JSON file must have the file extension .conf.
3. For each database in Tableau, add the following content to the JSON file:
   
   Tip You can use wildcards to capture multiple string combinations for any of these properties.
   

   Show me the supported wildcards

   Pattern	Description
   *	Matches everything.
   ?	Matches any single character.
   [seq]	Matches any character in "seq".
   [!seq]	Matches any character not in "seq".

   Property	Description
   collibraSystemNames	This section contains the system information for different Tableau data sources. Depending on the kind of data source or connection, you have to specify how to connect to this data source. Tip For more information, see the Tableau documentation. We also recommend to check the list of supported connectors in Tableau.
   files	This section contains connection information to one or more files in Tableau. Tip If you do not have files in Tableau, you can remove this section.
   filePath	The full path to the file. For example, the path to a JSON file.
   collibraSystemName	The system name of the file.
   connectors	This section contains connection information to one or more connectors in Tableau. Tip  If you do not have connectors in Tableau, you can remove this section. The values that you specify for this property are not case-sensitive.
   connectorUrl	The URL of the connector. For example, the URL to Google Analytics.
   collibraSystemName	The system name of the connector.
   cloudFiles	This section contains connection information to one or more cloud files in Tableau's input data. Tip If you do not have cloud files in Tableau, you can remove this section.
   name	The name of the file. For example, the name of a Zendesk file.
   collibraSystemName	The system name of the cloud file.
   hostnameMapping	This section allows you to map Tableau technical database, server and schema names to the respective real names, to preserve stitching. Warning  hostnameMapping replaces the following deprecated properties, which have been removed from this topic: The databaseMapping property. The databases sub-section of the collibraSystemNames section. hostnameMapping must not be used in combination with either of these properties. If you use the hostnameMapping section, you can still use the collibraSystemName property in conjunction with the files, connectors or cloudfiles sub-sections. Example configuration "hostnameMapping": { "found_dbname=databasename1;found_hostname=*;found_schema=test": { "dbname": "mssql-database-name", "schema": "mssql-schema-name", "dialect": "mssql", "collibraSystemName": "mssql-system-name" } }
   found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>	The database information of supported data sources in Tableau that is typically collected by the lineage harvester. It allows you to specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema).
   dbname	The name of the database of a supported data source in Tableau.
   schema	The name of the default schema of a supported data source in Tableau. If the lineage harvester fails to find a specific schema, it uses the default schema.
   dialect	The dialect of the supported data source in Tableau. You don't have to specify a dialect; it will automatically be detected. If, however, you are using a dialect that is not supported, you can use this property to specify a supported dialect that is a close comparison. That way, most of your queries will be detected and processed. Show me a list of dialects of supported data sources in Tableau. redshift, for an Amazon Redshift data source. azure, for an Azure SQL Server data source. bigquery, for a Google BigQuery data source. greenplum, for a Greenplum data source. hive, for a HiveQL data source. oracle, for an Oracle data source. postgres, for a PostgreSQL data source. mssql, for a Microsoft SQL Server data source. mysql, for a MySQL data source. netezza, for a Netezza data source. hana, for a SAP HANA data source. spark, for a Spark SQL data source. sybase, for a Sybase data source. teradata, for a Teradata data source.
   filters	This section defines: From which Tableau sites and projects you want to harvest metadata. Into which domains in Collibra you want to ingest the corresponding assets. Filtering is transitive, which means that all resources in a specified project, such as Tableau workbooks and all sub-projects, are ingested. Tableau assets that are not mapped to the specified domains, for example the Tableau Server assets and the parent projects (if you specify their sub-projects), are ingested in the default domain. Important  Filtering does not affect the amount of raw metadata that is harvested from Tableau and sent to the Collibra Data Lineage service instance. Rather, it determines which metadata is ingested as assets in Data Catalog. The domainMapping and filters sections are mutually exclusive. Do not include both domainMapping and filters sections in your JSON file. Tip  If you want to ingest all of the projects in a Tableau site into multiple domains in Collibra, use the domainMapping section. If you want to ingest all of the projects in a Tableau site into the default domain, use only the domainID property in the lineage harvester configuration file. The domainID property represents the default domain. If you want to ingest all of the projects in a Tableau site into a single domain in Collibra, use site filtering. If you want to ingest metadata from only some of the projects in a Tableau site, use project filtering. You can use site filtering and project filtering together: If filtering on the same site, this "filtering" is actually domain mapping, because nothing is filtered out. The contents of the projects are ingested in the specified domains, and the rest of the contents of the site are ingested in a different, specified domain. If you are site filtering on a specific site and project filtering a different site, then site filtering is again a form of domain mapping, and the filtered projects are ingested in their specified domains. If your lineage harvester configuration file includes sites that are not mentioned in the filters section of your <source ID> configuration file, those sites are ingested in the default domain.
   sites	The Tableau sites to be ingested and the domain in which you want to ingest metadata from the Tableau sites. Tip If you have only one Tableau site, do not include a sites section in your <source ID> file. Instead, use a projects section, to filter on Tableau projects. Include a sites section only if all of the following are true: You have more than one Tableau site. You want to ingest all of the metadata from only one Tableau site into a single domain in Collibra. The domain into which you want to ingest is not the default domain, meaning the domain specified in the domainId property in your lineage harvester configuration file.
   site_name: domain_id	site_name The name of the site to be ingested. The site name is case-sensitive. domain_id The unique reference ID of the domain in Collibra in which you want to ingest metadata. The domain ID is case-sensitive. To ingest all metadata from a Tableau site in the specified domain, specify the site name and a separate domain ID for each site that you list on the siteIds property in the lineage harvester configuration file for Tableau. If the site_name or domain_id property is not specified for a site, the metadata from the site is ingested in the default domain. How do I find a domain reference ID? Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold. Show me the example { "filters":{ "sites":{ "Training":"ca60b822-781b-4b3a-b44d-f65bd107ff92" }, "projects":{ "Testing > Databricks":"e8f4e4a8-4062-4a33-9b44-3ce3e18e4e22", "Product Demo > Customer Insights":"a305e6f7-7a49-49aa-aa85-41b1e689121b" } } }
   projects	The Tableau projects to be ingested and the domain in which you want to ingest metadata from the Tableau projects or sub-projects. Tip Project filtering is not relevant for those who have an Explorer role in Tableau, because Explorers need to configure permissions for each data object in Tableau that they want to ingest. As the Administrator role has access to all data objects, project filtering allows Administrators to specify which projects to ingest.
   site_name > project_name : domain_id	The site_name should be the Tableau site name. The project_name should be the Tableau project name. The domain_id should be the unique reference ID of the domain in Collibra in which you want to ingest metadata. When you specify the site and project names, the following rules apply: Add spaces before and after >. The spaces are separators between the site and project. Specify the full exact site and project names. The values are case-sensitive. When you specify a Tableau project, all assets in the project are ingested in the specified domain. If you want to ingest assets from different Tableau projects in one domain, you can specify the same value for domain id for different projects. Example "Collibra_tab_partner_site > JB_Test_2812": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
   site_name > project_name > sub-project_name : domain_id	The site_name should be the Tableau site name. The project_name should be the Tableau project name. Optionally, use sub-project_name to specify the Tableau sub-project name. The domain_id property should be the unique reference ID of the domain in Collibra in which you want to ingest metadata. When you specify the site, project and sub-project names, the following rules apply: Add spaces before and after >. The spaces are separators between the site and project. Specify the full exact site and project names. The values are case-sensitive. Example "Collibra_tab_partner_site > JB_Test_2812 > ProjectJJ2": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
   domainMapping	This section defines in which domains in Collibra you want to ingest assets from your Tableau sites and Tableau projects. Domain mapping is transitive, meaning that all resources, such as Tableau workbooks and data attributes in a parent Tableau site, project or sub-project, are ingested in the same domain as the parent. Important The domainMapping and filters sections are mutually exclusive. Do not include both domainMapping and filters sections in your JSON file. Tip  If you want to ingest all of the projects in a Tableau site into multiple domains in Collibra, use this domainMapping section. If you want to ingest all of the projects in a Tableau site into the default domain, use only the domainID property in the lineage harvester configuration file. The domainID property represents the default domain. Note Tableau assets that are not mapped to specific domains via this domainMapping section, for example Tableau Server assets, are ingested in that default domain. If you want to ingest all of the projects in a Tableau site into a single domain in Collibra, use site filtering. If you want to ingest metadata from only some of the projects in a Tableau site, use project filtering. Show me an example Let's say that you have a Tableau site named "Site-1". You want to ingest all Tableau projects in "Site-1" in a domain named "Domain-1" in Collibra, with the exception of one Tableau project named "Project-Default", which you want to ingest in "Domain-2". You should configure the domainMapping section as follows. "domainMapping": { "<Site-1>": "reference-id-of-Domain-1", "<Site-1> > <Project-Default>": "reference-id-of-Domain-2" } If you want to specify a domain for a sub-project of "Project-Default", use the <site name> > <project name> > <sub-project name> property, as described below. Tip For the properties in this domainMapping section, ensure that you maintain the spaces before and after ">", for example "Site-1 > Project-Default". The spaces serve as a separator between the site and the projects.
   site name	The Tableau site name, followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau site. Important In the configuration file, use the actual site name, along with the domain reference ID, for example: "Collibra_tab_partner_site": "afc8cfb0-91f1-4075-a3e5-7ce6d1f9bcc9"
   site name > project name	The Tableau project name, preceded by the name of the Tableau site to which it belongs, and followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau project. Important In the configuration file, use the actual site and project names, along with the domain reference ID, for example: "Collibra_tab_partner_site > JB_Test_2812": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
   site name > project name > sub-project name	The Tableau sub-project name, preceded by the name of the Tableau site and project to which it belongs, and followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau sub-project. Important In the configuration file, use the actual site, project and sub-project names, along with the domain reference ID, for example: "Collibra_tab_partner_site > JB_Test_2812 > ProjectJJ2": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"

4. Save the <source ID> configuration file.