Prepare the lineage harvester configuration file

Before you can visualize the technical lineage or ingest a BI source, you have to create a configuration file for the (meta)data sources that you want to process. This configuration file is used by the lineage harvester to extract data from (meta)data sources for which you want to create a technical lineage or you want to ingest.

Note

Technical lineage only supports a limited list of (meta)data sources.
You can only use UTF-8 or ISO-8859-1 characters in all lineage harvester files.
Each data source has an ID property. The ID string must be unique and human readable. The ID can be anything and is only used to identify the batch of metadata that is processed on the Collibra Data Lineage server.
The lineage harvester connects to different servers based on your geographical location and cloud provider. Make sure you have the correct system requirements before you run the lineage harvester. If your location or cloud provider changes, the lineage harvester rescans all your data sources.
Technical lineage supports the following means of authentication:
- For all data sources, except for external directories: username and password.
- Tableau: username and password or token-based authentication.
- Google BigQuery data sources: username and password or a service account key file. For more information, see the Google BigQuery documentation.
- No other authentication methods are supported.
The lineage harvester does not support proxy server authentication, but you can manually connect to a proxy server via command line. For more information, see Connecting to a proxy server.
Comments in the lineage harvester configuration file are not supported.
If you upgrade to lineage harvester 1.3.0 or newer, you have to follow an upgrade procedure.

Tip For complete information on ingesting metadata from the following BI tools and creating a technical lineage, see the dedicated sections:

Prerequisites

You have prepared the physical data layer in Data Catalog.
You have a global role that has the System administration global permission.
You have a global role that has the Manage all resources global permission.
You have a global role with the Technical lineage global permission.
You have a global role with the Data Stewardship Manager global permission.
You have downloaded the lineage harvester and you have the necessary system requirements to run it.
Java Runtime Environment version 11 or newer or OpenJDK 11 or newer.
You have added Firewall rules so that the lineage harvester can connect to:
- All Collibra Data Lineage servers within your geographical location:
- The host names of all databases in the lineage harvester configuration file.
If you want to use a previously loaded data source, you have downloaded the SQL files of the data source to the lineage harvester.
If you want to use an external directory, you have prepared a folder with data objects from the external directory.

You have the necessary permissions to all database objects that the lineage harvester accesses.

Tip

Some data sources require specific permissions.

You need read access on the SYS schema.

You need read access on the SYS schema and the View Definition Permission in your SQL Server.

You need read access on information_schema.

You need read access on information_schema. Only views that you own are processed.

SELECT, at table level. Grant this to every table for which you want to create a technical lineage.

A role with the LOGIN option.

Only SQL statements.

CONNECT ON DATABASE

You need a role that can access the Snowflake shared read-only database. To access the shared database, the account administrator must grant IMPORTED PRIVILEGES on the shared database to the user that runs the lineage harvester.

Tip If the role is not assigned in Snowflake, you can use the customConnectionProperties property in the lineage harvester configuration file to assign the Default role to the user. For example:
"customConnectionProperties": "role=default"

You need read access on the DBC.

You need read access to the following dictionary views:

all_tab_cols
all_col_comments
all_objects
ALL_DB_LINKS
all_mviews
all_source
all_synonyms
all_views

You need read access on definition_schema.

You need Admin permission on all objects that you want to harvest.

You have added the Matillion certificate to a Java truststore.

You have at least a Matillion Enterprise license.

You need a role with user access to the server from which you want to ingest:

You have a system-level role, which is at least a System user role.
You have an item-level role, which is at least a Content Manager role.

You need a role with user access to the relevant server and be able to access the metadata that is stored there.

Make sure that the lineage harvester can reach Power BI by registering Power BI in Azure and setting the necessary permission to harvest the metadata.

We highly recommend that you read about supported authentication methods before you register Power BI in Microsoft Azure. For more details, see Register Power BI in Microsoft Azure and set permissions.

You need to following minimum roles and permissions to harvest Tableau metadata:

You have a View permission on Tableau projects, workbooks and data sources you want to ingest.
You have a Viewer or Explorer (can publish) role with access to the Tableau REST API.

For a full ingestion, we recommend the following roles and permissions in Tableau:

You have at least a View permission on Tableau projects, workbooks and data sources you want to ingest.
You have the Explorer role with the Data Management Add-on.

Steps

Run the following command line to start the lineage harvester:
- Windows: .\bin\lineage-harvester.bat
- For other operating systems: chmod +x bin/lineage-harvester and then bin/lineage-harvester
An empty configuration file is created in the config folder.

Open the configuration file and enter the values for each property.

Tip You can use the configuration file generator to create an example configuration file with the properties of your choosing. You can easily copy this example to your configuration file and replace the values of the properties to match your data source information.

Properties

Description

general

This section describes the connection between Collibra lineage and Data Catalog.

catalog

This section contains information that is necessary to connect to Data Catalog.

Note Versions of the lineage harvester older than 1.1.2 show collibra instead of catalog.

url

The URL of your Collibra environment.

Note You can only enter the public URL of your Collibra environment. Other URLs will not be accepted.

username

The username that you use to sign in to Collibra.

useCollibraSystemName

Indication whether you want to use the system or server name of a data source to match to the System asset you created when you prepared the physical data layer. This is useful when you have multiple databases with the same name.

By default, the useCollibraSystemName property is set to false.

If you keep the useCollibraSystemName property set to false, the lineage harvester ignores the collibraSystemName property in the rest of the configuration file.
If you set the useCollibraSystemName property to true, the lineage harvester reads the value in the collibraSystemName property in all sections of the configuration file. It also reads the collibraSystemName property in the following files:
- The Informatica <source ID> configuration file
  Important You must prepare a <source ID> configuration file regardless of whether the useCollibraSystemName property in your lineage harvester configuration files is set to true or false.
- The IBM DataStage or SQL Server Integration Services connection definition configuration files.
- The Informatica Intelligent Cloud Services <source ID> configuration file.
  Important You must prepare a <source ID> configuration file regardless of whether the useCollibraSystemName property in your lineage harvester configuration files is set to true or false.
- The Power BI <source ID> configuration file.
- The JSON files with a predefined lineage.

Note For SQL data sources, if the useCollibraSystemName property is:

false, system or server names in table references in analyzed SQL code are ignored. This means that a table that exists in two different systems or servers is identified (either correctly or incorrectly) as a single data object, with a single asset full name.
true, system or server names in table references are considered to be represented by different System assets in Data Catalog. The value of the collibraSystemName property is used as the default system or server name.

By default, the useCollibraSystemName property is set to false. This property is not valid for Looker integration. We recommend that you leave this property set to false.

Indicates whether or not you intend to use a Power BI <source-ID> configuration file to specify the system or server name of a data source to match to the System asset in Data Catalog during automatic stitching. This is useful when you have multiple databases with the same name.

By default, the useCollibraSystemName property is set to false.

If you set this property to true, you must prepare a Power BI <source ID> configuration file.

Indicates whether or not you intend to use a Tableau <source ID> configuration file to specify the system or server name of a data source, to match to the System asset you created when you prepared the physical data layer. This is useful when you have multiple databases with the same name.

By default, the useCollibraSystemName property is set to false.

If you set this property to true, you must prepare a Tableau <source ID> configuration file.

Indicates whether or no you intend to use a SQL Server Reporting Services and Power BI Report Server <source ID> configuration file, to specify the system or server name of a data source. This is useful when you have multiple databases with the same name.

By default, the useCollibraSystemName property is set to false.

If you set this property to true, you must prepare a SQL Server Reporting Services and Power BI Report Server <source ID> configuration file.

sources

This section describes the data sources for which you want to create the technical lineage. You have to create a configuration section for each data source.

Note You can add multiple data sources to the same configuration file.

This configuration section contains the required information of one individual SQL directory with connection type "Folder".

The unique ID of the data source. For example, my_first_data_source.

type

The kind of data source. In this case, the value has to be SqlDirectory.

path

The full path to the SQL directory.

mask

The pattern of the file names in the directory. By default, this is *.

recursive

Indication of the files you want to harvest:

false (default): Only harvest the files in directly under the folder in the SQL directory path.
true: Harvest all files under the folder in the SQL directory path and subdirectories.

dialect

The dialect of the database.

database

The name of your database, which is the full name of your Database asset.

Note You have to use the same database name as the full name of the Database asset that you create when you prepare the physical data layer in Data Catalog.

Important

HiveQL, MySQL and Teradata data sources don't have schemas. Therefore, HiveQL, MySQL and Teradata databases are stored in Data Catalog and technical lineage as Schema assets. The technical lineage Browse tab pane shows the following names:

For HiveQL and Teradata:
- The database name is the name that you enter for the collibraSystemName property.
- The schema name is the name that you enter for the database property.
For MySQL:
- The database name is the name that you enter for the database property.

externalDbName

This property can be considered a means of database mapping, to help preserve stitching. It is relevant only for HiveQL, MySQL and Teradata data sources, specifically because they are database-less data sources.

You can add the key/value pair to the configuration file as follows: "externalDbName": "CDATA"

collibraSystemName

The name of the data source's system or server. This is also the full name of your System asset in Data Catalog.

You must use the same system name as the full name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

schema

The name of the default schema, if not specified in the data source itself. This corresponds to name of your Schema asset.

Note You must use the same schema name as the name of the Schema asset that you create when you prepare the physical data layer in Data Catalog.

verbose

Indication whether you want to enable verbose logging.

By default this is set to True. If you don't want to use verbose logging, set it to False.

This configuration section contains the required information to connect to the following data sources:

Informatica PowerCenter
SQL Server Integration Services (SSIS).
IBM InsfoSphere DataStage

Note Make sure that you have prepared a local folder with the Informatica objects, SSIS files or DataStage files for which you want to create a technical lineage.

collibraSystemName

The name of the data source's system or server. If the useCollibraSystemName property is set to true, you must prepare a configuration file to provide the system information.

The unique ID of your data source. For example, my_informatica.

type

The kind of data source. In this case, the value has to be ExternalDirectory.

dirType

The type of external directory. The value has to be one of the following:

infa, for an Informatica PowerCenter data source.
ssis, for a SQL Server Integration Service data source.
datastage, for a IBM InfoSphere DataStage source.

path

The full path to the folder where you stored the data source.

mask

The pattern of the file names in the directory. By default, this is *.

recursive

Indication whether you want to use recursive queries.

By default, this is set to False. If you want to use recursive query, set it to True.

This configuration section contains the required information to enable the lineage harvester to collect and process Data Integration objects.

Tip Make sure you have READ permission on all data objects that you want to harvest.

type

The kind of data source. In this case, the value has to be IICS.

The unique ID that is used to identify the data source on the Collibra Data Lineage server. For example, my_data_integration.

collibraSystemName

The name of the Informatica server or system.

Important You must prepare a <source ID> configuration file to provide this system information. This is true regardless of whether the useCollibraSystemName property is set to true or false.

loginURL

The URL of the Informatica Intelligent Cloud Services environment sign-in page. For example: https://dm-us.informaticaintelligentcloud.com.

username

The username you use to sign in to Informatica Intelligent Cloud Services.

objects

The objects that you want to export. Each object requires a path and a type, for example:

"objects": [
	{
		"path" : "Sales",
		"type" : "Project"
	}, 
	{
		"path" : "Finance/Task_Flows",
		"type" : "Folder"
	},
	{
		"path" : "Common/Task_Flows/tf_CalendarDimension",
		"type" : "Taskflow"
	}
]

The following section provides information to identify and access Data Integration objects.

Tip For more information about the objects that you can export and the required information, see the Informatica documentation.

path

The full path to the object.

type

The type of the object. For example, Taskflow.

IICS scanner's starting point is a Taskflow. Therefore the only meaningful types to export are: Taskflow, Project and Folder.

Note The types are not case sensitive.

paramFiles

The full path to the directory in which your parameter files are stored.

This is an optional parameter that allows you to harvest parameter files in Informatica Intelligent Cloud Services data sources.

Important The hierarchy of the files in the directory must be an exact match of the hierarchy of the files in your file system.

This section contains the required information for Matillion.

Tip When you create a new project in Matillion, you define in which group you want to create the project, the project name and the environment name. This information is needed to enable the lineage harvester to access Matillion and scan your metadata.

Important Currently, you can only create a technical lineage for Snowflake and Redshift projects in Matillion.

The unique ID that is used to identify the data source on the Collibra Data Lineage server. For example, my_matillion_data_integration.

type

The kind of data source. In this case, the value has to be Matillion.

url

The URL of your Matillion environment. For example, https://<domain name> or https://<IP address>.

groupName

The name of your group in Matillion.

projectName

The name of your project in Matillion.

You can only add the name of one project. If you want to create a technical lineage for other projects within the same group, create a new section in the lineage harvester configuration file.

environmentName

The name of your environment in Matillion.

You can only add the name of one environment. If you want to create a technical lineage for other environments within the same project, create a new section in the lineage harvester configuration file.

dialect

The dialect of the database.

You can enter one of the following values:

redshift, for an Amazon Redshift data source.
snowflake, for a Snowflake data source.

startTimestamp

The timestamp of tasks in Matillion. You can use this parameter to limit the amount of metadata that the lineage harvester scans.

If the startTimestamp field remains empty or is deleted from the configuration file, all accessible tasks are scanned.

Matillion automatically removes entries older than seven days.

collibraSystemName

The name of the Matillion system or server.

auth

The section contains the authentication details for signing in to Matillion.

type

The authentication method you want to use to sign in to Matillion.

The value must be either:

Basic, for username and password authentication.
Token, for token-based authentication.

Important These values are case-sensitive.

username

The username that you use to sign in to Matillion.

Important This property is only required if you are using the username and password authentication method. If you are using token-based authentication, do not include this property.

This section contains the required information to connect to a custom lineage. You create a custom lineage by adding connection properties to a JSON file containing a predefined technical lineage.

Make sure that you have prepared a local folder with the JSON file that contains the predefined technical lineage.

Note In the local folder that you need to create, you can only have one JSON file. You can, however, add other files in the harvested directory and subdirectories and refer to those files from within the JSON file.

The unique ID of your custom technical lineage. For example, MyCustomLineage.

type

The kind of data source. In this case, the value has to be ExternalDirectory.

dirType

The type of external directory. In this case, the value is custom-lineage.

path

The full path to the folder where you stored the data source or JSON file.

This configuration section contains the required information of one individual data source with connection type "JDBC".

The unique ID of your data source. For example, my_second_data_source.

type

The kind of data source. In this case, the value has to be Database.

username

The username that you use to sign in to your data source.

dialect

The dialect of the database.

databaseNames

The names or IDs of your databases.

Enter the database names of your data source between double quotes ("") and put everything between square brackets. If you want to include more than one database, separate them by a comma. For example, ["MyFirstDatabase", "MySecondDatabase"].

Note You have to use the same database names as the full names of the Database assets that you create when you prepare the physical data layer in Data Catalog.

Important

For HiveQL and Teradata:
- The database name is the name that you enter for the collibraSystemName property.
- The schema name is the name that you enter for the database property.
For MySQL:
- The database name is the name that you enter for the database property.

externalDbName

You can add the key/value pair to the configuration file as follows: "externalDbName": "CDATA"

connectAsServiceName

The option to determine whether your Oracle database uses an Oracle service name or SID.

True: Connect to an Oracle database that uses an Oracle service name. Enter the service name in the databaseNames property.
False: Connect to an Oracle database that uses an SID. Enter the SID in the databaseNames property.

Note This property is only valid for Oracle databases. It will be ignored for all other databases.

hostname

The name of your database host.

collibraSystemName

The name of the data source's system or server. This is also the full name of your System asset in Data Catalog.

If the useCollibraSystemName property is:

false (default), system or server names in table references in analyzed SQL code are ignored. This means that a table that exists in two different systems or servers is identified (either correctly or incorrectly) as a single data object, with a single asset full name.
true, system or server names in table references are considered to be represented by different System assets in Data Catalog. The value of the collibraSystemName field is used as the default system or server name.

port

The port number.

customConnectionProperties

An option to enable the lineage harvester to read additional connection parameters. This parameter is only required in very specific situations. If you don't need it, you can remove it from the configuration file.

Note You cannot currently use this property for Oracle data sources.

This configuration section contains the required information for a Google BigQuery database.

The unique ID of your data source. For example, my_third_data_source.

type

The kind of data source. In this case, the value has to be DatabaseBigQuery.

projectIDs

The IDs of your Google BigQuery project. You can add multiple projects. For example, [ "first-project", "second-project", "third-project" ].

Note You have to use the same project ID as the full name of the Database asset that you create when you prepare the physical data layer in Data Catalog.

region

The location of your BigQuery data. This is the region that you specified when you create a data set.

You can only add one location as value. However, you can create separate BigQuery entries per location in the configuration file. As a result, you create a complete technical lineage with Google BigQuery data from different locations.

Note This property is optional.

auth

The path to a JSON file that contains authentication information.

Tip For more information about setting up the authentication, see the Google Big Query user guide.

collibraSystemName

The name of the Google BigQuery system. This is also the full name of your System asset in Data Catalog.

This configuration section contains the required information for a Snowflake database.

The unique ID of your data source. For example, my_fourth_data_source.

type

The kind of data source. In this case, the value has to be DatabaseSnowflake.

username

The username that you use to sign in to your data source.

hostname

The URL that you use to access Snowflake web console. For example, <AccountName>.snowflakecomputing.com.

collibraSystemName

The name of the Snowflake system. This is also the full name of your System asset in Data Catalog.

databaseNames

The names of your databases.

Note You have to use the same database names as the full names of the Database assets that you create when you prepare the physical data layer in Data Catalog.

warehouse

The name of your virtual warehouse.

Note This property is optional.

customConnectionProperties

Example If you get an OSCP scan error, you can turn OSCP checking off by using the following value: insecureMode=true.

This configuration section contains the required information for SQL files of a data source that were previously downloaded by the lineage harvester and is stored in the lineage harvester output folder.

type

The kind of data source. In this case, the value has to be LoadedSource.

The unique ID of the data source that you uploaded to the lineage harvester folder. For example, my_loaded_snowflake_source.

zipFile

The full path to the ZIP file that was created in the lineage harvester folder.

This configuration section contains the required information for Tableau integration.

sources

This section contains all Tableau connection properties.

type

The kind of data source. In this case, the value has to be Tableau.

The unique ID to identify the Tableau metadata that was uploaded to the Collibra Data Lineage.

Tip This value can be anything as long as it is a unique. The lineage harvester uses the ID to identify a batch of data on the Collibra Data Lineage server.

url

The link to the data in Tableau.

username

The username you use to sign in to the Tableau server.

Important If you want to use token-based authentication, you need to replace username with tokenName. You must specify either username or tokenName; if both exist, then tokenName is used.

tokenName

The lineage harvester authentication token.

Note For token-based authentication, use this property in your lineage harvester configuration file, instead of the username property. If both properties are present, tokenName is used.

siteIds

The site IDs of the Tableau sites that you want to include in the ingestion process.

Warning Ensure that you specify the correct value. The correct value is the URL of the site to which you want to sign in. When you manually sign in to Tableau Server or Tableau Online, the site ID is the value that appears after /site/ in the browser address bar. In the following example URLs, the site ID is MarketingTeam:

Tableau Server: http://MyServer/#/site/MarketingTeam/projects
Tableau Online: https://10ay.online.tableau.com/#/site/MarketingTeam/workbooks

On Tableau Server, however, the URL of the Default site does not specify the site. For example, the URL for a view named Profits, on a site named Sales, is http://localhost/#/site/sales/views/profits. The URL for this same view on the Default site is http://localhost/#/views/profits. The site name Sales does not figure in the URL. If you can't see the site ID, leave this property empty: "siteIds": [""]

Example If you want to ingest two Tableau sites "Site 1" and "Site 2", you can enter the following information in the siteIds property: ["site ID of Site 1", "site ID of Site 2"].

siteNames

The site names of the corresponding site IDs.

Important This property is:

Optional for Tableau Server
Mandatory for Tableau Online.

Warning If you have Tableau Server and you don't use this property, you must delete it from your configuration file. Don't leave the property in the configuration file without a value.

restOnly

Indication whether or not you would like to use both the Tableau REST API and Tableau Metadata API to harvest Tableau metadata.

false (default): The lineage harvester will use the REST API and Metadata API to harvest Tableau metadata.
true: The lineage harvester will only use the REST API to harvest Tableau metadata.

Warning If you only allow the lineage harvester to use the Tableau REST API, the harvester won't be able to process the necessary information for the technical lineage and the automatic stitching of Column assets to Tableau Data Attribute assets will not be possible.

collibraSystemName

Regardless of the value set for the useCollibraSystemName property, the following is true:

You must include this property in your configuration file.
You can leave this property empty.
Any value that you give is ignored.

If the useCollibraSystemName property is set to true, you must prepare a Tableau <source ID> configuration file. In that case, the CollibraSystemName property in the <source ID> configuration file is taken into account.

Note This is a legacy property that will be deprecated in a future release.

domainId

The unique reference ID of the domain in Collibra Data Intelligence Cloud in which you want to ingest the Tableau assets.

Tip You can ingest Tableau assets in one or more domains in Collibra. The following table identifies which properties and which configuration files to use, depending on whether you want to ingest in one or multiple domains.

If you want to... Then...

Ingest in a single domain in Collibra

Refer to the single domain reference ID in this domainID property.

Ingest in multiple domains in Collibra

Do both of the following:

Mention a domain reference ID in this domainID property, for your Tableau Server asset.
Refer to all relevant domain reference IDs in the domainMapping section of the Tableau <source ID> configuration file, for your Tableau site, Tableau project and all child assets.

Important The domainID property represents the default domain. Tableau assets that are not mapped to specific domains via the domainMapping section of the Tableau <source ID> configuration file, for example Tableau Server assets, are ingested in this default domain.

excludeImages

Optional parameter for excluding the downloading of images.

To exclude the downloading of images, set this property to true.

paging

Optional parameter for customizing the Tableau API pagination settings.
The default values are sufficient in most cases; however, you can decrease them to help mitigate node limit errors, or increase them to speed up API calls.

The complete list of pagination settings, descriptions and default values

"paging": {
	"databasesPageSize": 100,
	"tablesPageSize": 100,
	"tablesColumnsPageSize": 100,
	"tableColumnsPageSize": 1000,
	"datasourcesPageSize": 50,
	"datasourcesFieldsPageSize": 50,
	"datasourceFieldsPageSize": 100,
	"worksheetsPageSize": 100,
	"worksheetsFieldsPageSize": 100,
	"worksheetFieldsPageSize": 1000,
	"dashboardsPageSize": 100,
	"columnsLimit": 20,
	"fieldsLimit": 20
	}

Settings per metadata type and descriptions

Metadata type	Setting and description
Dashboard	`dashboardsPageSize`: The number of dashboards per page.
Worksheet	`worksheetsPageSize`: The number of worksheets per page. `worksheetsFieldsPageSize`: The number of worksheet fields per page.
Database	`databasesPageSize`: The number of databases per page.
Table	`tablesPageSize`: The number of tables per page. `tablesColumnsPageSize`: The number of table columns per page.
Table columns	`tableColumnsPageSize`: The number of table columns per page.
Data source	`datasourcesPageSize`: The number of data sources per page. `datasourcesFieldsPageSize`: The number of data source fields per page. `columnsLimit`: The number of data source field columns per page. `fieldsLimit` : The number of referenced data source fields per page.
Data source field	`datasourceFieldsPageSize`: The number of data source fields per page. `columnsLimit`: The number of data source field columns per page. `fieldsLimit` : The number of referenced data source fields per page.

This configuration section contains the required information for Power BI integration.

Note You have to purchase the Power BI connector and lineage feature. Then you need to add the Power BI connection properties to both the lineage harvester configuration file and the Power BI harvester configuration file to ingest Power BI metadata into Data Catalog.

type

The kind of data source. In this case, the value has to be ExistingLineage.

The unique ID of the Power BI metadata you harvested via the Power BI harvester.

You must use the same ID as the value you used in the Power BI configuration file sourceID property.

This configuration section contains the required information for Looker integration.

collibraSystemName

The name of the Looker system or server. If the useCollibraSystemName property is set to true, you must prepare a Looker <source ID> configuration file to provide the system information.

The unique ID of your Looker metadata. For example, my_looker.

Tip This value can be anything as long as it is unique and human readable. The ID identifies the batch of Looker metadata on the Collibra Data Lineage server.

type

The kind of data source. In this case, the value has to be Looker.

lookerUrl

The URL to your Looker API.

Tip There are two ways to find the Looker API URL:

In the API Host URL field in the Looker Admin menu. If this field is empty, you can use the default Looker API URL which you can find in the interactive API documentation.
In the interactive API documentation URL. It is the part of the URL before /api-docs/.

clientId

The username you use to access the Looker API.

domainId

The unique ID of the domain in Collibra Data Intelligence Cloud in which you want to ingest the Looker assets.

This configuration section contains the required information for MicroStrategy integration.

type

The kind of data source. In this case, the value has to be MicroStrategy.

collibraSystemName

This property is deprecated for MicroStrategy integration. The lineage harvester does not take into account any value that you enter here.

The unique ID of your MicroStrategy metadata. For example, my_microstrategy.

Tip This value can be anything as long as it is unique and human readable. The ID identifies the batch of MicroStrategy metadata on the Collibra Data Lineage server.

domainId

The unique reference ID of the domain in Collibra Data Intelligence Cloud in which you want to ingest the MicroStrategy assets.

username

The username that you use to sign in to MicroStrategy.

hostname

The endpoint that you use to access the PostgreSQL repository or remote data source, depending on where you installed the lineage harvester.

For example remote.postgres.com.

port

The port number.

databaseName

Optionally, the name of your database. For example poc_metadata.

This configuration section contains the required information for SQL Server Reporting Services and Power BI Report Server integration.

collibraSystemName

Regardless of the value set for the useCollibraSystemName property, the following is true:

You must include this property in your configuration file.
You can leave this property empty.
Any value that you give is ignored.

If the useCollibraSystemName property is set to true, you must prepare a SQL Server Reporting Services and Power BI Report Server <source ID> configuration file. In that case, the CollibraSystemName property in the <source ID> configuration file is taken into account.

Note This is a legacy property that will be deprecated in a future release.

The unique ID to identify the SSRSmetadata that was uploaded to the Collibra Data Lineage server.

Tip This value can be anything as long as it is a unique. The lineage harvester uses the ID to identify a batch of data on the Collibra Data Lineage server.

type

The kind of data source. In this case, the value has to be SSRS or PBIRS.

Note There is no difference between type SSRS or PBIRS.

url

The URL to the server's web portal. By default, the URL is http://<computer-name>/reports. For example, "http://1.23.45.678/PowerBIReports".

username

The username you use to sign in to the web portal.

Tip If you use NTLM authentication, your username also contains the NTLM domain name. For example MyDomain\\username.

domainId

The unique ID of the domaindomain in Collibra Data Intelligence Cloud in which you want to ingest the SSRS assets.

folderFilter

An option to exclude specific folders that contain reports or KPIs from the ingestion process.

You can add multiple folders by listing folder names, providing the full path to folders or by using a wildcard:

Use folder names when the folder name is unique: ["folder 1", "folder 2"]
Use the full path to the folder to only ingest a specific folder: ["/database1/folder1", "/database2/folder2"]
Use a wildcard to ingest all child folders or a specific folder: ["/folder1/*", "/folder2/*"]

You can also use a combination of these methods. For example, ["folder 1", "/database/folder2", /folder3/*"]

Important This property must be included in your configuration file and it cannot be empty. If you want to ingest all folders, use *, for example: "folderFilter":["*"].

Tip For more information about connecting to a SSRS or PBRS folder, see the Microsoft documentation.

This configuration section contains the required information for Power BI (NEW) integration.

type

The kind of data source. In this case, the value has to be PowerBI.

The unique ID to identify the Power BI service metadata that was uploaded to the Collibra Data Lineage server.

tenantDomain

The Power BI tenant domain is the domain associated with the Microsoft Azure tenant.

This domain is either a default domain or a custom domain. For example, collibrapowerbi.onmicrosoft.com.

Note Usually, you can find a list of Power BI tenant or server domains in your Azure Active Directory or in the top right menu.

loginFlow

This section describes the authentication information for accessing your Power BI metadata.

The lineage harvester supports two authentication methods: service principal, and username and password. For complete information on your authentication options, see Authentication.

type

This depends on the authentication method you use.

Service principle: The value should be ServicePrincipal.
Username and password: The value should be ResourceOwnerPasswordCredentials.

applicationId

The unique ID of the Microsoft Azure Application (client) ID.

username

The email address of your Azure Active Directory user.

Tip This property only applies if you are using the username and password authentication method.

domainId

The reference ID of the domain in Collibra in which you want to ingest Power BI metadata.

collibraSystemName

Regardless of the value set for the useCollibraSystemName property, the following is true:

You must include this property in your configuration file.
You can leave this property empty.
Any value that you give is ignored.

If the useCollibraSystemName property is set to true, you must prepare a Power BI <source-ID> configuration file. In that case, the CollibraSystemName property in the <source ID> configuration file is taken into account.

Note This is a legacy property that will be deprecated in a future release.

Save the configuration file.
Start the lineage harvester again and do one of the following:
- To process data from all data sources in the configuration file, run the following command:
  For windows:
```
.\bin\lineage-harvester.bat full-sync
```
  For other operating systems:
```
./bin/lineage-harvester full-sync
```
- To process data from specific data sources in the configuration file, run the following command:
  For windows:
```
.\bin\lineage-harvester.bat full-sync -s "ID of the data source"
```
  For other operating systems:
```
./bin/lineage-harvester full-sync -s "ID of the data source"
```

When prompted, enter the passwords to connect to Collibra and your data sources. Do one of the following:
- Enter the passwords in the console.
  The passwords are encrypted and stored in /config/pwd.conf.
- Provide the passwords via command line.
  The passwords are stored locally and not in your lineage harvester folder.

Tip If the lineage harvester log shows an error message or the harvesting process fails, you can use the technical lineage troubleshooting guide to fix your issue.

What's next?

If you prepared the physical data layer and have the required permissions, you can go to the asset page of a Table, Column Power BI Column or Looker Look asset from the data source that you added in the configuration file and visualize the technical lineage. The technical lineage shows the data source information of data sources that have been successfully analyzed and processed.

The lineage harvester can also use scheduled jobs to synchronize the data sources on fixed times.

Tip You can check the progress of the technical lineage creation in Activities. The Results field indicates how many relations were imported into Data Catalog. Go to the status page to see the log files of the SQL analysis.