Add the Power BI technical lineage capability

After you have created a connection to Power BI in your Edge or Collibra Cloud site, you have to add the Power BI technical lineage capability to the connection.

Required permissions

You have a global role with the Product Rights > System administration global permission.
You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

Steps

Open a site.
1. On the main toolbar, click → Settings.
  The Settings page opens.
2. In the tab pane, click Edge.
  The Sites tab opens and shows a table with an overview of your sites.
3. In the table, click the name of the site whose status is Healthy.
  The site page opens.
In the Capabilities section, click Add capability.
The Add capability page appears.
Select the Technical Lineage for Power BI capability template.

Enter the required information.

Field Description Required

Name

The name of the capability.

Yes

Description

The description of the capability.

Source ID

The name of the data source. The name must be unique and cannot contain special characters, for example, /.

Warning

You can only specify one source ID per Power BI service. Ingesting the same Power BI service under different source IDs will fail.
Any single Power BI service can be ingested only once. If you create more than one connection for the same Power BI service, integration will fail. If you want to ingest from multiple unique Power BI services, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

Yes

TechLin Admin Connection (in preview)

If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

Power BI Connection

The Power BI connection that you created for ingestion in Data Catalog.

Tip Select the name that you provided in the Name field when you created a connection to Power BI.

Yes

API URL

The API URL of your Power BI service.

The default value is https://api.powerbi.com.

Important This property is only relevant for US government or national cloud Power BI customers, in which case you must include and specify values for both this property and the scope property. For complete information, consult Microsoft's documentation on Power BI for US government customers.

Scope

Optional property that is intended only for customers with a different scope, such as Chinese tenants.

Example https://analysis.chinacloudapi.cn/powerbi/api/.default

Important If you are a US government or national cloud Power BI customer, you must include and specify values for both this property and the apiUrl property. For complete information, consult Microsoft's documentation on Power BI for US government customers.

Domain ID

The unique reference ID of the domain in Collibra Platform in which you want to ingest the Power BI assets.

Yes

Source Configuration

This field allows you to provide JSON code for database mapping, workspace filtering and specifying the name of a System asset in Collibra.

Map the names of the server, database and schema that were collected by the lineage harvester to their true names.
Note Mapping doesn't work for custom SQL.
Configure filtering. We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.
If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in Power BI. Collibra Data Lineage uses the system names to match the structure of databases in Power BI to assets in Data Catalog.

If you previously integrated Power BI via the lineage harvester, you can copy and paste in this field the JSON code from your Power BI <source ID> configuration file.

Connection definition properties

Property

Description

Mandatory?

found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>

The database information of supported data sources in Power BI that is typically collected by the lineage harvester. Specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema). You then use the child properties to map the names collected by the lineage harvester to the true names.

Important The keys that you specify must be unique. The following configuration would result in an error, because the key found_dbname=databasename1;found_hostname=*;found_schema=schema1 is specified twice.

{
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "mssql-database-name",
		"schema": "mssql-schema-name",
		"dialect": "mssql",
		"collibraSystemName": "mssql-system-name"
	},
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "oracle-database-name",
		"schema": "oracle-schema-name",
		"dialect": "oracle",
		"collibraSystemName": "oracle-system-name"
	}
}

During metadata analysis, if Collibra Data Lineage cannot match a name that you provide in this mapping – let's say, for example, you mistype the name of the database – an analyze error is produced.

You can use wildcards to capture multiple connection string combinations:

Pattern	Description
*	Matches everything.
?	Matches any single character.
[seq]	Matches any character in "seq".
[!seq]	Matches any character not in "seq".

dbname

The true name (display name) of the database collected by the lineage harvester.

schema

The true name (display name) of the schema collected by the lineage harvester.

If the lineage harvester fails to find a specific schema, it uses the schema you specify in this property.

Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements.

dialect

The dialect of the supported data source in Power BI.

collibraSystemName

The system or server name of a database.

Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

How to configure this property if you have two databases with the same name

Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
	"dbname": "Customers",
	"schema": "mssql-schema-name",
	"dialect": "mssql",
	"collibraSystemName": "Customers-Europe"
},
"found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
	"dbname": "Customers",
	"schema": "oracle-schema-name",
	"dialect": "oracle",
	"collibraSystemName": "Customers-USA"
},

Yes

filters

This section allows you to specify the Power BI workspaces from which you want to ingest metadata.

If you specify a capacity, all of the workspaces in that capacity are also ingested.

Workspace filtering takes precedence over capacity filtering, meaning workspaces are filtered first. If there is no explicit exclusion of capacities containing workspaces, all capacities containing workspaces are ingested. Filtering of reports and dashboards is subordinate to workspace filtering, meaning that to include reports and dashboards from a certain workspace, that workspace has be ingested as well. Reports and dashboards from a single workspace cannot be ingested in different domains. Any configured dashboard and report filtering is then taken into consideration.

Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name Sale and Marketing [automobiles] must be formatted as follows:
Sale and Marketing [[]automobiles[]]

Important If you don't want to specify the Power BI workspaces from which to ingest, you must completely remove this filters section.

You can use wildcards to capture multiple connection string combinations:

Pattern	Description
*	Matches everything.
?	Matches any single character.
[seq]	Matches any character in "seq".
[!seq]	Matches any character not in "seq".

domainId

The unique resource ID of the domain (or domains), in Collibra Platform, in which you want to ingest the Power BI assets.

You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.

Yes

description

Any description, as you see fit.

workspaceNames

The names of Power BI workspaces from which you want to ingest metadata.

Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name "Sale and Marketing [automobiles]" should be formatted as follows:
Sale and Marketing [[]automobiles[]]

workspaceIds

The IDs of Power BI workspaces from which you want to ingest metadata.

We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.

capacityNames

The names of capacities on which you want to filter.

capacityIds

The IDs of capacities on which you want to filter.

Important All letters in a capacity ID must be in upper case.

excludeWorkspaceNames

The names of Power BI workspaces that you want to exclude from the ingestion job.

This is useful if you want to exclude, for example, dedicated development and testing workspaces.

The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content.

For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

excludeWorkspaceIds

The IDs of Power BI workspaces that you want to exclude from the ingestion job.

This is useful if you want to exclude, for example, dedicated development and testing workspaces.

For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

Example

Show me an example <source ID> configuration file

Copy

{
    "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
        "dbname": "mssql-database-name",
        "schema": "mssql-schema-name",
        "dialect": "mssql",
        "collibraSystemName": "mssql-system-name"
    },
    "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
        "dbname": "oracle-database-name",
        "schema": "oracle-schema-name",
        "dialect": "oracle",
        "collibraSystemName": "oracle-system-name"
    },
    "filters":[ 
        {
        "domainId": "<domain-ref-id>",
        "description": "FirstFilter",
        "workspaceNames": ["*"],
        "excludeWorkspaceIds": ["workspaceC", "workspaceD"]
        },
        {
        "domainId": "<domain-ref-id>",
        "description": "SecondFilter",
        "workspaceNames": ["workspace3", "workspace4"],
        "capacityIds": ["id1","id2"]
        }
    ]
}

Property

This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

You can use this field to set the HTTP timeout duration by adding the httpTimeout property:

Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:

Properties for Collibra Platform for Government customers

Type Value Type Name Value

Text

Plaintext

techlinHost

This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

Example: techlin-europe-west1.collibra.com

Text

Secret

techlinKey

This is the unique API key to connect to a Collibra Data Lineage service instance.

Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

<your-techlin-key>

Yes for US government customers.

Dependent On Sources

This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

To use this option, enter the source ID of the independent source.

Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:

An analyze error is raised, prompting you to provide the DDL file.
The only workaround is to consolidate your SQL statements and DDL file in a single data source.

For complete information, go to Sharing database models across data sources.

Delete Raw Metadata After Processing

Metadata is harvested and uploaded in a ZIP file to a Collibra Data Lineage service instance, for processing.

Use this optional property to specify whether or not the raw metadata should be deleted after it has been processed.

If you select this option, the raw metadata is deleted after processing. If you don't select this option, it is stored in an Amazon S3 bucket.

Analyze Only (Deprecated)

Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.

The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

Processing Level

Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
Metadata from data sources for which the value of this setting is Sync, is synchronized.

Value Description

Load

Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

When the job is done, you can download and review the metadata:

Open the Activities list.
In the row containing the job, click Result.
The Synchronization Results dialog box appears.
Click download and save the ZIP file to your hard drive.

Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

Analyze

Load and analyze the metadata on the Collibra Data Lineage service instance.

Synchronization does not start after analysis; it starts only after either:

You trigger synchronization of another data source for which you specify Sync in the Processing Level drop-down list.
You configure the Technical Lineage Admin Edge or Collibra Cloud site capability, and trigger synchronization via the Sync option in the Integration Configuration tab in Data Catalog.

Important If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.

For complete information and important considerations, go to Tips for successful lineage synchronization
For more information about the Sync option in the Technical Lineage Admin Edge or Collibra Cloud site capability, go to Technical lineage admin options.

Sync

Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

Yes

Active

The option determines whether to include or remove the technical lineage of the data source.

Select this option to include the technical lineage of this data source.

Clear the checkbox to exclude the technical lineage of this data source.

Use HTTP/1.1 protocol

Option to use HTTP/1.1 streams, in case file-size limitations are resulting in timeout errors when using the default HTTP/2 streams.

Enable lineage for DAX queries

Important This is a cloud-only feature. It is not supported for Collibra Platform for Government or Collibra Platform Self-Hosted (CPSH) environments.

Option to enable DAX analysis via Collibra AI. This feature:

Creates column-level lineage that includes your calculated columns and measures in Power BI.
Enables stitching between calculated columns in the technical lineage and the corresponding Power BI Column assets in Data Catalog.

Select this option to enable DAX analysis.

Clear the checkbox to disable DAX analysis.

For complete information on this feature, go to DAX analysis via Collibra AI.

Enable Attribute Level Lineage for Reports (Preview)

Option to enable attribute-level (field-level) lineage for Power BI. Enabling this feature will increase the processing time.

For more information, go to Attribute-level lineage for Power BI (in preview) .

Debug

This setting is not valid for this integration. It should be set to false.

Log level

Only complete this field on the request of or together with Collibra Support.

Click Create.
The capability is added to the Edge or Collibra Cloud site.
The fields become read-only.

What's next

You can now synchronize the technical lineage.