Power BI source configuration

The Source configuration field in the Power BI technical lineage Edge capability allows you to:

  • Map the names of the server, database, and schema that were collected by Collibra Data Lineage to their true names.
    Note Mapping doesn't work for custom SQL.
  • Configure filtering. Before configuring your filters, go to Power BI filtering for important information and guidance.
  • If you set the Collibra system name setting to True when you enable technical lineage, use the collibraSystemName property to specify the system name of databases in Power BI. Collibra Data Lineage uses the system names to match the structure of databases in Power BI to the correct System assets in Data Catalog.

This topic provides the source configuration properties and syntax examples for filtering and Legacy filtering. If you want to migrate from Legacy filtering to filtering, use the Legacy filtering converter.

Source configuration example with filtering

The value of the Source configuration field must be a valid block of JSON code, for example:

Copy 
{
  "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
    "dbname": "mssql-database-name",
    "schema": "mssql-schema-name",
    "dialect": "mssql",
    "collibraSystemName": "mssql-system-name"
  },
  "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
    "dbname": "oracle-database-name",
    "schema": "oracle-schema-name",
    "dialect": "oracle",
    "collibraSystemName": "oracle-system-name"
  },
  "enforceRemoteFiltering": true,
  "filters":[
    {
    "domainId": "default",
    "description": "Filter by display name",
    "workspaceFilter": {
      "includedIds": ["workspace-id-1", "workspace-id-2"]
    },
    "dashboardFilter": {
      "excludedNames": "*restricted*"
    },
    "reportFilter": {
      "excludedNames": ["report1", "report2", "report3"],
      "includedInApp": true
    }
  }
  ]
}

Source configuration properties

The following table describes the various properties you can use in your JSON code block.

Property

 Description

Mandatory?

found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>

The database information of supported data sources in Power BI that is typically collected by the lineage harvester. Specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema). You then use the child properties to map the names collected by the lineage harvester to the true names.

Important The keys that you specify must be unique. The following configuration would result in an error, because the key found_dbname=databasename1;found_hostname=*;found_schema=schema1 is specified twice.
{
  "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
    "dbname": "mssql-database-name",
    "schema": "mssql-schema-name",
    "dialect": "mssql",
    "collibraSystemName": "mssql-system-name"
  },
  "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
    "dbname": "oracle-database-name",
    "schema": "oracle-schema-name",
    "dialect": "oracle",
    "collibraSystemName": "oracle-system-name"
  }
}
During metadata analysis, if Collibra Data Lineage cannot match a name that you provide in this mapping – let's say, for example, you mistype the name of the database – an analyze error is produced.
  1. In the technical lineage viewer, go to the Setting tab pane, and then click Show status.
  2. In the Sources tab, select the relevant Power BI data source.
  3. In the Full-text search field, enter "source configuration".
  4. In the Filter by drop-down list, select ANALYZE_ERROR.
    The analyze error is shown in the filter results.

You can use wildcards to capture multiple connection string combinations:

Pattern Description
* Matches everything.
? Matches any single character.
[seq] Matches any character in "seq".
[!seq] Matches any character not in "seq".

No
dbname
 The true name (display name) of the database collected by the lineage harvester.

No
schema

The true name (display name) of the schema collected by the lineage harvester.

If the lineage harvester fails to find a specific schema, it uses the schema you specify in this property.

Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements.

No
dialect

The dialect of the supported data source in Power BI.

  • azure, for an Azure SQL Server data source.
  • bigquery, for a Google BigQuery data source.
  • db2, for a Db2 data source.
  • hana, for a SAP HANA data source.
  • hive, for a Hive data source.
  • impala, for a Impala data source.
  • mssql, for a Microsoft SQL Server data source.
  • mysql, for a MySQL data source.
  • netezza, for a Netezza data source.
  • oracle, for an Oracle data source.
  • postgres, for a PostgreSQL data source.
  • redshift, for an Amazon Redshift data source.
  • snowflake, for a Snowflake data source.
  • sybase, for a Sybase data source.
  • teradata, for a Teradata data source.

No
collibraSystemName

The system or server name of a database.

Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

How to configure this property if you have two databases with the same name

Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
  "dbname": "Customers",
  "schema": "mssql-schema-name",
  "dialect": "mssql",
  "collibraSystemName": "Customers-Europe"
},
"found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
  "dbname": "Customers",
  "schema": "oracle-schema-name",
  "dialect": "oracle",
  "collibraSystemName": "Customers-USA"
},

Yes
enforceRemoteFiltering

When set to true, workspaceFilter.includedIds is always applied before metadata collection, even if other filter properties are present in the same filter block. This forces workspace filtering to be performed remotely.

When set to false, the filtering behavior is determined automatically based on the filter configuration.

Use this property when you want to ensure that only specific workspaces are retrieved while still applying additional filters, for example, report or capacity filters.

The default value is false.

For more information, go to Where filtering is carried out.

  

filters

This section allows you to specify the Power BI workspaces from which you want to ingest metadata.

If you specify a capacity, all of the workspaces in that capacity are also ingested.

Workspace filtering takes precedence over capacity filtering, meaning workspaces are filtered first. If there is no explicit exclusion of capacities containing workspaces, all capacities containing workspaces are ingested. Filtering of reports and dashboards is subordinate to workspace filtering, meaning that to include reports and dashboards from a certain workspace, that workspace has be ingested as well. Reports and dashboards from a single workspace cannot be ingested in different domains. Any configured dashboard and report filtering is then taken into consideration.

Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name Sale and Marketing [automobiles] must be formatted as follows:
Sale and Marketing [[]automobiles[]]

Important If you don't want to specify the Power BI workspaces from which to ingest, you must completely remove this filters section.

You can use wildcards to capture multiple connection string combinations:

Pattern Description
* Matches everything.
? Matches any single character.
[seq] Matches any character in "seq".
[!seq] Matches any character not in "seq".

No
domainId

The unique resource ID of the domain (or domains), in Collibra Platform, in which you want to ingest the Power BI assets.

To move existing BI assets to a different domain, update this property and the Domain ID field in the technical lineage capability, and then synchronize the capability again. For more information, go to Move BI assets to a different domain.

You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.

Yes
description

Any description, as you see fit.

No
capacityFilter

This section allows you to specify the capacities from which you want to ingest metadata. You can include certain capacities and exclude others.

  
includedNames
 The names of the capacities from which you want to ingest metadata. No
includedIds
 The IDs of the capacities from which you want to ingest metadata. No
excludedNames
 The names of the capacities that you want to exclude from metadata ingestion. No
excludedIds
 The IDs of the capacities that you want to exclude from metadata ingestion. No
workspaceFilter

This section allows you to specify the workspaces from which you want to ingest metadata. You can include certain workspaces and exclude others.

Before configuring your filters, go to Power BI filtering for important information and guidance.

No
includedNames
 The names of the workspaces from which you want to ingest metadata. No
includedIds
 The IDs of the workspaces from which you want to ingest metadata. No
excludedNames

The names of the workspaces that you want to exclude from metadata ingestion.

This is useful if you want to exclude, for example, dedicated development and testing workspaces.

The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content.

 No
excludedIds

The IDs of the workspaces that you want to exclude from metadata ingestion.

 No
dashboardFilter

This section allows you to specify the dashboards from which you want to ingest metadata. You can include certain dashboards and exclude others.

 No
includedNames
 The names of the dashboards from which you want to ingest metadata. No
includedIds
 The IDs of the dashboards from which you want to ingest metadata. No
excludedNames
 The names of the dashboards that you want to exclude from metadata ingestion. No
excludedIds
 The IDs of the dashboards that you want to exclude from metadata ingestion. No
reportFilter

This section allows you to specify the reports from which you want to ingest metadata. You can include certain reports and exclude others.

 No
includedNames
 The names of the reports from which you want to ingest metadata. No
includedIds
 The IDs of the reports from which you want to ingest metadata. No
excludedNames
 The names of the reports that you want to exclude from metadata ingestion. No
excludedIds
 The IDs of the reports that you want to exclude from metadata ingestion. No
createAppReports

Use this keyword to specify that you don't want to ingest the in-app versions of reports.

If "createAppReports": false, in-app versions of reports are not ingested. Only the original reports are ingested.

If "createAppReports": true, in-app versions of reports are ingested, along with the original reports.

If you don't use the createAppReports keyword, in-app versions of reports are ingested, along with the original reports.

  
includedInApp

Use this keyword to specify how you want Collibra Data Lineage to address reports that are included in published Power BI apps.

If "includedInApp": true:

  • Original reports that are included in an app are ingested.
  • In-app versions of reports are ingested, unless "createAppReports": false.
  • Reports that are not included in an app are not ingested.

If "includedInApp": false, only reports that are not included in an app are ingested.

If you don't use the includedInApp keyword, all reports are ingested, including:

  • Reports that are included in apps, along with their in-app versions.
  • Reports that are not included in an app.

How this property works in conjunction with the "createAppReports" property

Let's say that you have 8 reports in Power BI:

  • 5 reports are not included in an app.
  • 3 reports are included in a app, which means there are also 3 in-app versions of these reports.

The following table shows which of these reports are ingested, based on how you use the 2 keywords.

  "createAppReports": true (or not used) "createAppReports": false
includedInApp is not used

11 reports are ingested:

  • The 5 reports not included in an app.
  • The 3 original reports included in an app.
  • The 3 in-app versions.

8 reports are ingested:

  • The 5 reports not included in an app.
  • The 3 original reports included in an app:

The 3 in-app versions of these reports are not created or ingested.
"includedInApp": true

6 reports are ingested:

  • The 3 original reports included in an app.
  • The 3 in-app versions.

3 reports are ingested:

  • The 3 original reports included in an app.

The 3 in-app versions of these reports are not created or ingested.
"includedInApp": false

5 reports are ingested:

  • The 5 reports not included in an app.

5 reports are ingested:

  • The 5 reports not included in an app.

No

Examples

In the following example:

  • Only reports with names that match ABC report* and are in workspace ABC1 are included.
  • Reports that are not in workspace ABC1 are not included.
  • Reports that are in capacity ABC Capacity are not included.

Copy 
{
  "domainId": "12g6d0dc-8291-476a-9bb0-9b13g6cc1356",
  "description": "Filter by display name",
  "capacityFilter": {
    "excludedNames": ["ABC Capacity"]
  },
  "workspaceFilter": {
     "includedNames": ["ABC1"]
  },
  "reportFilter": {
    "includedNames": ["ABC report*"]
  }
}

In the following example, reports with names that match ABC report*, in any workspace, are included.

Copy 
{
  "domainId": "12g6d0dc-8291-476a-9bb0-9b13g6cc1356",
  "description": "Filter by display name",
  "reportFilter": {
    "includedNames": ["ABC report*"]
  }
}

For report filtering, inclusion and exclusion filters used in combination with the includedInApp property are applied using the AND logical operator. In the following example:

  • In-app report named report1 is included.
  • Let's say that a report named report2 is not in an app. That report is not included.
Copy 
"reportFilter": {
    "includedNames": ["report1", "report2"],
    "includedInApp": true
  }

In the following example, all reports with names that match report1* are included, with the exception of report report1_backup.

Copy 
{
  "filters": [
    {
      "domainId": "12g6d0dc-8291-476a-9bb0-9b13g6cc1356",
      "description": "Some description",
      "reportFilter": {
        "includedNames": "report1*",
        "excludedNames": "*_backup"
      }
    }
  ]
}

In the following example, all reports named report1 in workspace workspace_name_1 (only) are included.

Copy 
{
  "filters": [
    {
      "domainId": "12g6d0dc-8291-476a-9bb0-9b13g6cc1356",
      "description": "description",
      "workspaceFilter": {
        "includedNames": "workspace_name_1"
      },
      "reportFilter": {
        "includedNames": "report1"
      }
    }
  ]
}

In the following example, all workspaces in capacity capacity1 and workspace workspace_name_1 are included.

Copy 
{
  "filters": [
    {
      "capacityFilter": {
        "includedNames": "capacity1"
      },
      "workspaceFilter": {
        "includedNames": "workspace_name_1"
      },
      "description": "workspace and capacity filter",
      "domainId": "12g6d0dc-8291-476a-9bb0-9b13g6cc1356"
    }
  ]
}

Source configuration with Legacy filtering

The value of the Source configuration field must be a valid block of JSON code, for example:

Copy 
{
    "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
        "dbname": "mssql-database-name",
        "schema": "mssql-schema-name",
        "dialect": "mssql",
        "collibraSystemName": "mssql-system-name"
    },
    "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
        "dbname": "oracle-database-name",
        "schema": "oracle-schema-name",
        "dialect": "oracle",
        "collibraSystemName": "oracle-system-name"
    },
    "filters":[ 
        {
        "domainId": "<domain-ref-id>",
        "description": "FirstFilter",
        "workspaceNames": ["*"],
        "excludeWorkspaceIds": ["workspaceC", "workspaceD"]
        },
        {
        "domainId": "<domain-ref-id>",
        "description": "SecondFilter",
        "workspaceNames": ["workspace3", "workspace4"],
        "capacityIds": ["id1","id2"]
        }
    ]
}

The following table describes the various properties you can use in your JSON code block.

Property

 Description

Mandatory?

found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>

The database information of supported data sources in Power BI that is typically collected by the lineage harvester. Specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema). You then use the child properties to map the names collected by the lineage harvester to the true names.

Important The keys that you specify must be unique. The following configuration would result in an error, because the key found_dbname=databasename1;found_hostname=*;found_schema=schema1 is specified twice.
{
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "mssql-database-name",
		"schema": "mssql-schema-name",
		"dialect": "mssql",
		"collibraSystemName": "mssql-system-name"
	},
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "oracle-database-name",
		"schema": "oracle-schema-name",
		"dialect": "oracle",
		"collibraSystemName": "oracle-system-name"
	}
}
During metadata analysis, if Collibra Data Lineage cannot match a name that you provide in this mapping – let's say, for example, you mistype the name of the database – an analyze error is produced.
  1. In the technical lineage viewer, go to the Setting tab pane, and then click Show status.
  2. In the Sources tab, select the relevant Power BI data source.
  3. In the Transformations code search field, enter "source configuration".
  4. In the Filter by drop-down list, select ANALYZE_ERROR.
    The analyze error is shown in the filter results.

You can use wildcards to capture multiple connection string combinations:

Pattern Description
* Matches everything.
? Matches any single character.
[seq] Matches any character in "seq".
[!seq] Matches any character not in "seq".

No
dbname
 The true name (display name) of the database collected by the lineage harvester.

No
schema

The true name (display name) of the schema collected by the lineage harvester.

If the lineage harvester fails to find a specific schema, it uses the schema you specify in this property.

Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements.

No
dialect

The dialect of the supported data source in Power BI.

  • azure, for an Azure SQL Server data source.
  • bigquery, for a Google BigQuery data source.
  • db2, for a Db2 data source.
  • hana, for a SAP HANA data source.
  • hive, for a Hive data source.
  • impala, for a Impala data source.
  • mssql, for a Microsoft SQL Server data source.
  • mysql, for a MySQL data source.
  • netezza, for a Netezza data source.
  • oracle, for an Oracle data source.
  • postgres, for a PostgreSQL data source.
  • redshift, for an Amazon Redshift data source.
  • snowflake, for a Snowflake data source.
  • sybase, for a Sybase data source.
  • teradata, for a Teradata data source.

No
collibraSystemName

The system or server name of a database.

Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

How to configure this property if you have two databases with the same name

Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
	"dbname": "Customers",
	"schema": "mssql-schema-name",
	"dialect": "mssql",
	"collibraSystemName": "Customers-Europe"
},
"found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
	"dbname": "Customers",
	"schema": "oracle-schema-name",
	"dialect": "oracle",
	"collibraSystemName": "Customers-USA"
},

Yes

filters

This section allows you to specify the Power BI workspaces from which you want to ingest metadata.

If you specify a capacity, all of the workspaces in that capacity are also ingested.

Workspace filtering takes precedence over capacity filtering, meaning workspaces are filtered first. If there is no explicit exclusion of capacities containing workspaces, all capacities containing workspaces are ingested. Filtering of reports and dashboards is subordinate to workspace filtering, meaning that to include reports and dashboards from a certain workspace, that workspace has be ingested as well. Reports and dashboards from a single workspace cannot be ingested in different domains. Any configured dashboard and report filtering is then taken into consideration.

Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name Sale and Marketing [automobiles] must be formatted as follows:
Sale and Marketing [[]automobiles[]]

Important If you don't want to specify the Power BI workspaces from which to ingest, you must completely remove this filters section.

You can use wildcards to capture multiple connection string combinations:

Pattern Description
* Matches everything.
? Matches any single character.
[seq] Matches any character in "seq".
[!seq] Matches any character not in "seq".

No
domainId

The unique resource ID of the domain (or domains), in Collibra Platform, in which you want to ingest the Power BI assets.

You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.

Yes
description

Any description, as you see fit.

No
workspaceNames

The names of Power BI workspaces from which you want to ingest metadata.

Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name "Sale and Marketing [automobiles]" should be formatted as follows:
Sale and Marketing [[]automobiles[]]

No
workspaceIds

The IDs of Power BI workspaces from which you want to ingest metadata.

Before configuring your filters, go to Power BI filtering for important information and guidance.

 No
capacityNames

The names of capacities on which you want to filter.

 No
capacityIds

The IDs of capacities on which you want to filter.

Important All letters in a capacity ID must be in upper case.

 No
excludeWorkspaceNames

The names of Power BI workspaces that you want to exclude from the ingestion job.

This is useful if you want to exclude, for example, dedicated development and testing workspaces.

The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content.

For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

 No
excludeWorkspaceIds

The IDs of Power BI workspaces that you want to exclude from the ingestion job.

This is useful if you want to exclude, for example, dedicated development and testing workspaces.

For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

 No