Power BI workspaces
Power BI workspaces represent the most used metadata in Power BI. They contain, for example, reports and data sets. If you want a full ingestion, you have to make sure that the lineage harvester can access all metadata in your Power BI workspaces.
Important considerations
- Depending on the authentication type, you must have specific roles and permissions to access the metadata in the Power BI workspaces.
- You can only fully ingest new Power BI workspaces. This means that classic workspaces and My Workspace in Power BI are not supported.
-
To ingest Power BI dataflows:
- You need access to the Power BI environment in which the data flow is stored.
- The data set in the data flow must exist in a premium workspace.
Filtering Power BI workspaces
By default, the lineage harvester accesses the metadata of all Power BI workspaces. If you don't use filtering, the metadata of all workspaces is uploaded to the Collibra Data Lineage service instance and ingested in Data Catalog. Filtering allows you to process and ingest only the metadata that matters most to you.
You use the Power BI <source ID>_filter configuration file to configure filtering for Power BI workspaces.
Inclusion and exclusion filters
You can use the following inclusion filters to ingest only the Power BI capacities and workspaces you specify:
capacityNames
capacityIds
workspaceNames
workspaceIds
You can use the following exclusion filters to ingest all workspaces except for those you specify:
excludeWorkspaceNames
excludeWorkspaceIds
- Wildcards are supported for the
capacityNames
,workspaceNames
andexcludeWorkspaceNames
properties. - You can combine inclusion and exclusion filters in the same <source ID> configuration file.Show me an example
In this example, the metadata from all workspaces is uploaded to the Collibra Data Lineage service instance. Then, the metadata in all of the workspaces in CapacityABC, except for Workspace1, is ingested in Data Catalog.
{ "filters":[ { "domainId":"07d5d441-b9f8-4add-982f-d7a5d6ba06cc", "description":"Domain for BICatalogJBTest1", "capacityNames":["CapacityABC"], "excludeWorkspaceNames":["Workspace1"] } ] }
- In the Power BI <source ID> configuration file, you can also specify the domain (or domains) in which you want to ingest, to help structure your Power BI assets in Collibra.
Two filtering methods
The filter properties that you use in your Power BI <source ID> configuration file determine whether filtering is done by the lineage harvester or done on the Collibra Data Lineage service instance. The following table highlights the advantages, limitations and configuration considerations of the two methods.
Filtering method | Description |
---|---|
By the lineage harvester |
The lineage harvester accesses only the workspaces specified in your <source ID> configuration file, and sends metadata from only those workspaces to the Collibra Data Lineage service instance for processing and ingestion in Data Catalog. Advantages
Limitations
Show me an example setup for the <source ID> configuration file
{ "filters":[ { "domainId":"b5d02896-8a79-49a3-bab0-12a7b37f45c6", "description":"Any description, for your internal use", "workspaceIds":["ee23f25b-0ed9-490a-9cca-8a0e8886173e", "8e86429d-f985-4a81-818d-8e05ac256a74"] } ] } |
On the Collibra Data Lineage service instance |
The lineage harvester accesses all workspaces and filtering is carried out only after knowing the names and IDs of all workspaces and capacities. As a result, the raw metadata is accessed by the lineage harvester, but only the filtered metadata is processed on the Collibra Data Lineage service instance and ingested in Data Catalog. Advantages
Limitations
Show me an example setup for the <source ID> configuration file
{ "filters":[ { "domainId":"07d5d441-b9f8-4add-982f-d7a5d6ba06cc", "decription":"Domain for BICatalogJBTest1", "capacityNames":["CapacityABC"], "excludeWorkspaceNames":["Workspace1"] } ] } |
Note The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content.
Best practices
You can filter on a capacity to ingest the metadata from all workspaces in that capacity. Let's say, for example, that you have 50,000 workspaces but you only want to ingest metadata from the workspaces related to a specific department in your organization. You could specify each of the relevant workspaces in the configuration file, but that could be tedious if there are lots of workspaces. Furthermore, if someone in your organization creates a new workspace, it will have to be added to your configuration file. Instead, you can filter on a capacity. Then, when a new workspace is created, ensure that it is added to the department' s capacity and metadata from that workspace will be automatically ingested, without having to update the configuration file.
Let’s say that you have three workspaces, each dedicated to a different department in your organization: HR, Finance, and Marketing. You want to ingest the metadata from these three workspaces into three different domains and set the permissions in Collibra so that people can access only the domain for the department to which they belong.
In this case, you can use the workspaceIds
property in your filter configuration, to specify which workspaces are ingested into which domains.
{ "filters":[ { "domainId":"b5d02896-8a79-49a3-bab0-12a7b37f45c6", "description":"<Any description, for the HR department>", "workspaceIds":"<HR's workspace ID>" }, { "domainId":"b5d3h796-wg79-86a3-hab3-e7cdj48r9fk49", "description":"<Any description, for the Finance department>", "workspaceIds":"<Finance's workspace ID>" }, { "domainId":"8sue2896-0s79-93d3-ki90-2fr459jht4s4", "description":"<Any description, for the Marketing department>", "workspaceIds":"<Marketing's workspace ID>" } ] }
workspaceIds
property, to enhance performance and security
If you include only the workspaceIds
property in your filter configuration:
- The lineage harvester (via CLI or Edge) connects only to those workspaces.
- Collibra bypasses the endpoint:
GetModifiedWorkspaces
, which lists all of the workspaces in your tenant.
By specifying only the IDs of the workspaces you want to ingest and bypassing the GetModifiedWorkspaces
endpoint, Collibra is unaware of any other workspaces that exist in your tenant. This makes the harvesting job a lot faster and more secure.
If this method is not preferred in your organization, you can instead use another filter property and include the deleteMetadataAfterProcessing
property in your configuration file (or select the option in your Edge capacity) and set the value to true
. In this case, filtering is done as described in the table above: Filtering on the Collibra Data Lineage service instance:
- All of the raw metadata in the capacity is harvested and sent to the Collibra Data Lineage service instance.
- Only the metadata from the workspaces you want to ingest are ingested in Collibra.
- All metadata is permanently deleted from the Collibra Data Lineage service instance.
This approach, however, does not enhance performance.
Workspace states
On Power BI Workspace asset pages, you can include the attribute type State, to show the state of ingested Power BI workspaces, for example Active, Orphaned or Deleted. To do so, you have to edit the global assignment of the Power BI Workspace asset type and assign the attribute type State.
For complete information on Power BI workspaces and possible states, see the Microsoft Power BI documentation.
- Ensure that the attribute type State is assigned to the Power BI Workspace asset type via the global assignment.
- Go to the Global view, and then create an advance filter and filter by the following clauses:
- Asset type equals Power BI Workspace
- Characteristic State equals Active.
Deleted workspaces
If you delete a Power BI workspace, the workspace is maintained for a 90-day grace period, during which a Power BI administrator can restore the workspace. During the grace period, the workspace has the state Deleted. When you ingest Power BI metadata in Data Catalog, this deleted workspace is ingested.
When the grace period elapses, the state of the workspace becomes Removing, for a short time, while it is being permanently removed. The state then becomes Not found. At this point, as the workspace no longer exists in Power BI, the Power BI Workspace asset in Collibra will also be deleted upon the next synchronization.
If a workspace becomes inactive, meaning no reports or dashboards have been viewed in the past 60 days, it is excluded from the ingestion.
Why are deleted workspaces ingested?
Let's image that you ingest a Power BI workspace with the Active state and that over time, you add comments, tags and characteristics to the asset in Collibra. Now let's imagine that the workspace is deleted in Power BI and we do not ingest the deleted workspace. In this case, the Power BI Workspace asset in Collibra is deleted upon the next synchronization. But what if the Power BI administrator decides, during the 90-day grace period, to restore the workspace in Power BI? Upon the next synchronization, a new Power BI Workspace asset is created in Collibra, but all of the comments, tags and characteristics that were part of the deleted asset are lost.
By ingesting deleted Power BI workspaces, we safeguard against losing any of the additional information on the Power BI Workspace asset, in case a Power BI administrator decides to restore a workspace during the grace period.