Databricks Unity Catalog lineage integration preflight checks

To ensure successful metadata ingestion and lineage generation, complete the following preflight checks.

In your Databricks Unity Catalog environment

Compute permissions

Depending on the compute type, one of the following permission sets for your compute resource:

• Classic compute clusters: CAN ATTACH TO and CAN RESTART.
• SQL warehouses (including serverless SQL warehouses): CAN USE to only integrate source tags. If you also want the integration to stop the compute resource after extraction, the access token or OAuth client needs CAN MANAGE instead.

You can create a dedicated compute resource for generating technical lineage, or use an existing compute resource with Unity Catalog support. For details, go to Compute permissions in Databricks documentation.

To prevent the compute resource from running for the entire duration of the synchronization, you can also configure automatic stop settings in Databricks:

• Compute cluster: Terminate after … minutes of inactivity
• SQL warehouse (including serverless SQL warehouse): Auto Stop

The setting ensures that compute resources automatically stop after a period of inactivity. For more information, go to the Databricks documentation for compute clusters or SQL warehouses.

Unity Catalog permissions

• To automatically populate the Catalog name field, the Databricks access token, OAuth client, or Microsoft Entra ID service principal must have BROWSE or USE CATALOG permission on each Unity Catalog catalog.

• To generate technical lineage, configure access to the required Unity Catalog system tables in one of the following ways:

  • Option 1: Grant access to system tables

    Enable system tables

    Enable the lineage system tables. To include source code in the technical lineage viewer, enable the system.query.history system table.

    Important 

    • Enable the system.query.history table before adding permissions to the system table.
    • SQL source code is captured and becomes accessible only after the system.query.history table is enabled.

    Grant access permissions

    Ensure that the personal access token, OAuth client, or Entra ID principal used for the Databricks connection has the following minimum permissions:

    Permission	When it is required
    USE CATALOG on the system catalog	Required.
    USE SCHEMA on the system.access and system.query schemas	Required.
    SELECT on system.access.column_lineage	Required.
    SELECT on system.query.history Membership in the databricks_pii_access account group	Both are required to ingest SQL source code when you select Include SQL transformations on the synchronization page. Databricks masks query text by default. Without the required membership, synchronization succeeds but SQL source code is not ingested.
    SELECT on system.access.table_lineage	Required only if you select Also ingest lineage from table_lineage in the capability. This option enables Collibra Data Lineage to ingest lineage that exists only in this table.
    SELECT on system.access.workspaces_latest	Required only if you select Resolve notebook links across workspaces on the synchronization page. If this permission isn't available, notebook links fall back to the configured workspace URL. Synchronization isn't affected.

    How to verify whether you have the right accesses in Databricks?

    If you do not have the right accesses, the Could not get column lineage data error occurs when you synchronize the Technical Lineage for Databricks Unity Catalog capability. Contact Databricks support if you encounter issues on getting access to the system tables. Complete the following steps to verify:

    1. Log into Databricks with the credentials that you use to create a Databricks connection on Edge.
    2. Run the following SQL query against your system.access catalog and schema in Unity Catalog:

       SELECT count(*) FROM column_lineage

  • Option 2: Use a custom catalog

    If you prefer not to grant direct access to the system tables, create a custom catalog with views of the required system tables:

    1. Create a catalog and grant the following privileges: BROWSE, EXECUTE, READ VOLUME, SELECT, USE CATALOG, and USE SCHEMA.
       Show an example

       This example shows a custom catalog named demo with its associated privileges.

       
    2. Create two schemas in the catalog: access and query.
       Show an example

       
    3. Grant the following privileges on the access and query schemas: EXECUTE, READ VOLUME, SELECT, and USE SCHEMA.
       Some privileges might be inherited from the catalog privileges granted in step 1.
       Show examples

       This example shows the privileges for the access schema.

       

       This example shows the privileges for the query schema.

       
    4. In the access schema in the demo catalog, create the following views:
       • A column_lineage view based on the system.access.column_lineage table.
       • A table_lineage view based on the system.access.table_lineage table.
       Show an example

       
    5. In the query schema in the demo catalog, create a history view based on the system.query.history table.
       Show an example

       
    6. Grant the SELECT privilege on the following views created in steps 4 and 5: column_lineage, table_lineage, and history.
       Show examples

       

In your Collibra environment

Lineage enablement

• Technical lineage via Edge is enabled in your Collibra environment.
• You are using Collibra Platform 2024.02 or newer.
• Be sure to review the Supported transformation details topic to understand the lineage information Collibra Data Lineage ingests from Databricks Unity Catalog.

Edge

• You either created and installed an Edge site or were granted a Collibra Cloud site.
  
  Important If you're using a Collibra Cloud site, go the Collibra Cloud site documentation to check if your data source is supported.
• The Edge site status must be Healthy.
• You've either integrated Databricks Unity Catalog or registered a Databricks file system.

Network and proxy configuration

• Edge can connect to all Collibra Data Lineage service instances in your geographic location.
• Optionally, you've connected to a proxy server.
• Optionally, use a custom certificate to allow the Edge capability to connect to your data source. In this case, you've saved the certificate as "ca.pem" in the same directory as the Edge site installer. If you've saved the certificate in another directory, use the --ca argument in the Edge site installation command.

Collibra permissions

As a technical lineage user, you can connect to Collibra Data Lineage by using the basic or OAuth authentication method. If you use the basic authentication method, ensure you have the Catalog Authorglobal role with the following global permissions. The username you use as the technical lineage user must match the value you entered in the DGC user name field when you enabled technical lineage via Edge.

• Catalog > Advanced Data Type > Add
• Catalog > Advanced Data Type > Remove
• Catalog > Advanced Data Type > Update
• Catalog > Technical lineage

As a Data Catalog user, ensure that your Edge integration engineer global role has the following global permissions. With these permissions, you can create connections and capabilities on Edge, configure the integration, and synchronize the integration.

• Manage connections and capabilities
• View Edge connections and capabilities

To add an Edge capability:

• You have a global role with the Product Rights > System administration global permission.
• You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

To synchronize technical lineage:

• A global role that has the following global permission:
  • Catalog, for example Catalog Author
  • View Edge connections and capabilities
• A resource role with Configure external system resource permission, for example Owner.