Databricks Unity Catalog lineage integration preflight checks

To ensure successful metadata ingestion and lineage generation, complete the following preflight checks.

In your Databricks Unity Catalog environment

Important Collibra Data Lineage support for Databricks Unity Catalog leverages the system tables feature in Databricks Unity Catalog. The system tables feature is in Public Preview. For details, go to Databricks Previews support & details in Databricks documentation.

As a Databricks Unity Catalog user, ensure that you have the following privileges and permissions:

  • The CAN ATTACH TO and CAN RESTART permissions for your compute resource.
    You can create a dedicated compute resource for generating technical lineage, or use an existing compute resource with Unity Catalog support. For details, go to Compute permissions in Databricks documentation.
    To prevent clusters from running for the entire synchronization duration, you can also configure the Terminate after ... minutes of inactivity setting in Databricks. The setting ensures that clusters automatically stop after a period of inactivity. For more information, go to the Databricks documentation.

  • The following permissions on the system tables or a custom catalog as a workaround if you prefer not to grant permissions to the system tables:

      • Enable the lineage system tables. To include source code in the technical lineage viewer, enable the system.query.history system table.
        Important 
        • Enable the system.query.history table before adding permissions to the system table.
        • SQL source code is captured and becomes accessible only after the system.query.history table is enabled.
      • Ensure that the personal access token, OAuth client, or Entra ID principal used for the Databricks connection has the following minimum permissions:
        1. USE CATALOG on the system catalog.
        2. USE SCHEMA on the system.access and system.query schemas.
        3. SELECT on the system.access.column_lineage table.
        4. SELECT on the system.access.table_lineage table. This permission is required only for Collibra Data Lineage to ingest lineage that exists only in the system.access.table_lineage table. Ensure that you select the Also ingest lineage from table_lineage option in your capability to enable this functionality.
        5. SELECT on the system.query.history table to ingest SQL source code.

      • If you do not have the right accesses, the Could not get column lineage data error occurs when you synchronize the Technical Lineage for Databricks Unity Catalog capability. Contact Databricks support if you encounter issues on getting access to the system tables. Complete the following steps to verify:

        1. Log into Databricks with the credentials that you use to create a Databricks connection on Edge.
        2. Run the following SQL query against your system.access catalog and schema in Unity Catalog: 
          SELECT count(*) FROM column_lineage
      1. Create a catalog and grant the following privileges: BROWSE, EXECUTE, READ VOLUME, SELECT, USE CATALOG, and USE SCHEMA.

        This example shows a custom catalog named demo with its associated privileges.

      2. Create two schemas in the catalog: access and query.
      3. Grant the following privileges on the access and query schemas: EXECUTE, READ VOLUME, SELECT, and USE SCHEMA.
        Some privileges might be inherited from the catalog privileges granted in step 1.

        This example shows the privileges for the access schema.

        This example shows the privileges for the query schema.

      4. In the access schema in the demo catalog, create the following views:
        • A column_lineage view based on the system.access.column_lineage table.
        • A table_lineage view based on the system.access.table_lineage table.
      5. In the query schema in the demo catalog, create a history view based on the system.query.history table.
      6. Grant the SELECT privilege on the following views created in steps 4 and 5: column_lineage, table_lineage, and history.

In your CPSH environment

Lineage enablement

  • Technical lineage via Edge is enabled in your CPSH environment.
  • You are using Collibra Platform Self-Hosted 2024.02 or newer.
  • Be sure to review the Supported transformation details topic to understand the lineage information Collibra Data Lineage ingests from Databricks Unity Catalog.

Edge

Network and proxy configuration

  • Edge can connect to all Collibra Data Lineage service instances in your geographic location.
  • Optionally, you've connected to a proxy server.
  • Optionally, use a custom certificate to allow the Edge capability to connect to your data source. In this case, you've saved the certificate as "ca.pem" in the same directory as the Edge site installer. If you've saved the certificate in another directory, use the --ca argument in the Edge site installation command.

CPSH permissions

As a technical lineage user, you can connect to Collibra Data Lineage by using the basic or OAuth authentication method. If you use the basic authentication method, ensure you have the Catalog Authorglobal role with the following global permissions. The username you use as the technical lineage user must match the value you entered in the DGC user name field when you enabled technical lineage via Edge.

Note If you use the OAuth authentication method, the following requirements and permissions are not required.
  • Catalog > Advanced Data Type > Add
  • Catalog > Advanced Data Type > Remove
  • Catalog > Advanced Data Type > Update
  • Catalog > Technical lineage

As a Data Catalog user, ensure that your Edge integration engineer global role has the following global permissions. With these permissions, you can create connections and capabilities on Edge, configure the integration, and synchronize the integration.

  • Manage connections and capabilities
  • View Edge connections and capabilities

To add an Edge capability:

To synchronize technical lineage: