Databricks Unity Catalog: Supported transformation details

Collibra Data Lineage retrieves lineage information from Databricks Unity Catalog system tables to provide column-level visibility into data transformations and lineage across assets in the Databricks workspace.

Function scope

The integration captures lineage for the following Databricks assets:

• Catalog hierarchy
  Databases, Schemas, Tables, and Columns.
• Metric views
  

  Lineage for metric views is generated at the table level for data that flows into the metric view, and at the column level for data that flows from the metric view to downstream assets. Column-level lineage from source columns to a metric view isn't available, because Databricks doesn't record this information in its lineage system tables.

  Table-level lineage into a metric view is only visible when View Type is set to Objects in the technical lineage display options.

  If the source tables and the metric view are in different catalogs, make sure that the include filter covers all of them. Otherwise, the lineage between the source tables and the metric view isn't ingested.

• Delta Live Tables (DLT)
  Technical lineage for Streaming Tables and Materialized Views at both table and column levels.
• External delta tables
  Lineage is generated for external Delta tables referenced by external paths when direct queries are used.
  
  Example 

  If the following SQL is used in Databricks Unity Catalog, lineage will be created in Collibra.
  CREATE OR REPLACE TABLE table_from_direct_delta_query AS (SELECT * FROM delta.`s3://kktesting/testfolder`)

• Volumes (in preview)
  Collects lineage information from volumes. Only lineage relationships are ingested; volume assets are not created in Data Catalog. Automatic stitching for volumes is planned for a future release.
• Notebooks
  Captures lineage relationships for notebooks, including identifiers and direct URLs to link to sources in Databricks. Notebook content and full metadata are not ingested into Data Catalog.

Some assets require specific settings for lineage ingestion. For details, go to Synchronization configuration.

Lineage extraction mechanism

Collibra Data Lineage uses a system-driven approach to lineage extraction rather than manual code parsing:

• Lineage is extracted from the system.access.column_lineage table.
  Because Collibra relies on these internal Databricks records, all languages supported by Unity Catalog (including SQL, Python, R, and Scala) are supported.
  For examples of how Unity Catalog captures and presents data lineage, go to Capture and view data lineage with Unity Catalog in the Databricks documentation.
• Transformation details and source code are extracted and displayed in the technical lineage viewer for notebooks, jobs, SQL queries, and dashboards.
  To extract SQL source code, the system.query.history table must be enabled in the Databricks environment.
• Collibra Data Lineage ingests cumulative lineage.
  Collibra Data Lineage extracts column lineage from the system.access.column_lineage table in Databricks Unity Catalog. The system.access.column_lineage table records lineage over time. This ensures the lineage graph reflects transformations that occur over a defined time window, not just the most recent execution.
• For environments with large volumes of lineage events, retrieving and processing the metadata can take a significant amount of time. Collibra Data Lineage provides different strategies for reading and processing this data. For details, go to Synchronize Databricks Unity Catalog lineage.

Synchronization configuration

To ensure a complete lineage representation, verify that the following settings are configured.

Feature	Requirement
Metric views	Select the Also ingest lineage from table_lineage option when you add the technical lineage for Databricks Unity Catalog capability
Notebooks	Select the Include notebooks option on the synchronization page.
SQL transformation code	The system.query.history table must be enabled in Databricks and the Include SQL transformations option must be selected on the synchronization page when you synchronize your technical lineage. You can configure the SQL statement types for which transformation logic is retrieved. This setting does not affect which lineage relations are created; it only determines which relations include transformation logic. For details, go to Synchronize Databricks Unity Catalog lineage.
Volumes	Select the Include volumes (In preview) option on the synchronization page.