Synchronize Databricks Unity Catalog lineage

You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

If your technical lineage capability includes multiple databases, you don't need to synchronize each database separately. A single synchronization job harvests metadata from all included databases.

Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

Steps

  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. In the tab bar, click Integrations.
    The Integrations page opens.
  3. Click the Integration Configuration tab.
  4. Locate the Databricks Workspace connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the Databricks Workspace connection, expand them to locate your technical lineage capability.
    The technical lineage capability configuration page opens.
  5. In the Synchronization Configuration section, click the Edit icon.
  6. Complete the fields as needed.
    FieldAction
    Options updated: <timestamp>

    Options updated: <timestamp> next to Synchronization configuration indicates the last time the data was retrieved.

    A scheduled refresh job is automatically created when you complete the capability configuration. You can also manually refresh the metadata by clicking Options updated: <timestamp>. The scheduled refresh job appears on the Jobs dashboard of your Edge site with the same name as your lineage capability.

    If two jobs with the same name appear on the Jobs dashboard, one is the scheduled refresh job and the other is the lineage synchronization job. By default, the scheduled refresh job runs daily at 23:00 server time. Unlike the lineage synchronization job, the scheduled refresh job appears only on the Jobs dashboard and not in the Activities list.

    This action prepopulates the Catalog name field.

    SystemSelect the System asset in which the Databricks assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
    Catalog name

    The field is prepopulated with the catalog names available in your Databricks workspace. Loading the catalog names into the dropdown list can take some time.

    If the catalog names can't be retrieved due to insufficient permissions, you can enter the catalog name manually. This doesn't affect lineage extraction.

    If you want to create technical lineage using views in a custom catalog instead of system tables, enter the custom catalog name in this field.
    For more information about the custom catalog, go to Databricks Unity Catalog lineage integration preflight checks.

    Include filterTo include specific workspaces, catalogs, or schemas in technical lineage, click Add Include pattern under Include filter. If you do not specify an include filter, technical lineage includes all lineage from Databricks Unity Catalog.

    The following rules apply when you enter the include pattern:

    • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
    • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
    • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
    • Lineage is collected only when both the source and the target are in the schema, catalog, or workspace specified in the include filter.
    • * > HR: Includes all schemas under the HR catalog.
    • * > Orders > fk*: Includes schemas starting with fk under the Orders catalog.
    • * > * > profiling: Includes all schemas named profiling under any catalog.
    Exclude filterTo exclude certain workspaces, catalogs, or schemas from technical lineage, click Add Exclude pattern under Exclude filter. If you do not specify an exclude filter, technical lineage includes all lineage from Databricks Unity Catalog.

    The following rules apply when you enter the exclude pattern:

    • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
    • The exclude filter takes precedence over the include filter.
    • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
    • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
    • * > testDB: Excludes all schemas under the testDB catalog.
    SQL sources limit

    Specify the limit on the number of SQL statements included for each relation in the technical lineage graph. The default value is 5.

    Include SQL transformations

    Select this option to enable Collibra Data Lineage to extract transformation logic from notebooks, jobs, SQL queries, and dashboards, and include it in the technical lineage viewer. You can view the transformation logic in the Source code pane of the technical lineage viewer.

    Clear the checkbox if you do not want Collibra Data Lineage to ingest transformation logic.

    Note If the personal access token, OAuth client, or Entra ID principal used for the Databricks Workspace connection does not have SELECT permission on the system.query.history table, you must clear this checkbox. Otherwise, connection errors might occur.
    Include external locations

    Select this option for Collibra Data Lineage to collect lineage information from the external locations to create end-to-end lineage.

    Clear the checkbox to exclude metadata from external locations.

    Include volumes (In preview)

    Select this option to collect lineage information from volumes to create end-to-end lineage. Only lineage relationships are collected; volume assets are not created in Data Catalog.

    Clear the checkbox to exclude metadata volume lineage.

    Include notebooks

    Select this option to collect lineage information for transformations from different entities like notebooks.

    Clear the checkbox to exclude lineage information for transformations from different entities.

    Filter dropped columns (In preview)

    Select this option to exclude columns that no longer exist in the current Databricks table schema from technical lineage. Databricks Unity Catalog stores lineage as a historical record, so columns from dropped or recreated tables persist in the lineage data. Selecting this option filters those columns from the lineage results in Collibra.

    Clear the checkbox to include all historical lineage data, including columns from dropped or recreated tables.

    Ingest metric views as tables

    If you select this option, Collibra Data Lineage ingests Databricks metric views as Table assets instead of Metric View assets. Use this option if your BI lineage expects metric views to be represented as tables, enabling successful cross-platform stitching.

    If you clear this option, Collibra Data Lineage preserves the native Metric View asset type from Databricks.

    Advanced settings
    Resolve notebook links across workspaces

    Select this option to build each notebook link using the workspace that the notebook belongs to, instead of the workspace configured on the connection. You can find the notebook link in the Transformation viewer by selecting the notebook node in the technical lineage viewer.

    Enable this option when multiple Databricks workspaces share a Unity Catalog metastore and notebook links otherwise point to the wrong workspace.

    When you enable this option, consider the following:

    • It applies only when Include notebooks is enabled.
    • It requires SELECT permission on system.access.workspaces_latest. If this permission isn't available, notebook links fall back to the configured workspace URL, and a warning appears in the Synchronization Report. Synchronization isn't affected.
    • The first synchronization after you enable the option uploads all notebook transformation files again.
  7. Click Save.
  8. Click Synchronize.
    A notification indicates synchronization has started.
    The synchronization job is started. Collibra Data Lineage ingests the metadata from Databricks Unity Catalog and processes the metadata to create technical lineage.
  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. In the tab bar, click Integrations.
    The Integrations page opens.
  3. Click the Integration Configuration tab.
  4. Locate the Databricks Workspace connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the Databricks Workspace connection, expand them to locate your technical lineage capability.
    The technical lineage capability configuration page opens.
  5. In the Synchronization Configuration section, click the Edit icon.
  6. Complete the fields as needed.
    FieldAction
    Options updated: <timestamp>

    Options updated: <timestamp> next to Synchronization configuration indicates the last time the data was retrieved.

    A scheduled refresh job is automatically created when you complete the capability configuration. You can also manually refresh the metadata by clicking Options updated: <timestamp>. The scheduled refresh job appears on the Jobs dashboard of your Edge site with the same name as your lineage capability.

    If two jobs with the same name appear on the Jobs dashboard, one is the scheduled refresh job and the other is the lineage synchronization job. By default, the scheduled refresh job runs daily at 23:00 server time. Unlike the lineage synchronization job, the scheduled refresh job appears only on the Jobs dashboard and not in the Activities list.

    This action prepopulates the Catalog name field.

    SystemSelect the System asset in which the Databricks assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
    Catalog name

    The field is prepopulated with the catalog names available in your Databricks workspace. Loading the catalog names into the dropdown list can take some time.

    If the catalog names can't be retrieved due to insufficient permissions, you can enter the catalog name manually. This doesn't affect lineage extraction.

    If you want to create technical lineage using views in a custom catalog instead of system tables, enter the custom catalog name in this field.
    For more information about the custom catalog, go to Databricks Unity Catalog lineage integration preflight checks.

    Include filterTo include specific workspaces, catalogs, or schemas in technical lineage, click Add Include pattern under Include filter. If you do not specify an include filter, technical lineage includes all lineage from Databricks Unity Catalog.

    The following rules apply when you enter the include pattern:

    • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
    • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
    • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
    • Lineage is collected only when both the source and the target are in the schema, catalog, or workspace specified in the include filter.
    • * > HR: Includes all schemas under the HR catalog.
    • * > Orders > fk*: Includes schemas starting with fk under the Orders catalog.
    • * > * > profiling: Includes all schemas named profiling under any catalog.
    Exclude filterTo exclude certain workspaces, catalogs, or schemas from technical lineage, click Add Exclude pattern under Exclude filter. If you do not specify an exclude filter, technical lineage includes all lineage from Databricks Unity Catalog.

    The following rules apply when you enter the exclude pattern:

    • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
    • The exclude filter takes precedence over the include filter.
    • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
    • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
    • * > testDB: Excludes all schemas under the testDB catalog.
    SQL sources limit

    Specify the limit on the number of SQL statements included for each relation in the technical lineage graph. The default value is 5.

    Include SQL transformations

    Select this option to enable Collibra Data Lineage to extract transformation logic from notebooks, jobs, SQL queries, and dashboards, and include it in the technical lineage viewer. You can view the transformation logic in the Source code pane of the technical lineage viewer.

    Clear the checkbox if you do not want Collibra Data Lineage to ingest transformation logic.

    Note If the personal access token, OAuth client, or Entra ID principal used for the Databricks Workspace connection does not have SELECT permission on the system.query.history table, you must clear this checkbox. Otherwise, connection errors might occur.
    Include external locations

    Select this option for Collibra Data Lineage to collect lineage information from the external locations to create end-to-end lineage.

    Clear the checkbox to exclude metadata from external locations.

    Include volumes (In preview)

    Select this option to collect lineage information from volumes to create end-to-end lineage. Only lineage relationships are collected; volume assets are not created in Data Catalog.

    Clear the checkbox to exclude metadata volume lineage.

    Include notebooks

    Select this option to collect lineage information for transformations from different entities like notebooks.

    Clear the checkbox to exclude lineage information for transformations from different entities.

    Filter dropped columns (In preview)

    Select this option to exclude columns that no longer exist in the current Databricks table schema from technical lineage. Databricks Unity Catalog stores lineage as a historical record, so columns from dropped or recreated tables persist in the lineage data. Selecting this option filters those columns from the lineage results in Collibra.

    Clear the checkbox to include all historical lineage data, including columns from dropped or recreated tables.

    Ingest metric views as tables

    If you select this option, Collibra Data Lineage ingests Databricks metric views as Table assets instead of Metric View assets. Use this option if your BI lineage expects metric views to be represented as tables, enabling successful cross-platform stitching.

    If you clear this option, Collibra Data Lineage preserves the native Metric View asset type from Databricks.

    Advanced settings
    Resolve notebook links across workspaces

    Select this option to build each notebook link using the workspace that the notebook belongs to, instead of the workspace configured on the connection. You can find the notebook link in the Transformation viewer by selecting the notebook node in the technical lineage viewer.

    Enable this option when multiple Databricks workspaces share a Unity Catalog metastore and notebook links otherwise point to the wrong workspace.

    When you enable this option, consider the following:

    • It applies only when Include notebooks is enabled.
    • It requires SELECT permission on system.access.workspaces_latest. If this permission isn't available, notebook links fall back to the configured workspace URL, and a warning appears in the Synchronization Report. Synchronization isn't affected.
    • The first synchronization after you enable the option uploads all notebook transformation files again.
  7. Click Save.
  8. On the Synchronization Schedule tab pane, click Add Schedule.
  9. Enter the required information and click Save:
    FieldDescription
    RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
    Cron

    The Quartz Cron expression that determines when the synchronization takes place.

    This field is only visible if you select Cron expression in the Repeat field.

    Every

    The day on which you want to synchronize, for example, Sunday.

    This field is only visible if you select Weekly in the Repeat field.

    Every first

    The day of the month on which you want to synchronize, for example, Tuesday.

    This field is only visible if you select Monthly in the Repeat field.

    At

    The time at which you want to synchronize automatically, for example, 14:00.

    • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
    • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
    Time zoneThe time zone for the schedule.

Helpful resources

Data Lineage Error: 'Could not connect to Databricks'