Steps overview: Integrate Databricks Unity Catalog via Edge

The following table shows the required steps for integration.

# Step Description
1 Review the preflight checks. Key considerations to help ensure successful integration, including required Edge, technical lineage, and data source-specific permissions, network requirements and more.
2

Create a Databricks connection.

Create a Databriks connection to allow Collibra Data Lineage to connect to and retrieve metadata from Databricks Unity Catalog.
3

Add the Technical Lineage for Databricks Unity Catalog capability.

 Add the technical lineage capability to your Edge or Collibra Cloud site. The capability allows the lineage harvester to retrieve data from your data source.
4 Synchronize the technical lineage.

You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.
Important 
  • Collibra Data Lineage ingests lineage for Databases, Schemas, Tables, Columns, Volumes (in preview), and Notebooks (in preview), but does not ingest any other assets such as Workflows.

    For more information, go to Supported transformation details.

  • Collibra Data Lineage creates technical lineage for Databricks Unity Catalog by using the custom technical lineage scanner. Therefore, the scanner type is listed as CUSTOM-LINEAGE on the technical lineage Sources tab page.

What's next

After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra. If you selected the Save Input Metadata checkbox when you added the Edge capability, click Download to save the extracted metadata file.

If there are errors in the synchronization result, go to Technical lineage via Edge troubleshooting for Databricks in Collibra Support Portal for more information.

Helpful resources