Tip You can set up this integration in the unified integrations experience (in preview). This feature is in public preview and applies to metadata, AI, SAP, and lineage integrations. In one guided flow, you select an Edge site, create the connection, add the integration, and configure it. To get started, in the banner on the Integration configuration tab, click Try the new Integrations page. For more information, go to About the unified integrations experience (in preview).
The following table shows the required steps for creating technical lineage for Databricks Unity Catalog.
- Tip
- You can also integrate AI models using the Databricks Unity Catalog for AI integration. If you want to integrate Databricks AI models, ensure that AI Governance is enabled. Without AI Governance, the AI integration functionality is limited. For more information, go to the Databricks Unity Catalog for AI documentation.
- You can also integrate the metadata of Databricks databases, schemas, tables, database views, and columns using the Databricks Unity Catalog metadata integration. For more information, go to the Databricks Unity Catalog documentation.
| # | Step | Description |
|---|---|---|
| 1 | Review the preflight checks. | Key considerations to help ensure successful integration, including required Edge, technical lineage, and data source-specific permissions, network requirements and more. |
| 2 |
Create a Databricks Workspace connection. |
Create a Databricks Workspace connection to allow Collibra Data Lineage to connect to and retrieve metadata from Databricks Unity Catalog. |
| 3 |
Add the Technical Lineage for Databricks Unity Catalog capability. |
Add the technical lineage capability to your Edge or Collibra Cloud site. The capability allows Collibra Data Lineage to retrieve lineage information from your data source. |
| 4 | Synchronize the technical lineage. |
You can synchronize your technical lineage manually or automatically by adding a synchronization schedule. |
-
Collibra Data Lineage ingests lineage for Databases, Schemas, Tables, Columns, Volumes (in preview), and Notebooks, but does not ingest any other assets such as Workflows.
For more information, go to Supported transformation details.
- Databricks Unity Catalog technical lineage appears with the source type CUSTOM-LINEAGE on the technical lineage Sources tab page.
After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra. If you selected the Save Input Metadata checkbox when you added the Edge capability, click Download to save the extracted metadata file.
If there are errors in the synchronization result, go to Technical lineage via Edge troubleshooting for Databricks in Collibra Support Portal for more information.
Helpful resources
- Databricks Unity Catalog integration preflight checks
- Collibra Data Lineage service instances
- Connect to a Collibra Data Lineage service instance via OAuth authentication
- Connect to a proxy server
- Databricks Unity Catalog: Supported transformation details
- Supported SQL statements
- Automatic stitching for technical lineage
- Technical lineage admin options
- Delete a technical lineage