The following table shows the required steps for integration.
| # | Step | Description |
|---|---|---|
| 1 | Review the preflight checks. | Key considerations to help ensure successful integration, including required Edge, technical lineage, and data source-specific permissions, network requirements and more. |
| 2 |
Create a Databricks Workspace connection. |
Create a Databricks Workspace connection to allow Collibra Data Lineage to connect to and retrieve metadata from Databricks Unity Catalog. |
| 3 |
Add the Technical Lineage for Databricks Unity Catalog capability. |
Add the technical lineage capability to your Edge or Collibra Cloud site. The capability allows Collibra Data Lineage to retrieve lineage information from your data source. |
| 4 | Synchronize the technical lineage. |
You can synchronize your technical lineage manually or automatically by adding a synchronization schedule. |
-
Collibra Data Lineage ingests lineage for Databases, Schemas, Tables, Columns, Volumes (in preview), and Notebooks (in preview), but does not ingest any other assets such as Workflows.
For more information, go to Supported transformation details.
- Collibra Data Lineage creates technical lineage for Databricks Unity Catalog by using the custom technical lineage scanner. Therefore, the scanner type is listed as CUSTOM-LINEAGE on the technical lineage Sources tab page.
After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra. If you selected the Save Input Metadata checkbox when you added the Edge capability, click Download to save the extracted metadata file.
If there are errors in the synchronization result, go to Technical lineage via Edge troubleshooting for Databricks in Collibra Support Portal for more information.
Helpful resources
- Databricks Unity Catalog integration preflight checks
- Collibra Data Lineage service instances
- Connect to a Collibra Data Lineage service instance via OAuth authentication
- Connect to a proxy server
- Databricks Unity Catalog: Supported transformation details
- Supported SQL statements
- Automatic stitching for technical lineage
- Technical lineage admin options
- Delete a technical lineage