About the Databricks Unity Catalog integration via Edge

Databricks Unity Catalog is a technical catalog that provides schema information for all the databases in your connected Databricks instances.

The Databricks Unity Catalog integration enables seamless integration between Collibra and Databricks Unity Catalog. It supports both inbound and outbound (in preview) metadata synchronization to keep your catalog in sync with Databricks Unity Catalog. You can integrate Databricks Unity Catalog only via Edge.

Integrating Databricks Unity Catalog is the recommended way to work with Databricks Unity Catalog because it shows the hierarchy of the assets and allows you to set up sampling, profiling, and classification (in preview).

  • Inbound synchronization: Integrates metadata of all or multiple databases from Databricks Unity Catalog into Collibra. The integrated assets are Databricks databases, schemas, tables, and columns.
  • Outbound synchronization (in preview): Propagates attribute values from Collibra to tags in Databricks Unity Catalog, as mapped in the outbound configuration.The outbound flow is supported for Database, Schema, Table, Database View, and Column assets. If any tag name/value is left blank, the outbound synchronization will remove the tag from the asset in Databricks Unity Catalog. The integration supports mapping the following two types of tags to Databricks Unity Catalog:
    • Custom tags: Custom tags accept any user-defined key-value pair. These are suitable for asset attributes such as description or any other custom attributes where the possible values are not predefined.
    • Governed tags: Governed tags are a subset of custom tags. These are linked to tag policies in Databricks Unity Catalog that define allowed values. In contrast to custom tags, governed tags are ideal for attributes such as status, where all possible values are known in advance. Note that governed tags only accept predefined allowed values and reject any others. If you assign a tag value that is not allowed, it will not be propagated to the governed tag in Databricks Unity Catalog. In this case, the new value will appear in the asset attribute but the associated governed tag will remain unchanged. For attributes with multiple values, the combined, semicolon-separated value must itself match an allowed value, so use a custom tag or policy that allows the combined values. For more information on governed tags, go to the Databricks Unity Catalog documentation.
      Note 
    • Outbound synchronization is only supported for assets ingested through the Databricks Unity Catalog integration, and not for assets ingested through a JDBC data source using the scalable ingestion flow.
    • Support for outbound synchronization of attributes such as responsibilities, data classification, articulation scores, Collibra tags, and data quality scores from Collibra to Databricks Unity Catalog will be added in the future.

Suppose that you want to add 10 databases and profile the data.

If using only the Databricks JDBC connector

  1. Create 10 JDBC connections.
  2. Add the required capabilities to each connection.
  3. Register and synchronize each database individually.

If using the Databricks Unity Catalog integration with profiling

  1. Create 2 connections: one for the integration and one for JDBC.
  2. Add the required capabilities to the JDBC connection.
  3. Integrate Databricks Unity Catalog.
    The resynchronization for the databases is managed through the Databricks Unity Catalog capability, and profiling is performed via the Database asset.

If you previously used a combination of integrating Databricks Unity Catalog and registering an individual Databricks database via the Databricks JDBC driver, and you want to switch to using only the integration, go to Switching to working only with Databricks Unity Catalog integration (in preview).

Supported table types

The Databricks Unity Catalog integration supports the following table types:

  • EXTERNAL
  • MANAGED
  • STREAMING_TABLE
  • VIEW

Helpful resources

To learn more about the Databricks Unity Catalog integration, follow our University course.

For more information about Databricks, go to the Databricks documentation.