Databricks Unity Catalog is a technical catalog on Databricks side that provides schema information for all the Databricks databases that are available in the connected Databricks instances.
If you integrate Databricks Unity Catalog, you integrate the metadata of all databases in the Databricks Unity Catalog metastore into Collibra Data Intelligence Platform. The resulting assets represent the Databricks databases, schemas, tables and columns.
- Because we only integrate the metadata, you cannot get sample data for the columns and tables, nor profile and classify them. If you want to do that, you need to register the Databricks database via the Databricks JDBC driver. For information, go to combining the integration and the JDBC driver.
- The Databricks Unity Catalog integration supports the integration of following tables: EXTERNAL, MANAGED, STREAMING_TABLE, and VIEW tables.
- You can follow a course on this Databricks Unity Catalog integration via Collibra University.
- In the latest UI, you can also integrate Databricks AI models. To do so, make sure to set up your connection permissions and define the required configuration.
- You can integrate Databricks Unity Catalog only via Edge. You cannot integrate Databricks Unity Catalog via Jobserver.
- You cannot retrieve sample data or profile and classify the data for the Tables and Column assets created via the Databricks Unity Catalog integration. If you want to do that, you need to register the Databricks database via the Databricks JDBC driver.
Why use the Databricks Unity Catalog integration?
The Databricks Unity Catalog integration allows to get all the metadata from Databricks Unity Catalog into Collibra in one action, which means you quickly get an overview of all your Databricks databases in Collibra Data Intelligence Platform. You can also register Databricks databases into Collibra Data Intelligence Platform via the Databricks JDBC connection. However, you must register each database individually.
If you used the JDBC Databricks driver to register a specific Databricks database before, the related Database assets are not integrated again when you run the Databricks Unity Catalog integration.
For more details on the different ways of working with Databricks in Collibra and how to combine the integration and individual registration of databases, go to Ways to work with Databricks.
For more information about Databricks Unity Catalog, go to the Databricks Unity Catalog documentation.