Synchronize Databricks Unity Catalog

Important

In Collibra 2024.05, we launched a new user interface (UI) for Collibra Data Intelligence Platform! You can learn more about this latest UI in the UI overview.

Use the following options to see the documentation in the latest UI or in the previous, classic UI:

Latest UI Classic UI

Synchronizing Databricks Unity Catalog is the process of integrating metadata from the databases connected to Databricks Unity Catalog and making this metadata available in Collibra Data Intelligence Platform. You can also use the synchronization to add Databricks AI Model assets in Collibra.
You can synchronize manually, or you can automate it by adding a synchronization schedule.

Before you begin

You have created a connection to Databricks in your Edge site.
You have added the Databricks Unity Catalog capability for the connection.
For metadata synchronization, you know in which System asset you want to add the Databricks Unity Catalog assets.
- If you have registered Databricks databases via the JDBC driver before, use the same System asset.
- If you never registered Databricks databases before, create a new System asset manually and use that one.
For AI model synchronization, you know in which domain you want to add the Databricks AI Model assets.

Requirements and permissions

You have a resource role with the Configure external system resource permission, for example, Owner.
You have a global role with the Catalog global permission, for example, Catalog Author.
You have a global role with the View Edge connections and capabilities global permission, for example, Edge integration engineer.

Steps

Manually synchronize Databricks Unity Catalog
Add a synchronization schedule

On the main toolbar, click , and then click Catalog.
The Catalog Home opens.
In the tab bar, click Integrations.
The Integrations page opens.
Click the Integration Configuration tab.
On the main toolbar, click .
The Create dialog box appears.
In the Register with Edge section of the Create dialog box, click Register a data sourceIntegration Configuration.
The Register contentIntegration Configuration tab page opens.
Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Data sources/Capabilities column.
The synchronization configuration page opens.
In the Configuration SectionSynchronization Configuration section, click Add Configuration.
In Ingestion Type, select whether you want to integrate the metadata, AI models, or metadata and AI models.
Depending on your selection, extra fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.

Complete the fields as needed.

Field	Available if you integrate	Action
System	Metadata	In System, select the System asset in which you want to link the Databricks assets.
Default Asset Status	Metadata AI models	In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are: Implemented: Implemented means that all assets receive the Implemented status. No Status: No status means that newly created assets receive the first status listed in your operating model statuses, and that existing assets keep their assigned status. Example In these examples, you register a data source with assets A, B, and C. Show example 1 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Implemented Asset B: In Review Asset C: Implemented Asset D: Candidate, because this status is the first one listed in your operating model. Show example 2 During the first integration, you set the Default Asset Status field to No status. As a result, A, B, and C receive the Candidate status, because this status is the first one listed in your operating model. You change the status of asset B to Implemented. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Candidate Asset B: Implemented Asset C: Candidate Asset D: Candidate Show example 3 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to Implemented. As a result, the assets receive the following status: Asset A: Implemented Asset B: Implemented Asset C: Implemented Asset D: Implemented
Domain Include Mappings	Metadata	Optionally, in Domain Include Mappings, specify which databases and schemas that you want to integrate and the Collibra domains where they need to be added. Important If you don't define include mappings, the integration automatically creates new domains for each Database and Schema asset in the same domain as the System asset. A match with a database has priority over a match with a schema. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. Tip You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account. Example Path `Orders` and domain `Domain B` In this case, the Orders Database asset and all its related assets will be integrated in Domain B. Path `Orders > fk` and domain `Domain B` In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas that start with fk and their related assets will be integrated in Domain B. Path `Orders > ` and domain `Domain B` In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas in the Orders catalog and their related assets will be integrated in Domain B. In Domain, select the Collibra domain in which you want to integrate the metadata.
Domain Exclude Mappings	Metadata	Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. Tip You can use the ? and * wildcards in the catalog and schema names. For example:`* > test`.
Extensible Properties Mappings	Metadata	Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects. Optionally, in Extensible Properties Mappings, specify which additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into Collibra. You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in Collibra assets. You do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute. Important If you use this feature, make sure to add all required characteristics to the asset type assignments. The name of the property starts with the object type, for example `catalogs.systemAttributes.metastore_id`. `catalogs` refers to Database assets, `schemas` to Schema assets, `table` to Table assets, and `views` to Database View assets. The following system properties are supported: Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by". Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by". Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Show steps to add a property mapping Click Add Another Mapping. In Property Name, do one of the following: To add a system attribute, select the Databricks Unity Catalog property name from the drop-down list. To add a custom attribute, type the name of the custom property manually. Use the following naming convention: `[object type].customParameters.[name of parameter]`. For example: catalogs.customParameters.Parameter1 schemas.customParameters.catalogAndNamespace.part.1 table.customParameters.view.catalogAndNamespace.part.2 views.customParameters.Paramerer2 In Attribute, select the attribute in which you want to see the value.
Domain	AI models	In Domain, select the domain in which you want to add the Databricks AI Model assets.
Custom AI Metrics Mappings	AI models	Optionally, in Custom AI Metrics Mappings, define which custom Databricks AI Model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data. Important If you use this feature, make sure to add all required characteristics to the Databricks AI Model asset type assignment. Show steps to add a custom AI metrics mapping Click Add Custom AI Metrics Mappings. In Metric, type the name of the custom metric manually. Use the exact name as in Databricks Unity Catalog. In Attribute, select the attribute in which you want to see the value. Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
Exclude system AI Models	AI models	Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models. By default, No is selected, and all accessible AI models are integrated. If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration. For more information about these pretrained Databricks AI models, go to the Databricks documentation.

Important

When you integrate a data source without applying Include or Exclude Mappings rules, and then later exclude a registered asset using an Include or Exclude Mapping during resynchronization, the related assets will receive the Missing from Source status.

Click Save Configuration.
Click Synchronize.
A notification indicates the synchronization has started.

On the main toolbar, click , and then click Catalog.
The Catalog Home opens.
In the tab bar, click Integrations.
The Integrations page opens.
Click the Integration Configuration tab.
On the main toolbar, click .
The Create dialog box appears.
In the Register with Edge section of the Create dialog box, click Register a data sourceIntegration Configuration.
The Register contentIntegration Configuration tab page opens.
Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Data sources/Capabilities column.
The synchronization configuration page opens.
In the Configuration SectionSynchronization Configuration section, click Add Configuration.
In Ingestion Type, select whether you want to integrate the metadata, AI models, or metadata and AI models.
Depending on your selection, extra fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.

Complete the fields as needed.

Field	Available if you integrate	Action
System	Metadata	In System, select the System asset in which you want to link the Databricks assets.
Default Asset Status	Metadata AI models	In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are: Implemented: Implemented means that all assets receive the Implemented status. No Status: No status means that newly created assets receive the first status listed in your operating model statuses, and that existing assets keep their assigned status. Example In these examples, you register a data source with assets A, B, and C. Show example 1 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Implemented Asset B: In Review Asset C: Implemented Asset D: Candidate, because this status is the first one listed in your operating model. Show example 2 During the first integration, you set the Default Asset Status field to No status. As a result, A, B, and C receive the Candidate status, because this status is the first one listed in your operating model. You change the status of asset B to Implemented. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Candidate Asset B: Implemented Asset C: Candidate Asset D: Candidate Show example 3 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to Implemented. As a result, the assets receive the following status: Asset A: Implemented Asset B: Implemented Asset C: Implemented Asset D: Implemented
Domain Include Mappings	Metadata	Optionally, in Domain Include Mappings, specify which databases and schemas that you want to integrate and the Collibra domains where they need to be added. Important If you don't define include mappings, the integration automatically creates new domains for each Database and Schema asset in the same domain as the System asset. A match with a database has priority over a match with a schema. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. Tip You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account. Example Path `Orders` and domain `Domain B` In this case, the Orders Database asset and all its related assets will be integrated in Domain B. Path `Orders > fk` and domain `Domain B` In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas that start with fk and their related assets will be integrated in Domain B. Path `Orders > ` and domain `Domain B` In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas in the Orders catalog and their related assets will be integrated in Domain B. In Domain, select the Collibra domain in which you want to integrate the metadata.
Domain Exclude Mappings	Metadata	Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. Tip You can use the ? and * wildcards in the catalog and schema names. For example:`* > test`.
Extensible Properties Mappings	Metadata	Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects. Optionally, in Extensible Properties Mappings, specify which additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into Collibra. You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in Collibra assets. You do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute. Important If you use this feature, make sure to add all required characteristics to the asset type assignments. The name of the property starts with the object type, for example `catalogs.systemAttributes.metastore_id`. `catalogs` refers to Database assets, `schemas` to Schema assets, `table` to Table assets, and `views` to Database View assets. The following system properties are supported: Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by". Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by". Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Show steps to add a property mapping Click Add Another Mapping. In Property Name, do one of the following: To add a system attribute, select the Databricks Unity Catalog property name from the drop-down list. To add a custom attribute, type the name of the custom property manually. Use the following naming convention: `[object type].customParameters.[name of parameter]`. For example: catalogs.customParameters.Parameter1 schemas.customParameters.catalogAndNamespace.part.1 table.customParameters.view.catalogAndNamespace.part.2 views.customParameters.Paramerer2 In Attribute, select the attribute in which you want to see the value.
Domain	AI models	In Domain, select the domain in which you want to add the Databricks AI Model assets.
Custom AI Metrics Mappings	AI models	Optionally, in Custom AI Metrics Mappings, define which custom Databricks AI Model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data. Important If you use this feature, make sure to add all required characteristics to the Databricks AI Model asset type assignment. Show steps to add a custom AI metrics mapping Click Add Custom AI Metrics Mappings. In Metric, type the name of the custom metric manually. Use the exact name as in Databricks Unity Catalog. In Attribute, select the attribute in which you want to see the value. Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
Exclude system AI Models	AI models	Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models. By default, No is selected, and all accessible AI models are integrated. If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration. For more information about these pretrained Databricks AI models, go to the Databricks documentation.

Important

Click Save Configuration.
In the Synchronization Schedule section, click Add Schedule.

Enter the information.

Field	Description
Repeat	The interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
Cron	The Quartz Cron expression that determines when the synchronization takes place. This field is only visible if you select `Cron expression` in the Repeat field.
Every	The day on which you want to synchronize, for example, Sunday. This field is only visible if you select `Weekly` in the Repeat field.
Every first	The day of the month on which you want to synchronize, for example, Tuesday. This field is only visible if you select `Monthly` in the Repeat field.
At	The time at which you want to synchronize automatically, for example, 14:00. You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45. This field is only visible if you select `Daily`, `Weekly`, or `Monthly` in the Repeat field.
Time zone	The time zone for the schedule.

Click Save

What's next?

Depending on your selection, the synchronization job integrates the AI models, the metadata of the databases, schemas, tables, and columns, or both.
After the synchronization:

The synchronization job integrates the metadata of the databases, schemas, tables and columns.
After the synchronization:

You can view a summary of the results from the Activities list.
For metadata synchronization, the resulting assets get a relation to the System asset that you selected.
For information on the integrated data, go to Integrated Databricks data.