Synchronize Databricks Unity Catalog

After Edge is ready to integrate Databricks Unity Catalog, you can start the synchronization process.

Synchronizing Databricks Unity Catalog is the process of integrating metadata from the databases connected to Databricks Unity Catalog and making this metadata available in Collibra. You can also use the synchronization to add Databricks AI Model assets in Collibra.

You can either synchronize manually or automate the process by adding a synchronization schedule.

Prerequisites

In your Collibra environment:

Steps

  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. In the tab bar, click Integrations.
    The Integrations page opens.
  3. Click the Integration Configuration tab.
  4. Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Capabilities column.
    The synchronization configuration page opens.
  5. In the Synchronization Configuration section, click the Edit icon.
  6. In Ingestion Type, select what you want to integrate.
    The available options are: metadata, AI models, and metadata and AI models.

    Based on your selection, additional fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.

  7. Complete the fields as needed.
    FieldAvailable if you integrateAction
    SystemMetadata

    In System, select the System asset in which you want to link the Databricks assets.

    You can update the System asset in the Databricks Unity Catalog integration synchronization, if needed. However, if you change the System asset, don't reuse a previously used System asset for the integration. For example: sys1 > sys2 > ... > sysN > sys1.
    Default Asset Status (Deprecated)

    Metadata

    AI models

    In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are:

    • Implemented: Implemented means that all assets receive the "Implemented" status.
    • No Status: No status means that newly created assets receive the first status listed in your Operating Model statuses, and that existing assets keep their assigned status.

    This field for applying the default asset status in the synchronization configuration is deprecated and will be removed in the future. You can define the default status using the Default Asset Status field in the capability configuration instead. Ensure that the value in this field matches the one in the capability configuration, as this field still takes precedence over the capability value.

    Version Metadata

    Select one of the following values to determine how schemas are integrated into domains:

    V0
    All schemas are included in domain mappings. Schemas that are not explicitly mapped must be excluded manually using the domain exclude mapping.
    V1
    Only schemas explicitly mapped are included. All other schemas are automatically excluded. With this option, you do not need to exclude schemas you do not want to integrate.
    Example 

    If you specify the following mappings in the Domain Include Mappings field:

    • Path Orders and domain Domain A
    • Path Orders > fk* and domain Domain B

    Order is the database, and fk* is any schema with a name that starts with fk.

    • If you select V0, the Order database and all other schemas that do not start with fk are integrated into Domain A.
      All schemas starting with fk are integrated into Domain B. You must manually exclude any additional schemas you do not want.
    • If you select V1, the Order database is integrated into Domain A, and all schemas starting with fk are integrated into Domain B.
      Schemas not explicitly mapped are automatically excluded, so you do not need to specify Domain Exclude Mappings.
    Domain Include MappingsMetadata

    Optionally, in Domain Include Mappings, specify the databases and schemas that you want to integrate, and optionally, the Collibra domains where they need to be added.
    This means you can use this field to limit the databases and schemas you integrate and define where they need to be added.

    Important 
    • If you don't define include any mappings, the integration automatically creates new domains for each Database and Schema asset in the same community as the System asset. For more information, go to Integrated Databricks Unity Catalog data.
    • If you include a path but don't define a domain, the integration automatically creates new domains in the same community as the System asset.
    • If you selected V0 for the Version field and add a domain include mapping for the database but not for a related schema, the automatically created domain for the schema is added in the same community as the domain of the database.
    • A match with a schema has priority over a match with a database.

    To limit the scope of metadata ingestion to specific domains in Collibra, add a domain include mapping:

    1. Click Add Domain Include Mappings.
    2. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. You can use the question mark (?) and asterisk (*) wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account.
    3. Optionally, in Domain, select the Collibra domain in which you want to integrate the metadata. If you don't define a domain, the integration automatically creates new domains in the same community as the System asset.
    Example 
    • Path Orders and domain Domain B
      In this case, the Orders Database asset and all its related assets will be integrated in Domain B.
    • Path Orders > fk* and domain Domain B
      In this case, the Orders Database asset will be integrated into its own domain in the same community as the System asset.
      All schemas that start with fk and their related assets will be integrated in Domain B.
    • Path Orders > * and domain Domain B
      In this case, the Orders Database asset will be integrated in the same domain as the System asset.
      All schemas in the Orders catalog and their related assets will be integrated in Domain B.

    You have a database Orders that includes multiple schemas.

    • If you want to make sure that the Orders database and related schemas are added to Domain B, add the following include mappings:
      • Path Orders and domain Domain B, to make sure the Database asset is added to Domain B.
      • Path Orders > * and domain Domain B, to make sure all Schema assets in Orders are added to Domain B.
    • If you want to make sure that the Orders database and related schemas are added to domain B, except for the schemas that start with test_, add the following include mappings: 
      • Path Orders and domain Domain B, to make sure the Database asset is added to Domain B.
      • Path Orders > test_* and domain Domain C, to make sure that all schemas in Order that start with test_ are added to Domain C.
      • Path Orders > * and domain Domain B, to make sure all other Schema assets in Orders are added to Domain B.
    Domain Exclude MappingsMetadata

    Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate.

    Note The exclude mapping has priority over the include mapping.

    To exclude specific metadata from being ingested into Collibra, add a domain exclude mapping:

    1. Click Add Domain Exclude Mappings.
    2. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. You can use the question mark (?) and asterisk (*) wildcards in the catalog and schema names.
      For example: * > test.
    Extensible Properties MappingsMetadata

    Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects.

    Optionally, in Extensible Properties Mappings, specify the additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into Collibra.
    You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in Collibra assets. You can do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute.

    Important 
    • If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.
    • The name of the property starts with the object type, for example catalogs.systemAttributes.metastore_id.
      catalogs refers to Database assets, schemas to Schema assets, table to Table assets, and views to Database View assets.
    • The following system properties are supported:  
      • Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by".
      • Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by".
      • Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition".
      • Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition".

    To add an additional property mapping:

    1. Click Add Another Mapping.
    2. In Property Name, do one of the following:
      • To add a system attribute, select the Databricks Unity Catalog property name from the dropdown list.
      • To add a custom attribute, type the name of the custom property manually.
        Use the following naming convention: [object type].customParameters.[name of parameter].
        For example:
        catalogs.customParameters.Parameter1
        schemas.customParameters.catalogAndNamespace.part.1
        table.customParameters.view.catalogAndNamespace.part.2
        views.customParameters.Paramerer2
    3. In Attribute, select the attribute in which you want to see the value.
    Stop Compute ResourceMetadata

    This field is important if the Compute Resource HTTP Path field was completed in the Databricks capability to allow for source tag integration.

    • If this field is set to Yes, the compute resource in Databricks Unity Catalog will be stopped right after the source tags are extracted.
    • If this field is set to No, the compute resource remains active.
    Tip To prevent clusters from running for the entire synchronization duration, you can also configure the Terminate after ... minutes of inactivity setting in Databricks. The setting ensures that clusters automatically stop after a period of inactivity. For more information, go to the Databricks documentation.
    DomainAI modelsIn Domain, select the domain in which you want to add the Databricks AI Model assets.
    Custom AI Metrics MappingsAI models

    Optionally, in Custom AI Metrics Mappings, define the custom Databricks AI Model metrics that you want to integrate. You can do this by adding the mapping between the custom metric and the Collibra attribute.

    • accuracy_score,
    • exact_match
    • example_count
    • f1_score
    • f1_score_micro
    • f1_score_macro
    • false_negatives
    • false_positives
    • log_loss
    • max_error
    • mean_absolute_error
    • mean_absolute_percentage_error
    • mean_on_target
    • mean_squared_error
    • precision
    • precision_recall_auc
    • r2_score
    • recall
    • roc_auc
    • root_mean_squared_error
    • sum_on_target
    • token_count
    • true_negatives
    • true_positives

    For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data.

    Important 

    If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.

    To add a custom AI metric mapping:

    1. Click Add Custom AI Metrics Mappings.
    2. In Metric, select the custom metric from the list of available Databricks AI metrics.
    3. In Attribute, select the attribute in which you want to see the value.
      Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
    Exclude system AI ModelsAI models

    Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models.

    • By default, No is selected, and all accessible AI models are integrated.
    • If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration.

    For more information about these pretrained Databricks AI models, go to the Databricks documentation.

    Important When you integrate a data source without applying Include or Exclude Mappings rules, and then later exclude a registered asset using an Include or Exclude Mapping during resynchronization, the related assets receive the Missing from Source status.
  8. Click Save.
  9. Click Synchronize.
    A notification indicates the synchronization has started.
  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. In the tab bar, click Integrations.
    The Integrations page opens.
  3. Click the Integration Configuration tab.
  4. Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Capabilities column.
    The synchronization configuration page opens.
  5. In the Synchronization Configuration section, click the Edit icon.
  6. In Ingestion Type, select whether you want to integrate the metadata, AI models, or metadata and AI models.

    Based on your selection, additional fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.

  7. Complete the fields as needed.
    FieldAvailable if you integrateAction
    SystemMetadata

    In System, select the System asset in which you want to link the Databricks assets.

    You can update the System asset in the Databricks Unity Catalog integration synchronization, if needed. However, if you change the System asset, don't reuse a previously used System asset for the integration. For example: sys1 > sys2 > ... > sysN > sys1.
    Default Asset Status (Deprecated)

    Metadata

    AI models

    In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are:

    • Implemented: Implemented means that all assets receive the "Implemented" status.
    • No Status: No status means that newly created assets receive the first status listed in your Operating Model statuses, and that existing assets keep their assigned status.

    This field for applying the default asset status in the synchronization configuration is deprecated and will be removed in the future. You can define the default status using the Default Asset Status field in the capability configuration instead. Ensure that the value in this field matches the one in the capability configuration, as this field still takes precedence over the capability value.

    Version Metadata

    Select one of the following values to determine how schemas are integrated into domains:

    V0
    All schemas are included in domain mappings. Schemas that are not explicitly mapped must be excluded manually using the domain exclude mapping.
    V1
    Only schemas explicitly mapped are included. All other schemas are automatically excluded. With this option, you do not need to exclude schemas you do not want to integrate.
    Example 

    If you specify the following mappings in the Domain Include Mappings field:

    • Path Orders and domain Domain A
    • Path Orders > fk* and domain Domain B

    Order is the database, and fk* is any schema with a name that starts with fk.

    • If you select V0, the Order database and all other schemas that do not start with fk are integrated into Domain A.
      All schemas starting with fk are integrated into Domain B. You must manually exclude any additional schemas you do not want.
    • If you select V1, the Order database is integrated into Domain A, and all schemas starting with fk are integrated into Domain B.
      Schemas not explicitly mapped are automatically excluded, so you do not need to specify Domain Exclude Mappings.
    Domain Include MappingsMetadata

    Optionally, in Domain Include Mappings, specify the databases and schemas that you want to integrate, and optionally, the Collibra domains where they need to be added.
    This means you can use this field to limit the databases and schemas you integrate and define where they need to be added.

    Important 
    • If you don't define include any mappings, the integration automatically creates new domains for each Database and Schema asset in the same community as the System asset. For more information, go to Integrated Databricks Unity Catalog data.
    • If you include a path but don't define a domain, the integration automatically creates new domains in the same community as the System asset.
    • If you selected V0 for the Version field and add a domain include mapping for the database but not for a related schema, the automatically created domain for the schema is added in the same community as the domain of the database.
    • A match with a schema has priority over a match with a database.

    To limit the scope of metadata ingestion to specific domains in Collibra, add a domain include mapping:

    1. Click Add Domain Include Mappings.
    2. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. You can use the question mark (?) and asterisk (*) wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account.
    3. Optionally, in Domain, select the Collibra domain in which you want to integrate the metadata. If you don't define a domain, the integration automatically creates new domains in the same community as the System asset.
    Example 
    • Path Orders and domain Domain B
      In this case, the Orders Database asset and all its related assets will be integrated in Domain B.
    • Path Orders > fk* and domain Domain B
      In this case, the Orders Database asset will be integrated into its own domain in the same community as the System asset.
      All schemas that start with fk and their related assets will be integrated in Domain B.
    • Path Orders > * and domain Domain B
      In this case, the Orders Database asset will be integrated in the same domain as the System asset.
      All schemas in the Orders catalog and their related assets will be integrated in Domain B.

    You have a database Orders that includes multiple schemas.

    • If you want to make sure that the Orders database and related schemas are added to Domain B, add the following include mappings:
      • Path Orders and domain Domain B, to make sure the Database asset is added to Domain B.
      • Path Orders > * and domain Domain B, to make sure all Schema assets in Orders are added to Domain B.
    • If you want to make sure that the Orders database and related schemas are added to domain B, except for the schemas that start with test_, add the following include mappings: 
      • Path Orders and domain Domain B, to make sure the Database asset is added to Domain B.
      • Path Orders > test_* and domain Domain C, to make sure that all schemas in Order that start with test_ are added to Domain C.
      • Path Orders > * and domain Domain B, to make sure all other Schema assets in Orders are added to Domain B.
    Domain Exclude MappingsMetadata

    Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate.

    Note The exclude mapping has priority over the include mapping.

    To exclude specific metadata from being ingested into Collibra, add a domain exclude mapping:

    1. Click Add Domain Exclude Mappings.
    2. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. You can use the question mark (?) and asterisk (*) wildcards in the catalog and schema names.
      For example: * > test.
    Extensible Properties MappingsMetadata

    Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects.

    Optionally, in Extensible Properties Mappings, specify the additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into Collibra.
    You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in Collibra assets. You can do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute.

    Important 
    • If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.
    • The name of the property starts with the object type, for example catalogs.systemAttributes.metastore_id.
      catalogs refers to Database assets, schemas to Schema assets, table to Table assets, and views to Database View assets.
    • The following system properties are supported:  
      • Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by".
      • Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by".
      • Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition".
      • Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition".

    To add an additional property mapping:

    1. Click Add Another Mapping.
    2. In Property Name, do one of the following:
      • To add a system attribute, select the Databricks Unity Catalog property name from the dropdown list.
      • To add a custom attribute, type the name of the custom property manually.
        Use the following naming convention: [object type].customParameters.[name of parameter].
        For example:
        catalogs.customParameters.Parameter1
        schemas.customParameters.catalogAndNamespace.part.1
        table.customParameters.view.catalogAndNamespace.part.2
        views.customParameters.Paramerer2
    3. In Attribute, select the attribute in which you want to see the value.
    Stop Compute ResourceMetadata

    This field is important if the Compute Resource HTTP Path field was completed in the Databricks capability to allow for source tag integration.

    • If this field is set to Yes, the compute resource in Databricks Unity Catalog will be stopped right after the source tags are extracted.
    • If this field is set to No, the compute resource remains active.
    Tip To prevent clusters from running for the entire synchronization duration, you can also configure the Terminate after ... minutes of inactivity setting in Databricks. The setting ensures that clusters automatically stop after a period of inactivity. For more information, go to the Databricks documentation.
    DomainAI modelsIn Domain, select the domain in which you want to add the Databricks AI Model assets.
    Custom AI Metrics MappingsAI models

    Optionally, in Custom AI Metrics Mappings, define the custom Databricks AI Model metrics that you want to integrate. You can do this by adding the mapping between the custom metric and the Collibra attribute.

    • accuracy_score,
    • exact_match
    • example_count
    • f1_score
    • f1_score_micro
    • f1_score_macro
    • false_negatives
    • false_positives
    • log_loss
    • max_error
    • mean_absolute_error
    • mean_absolute_percentage_error
    • mean_on_target
    • mean_squared_error
    • precision
    • precision_recall_auc
    • r2_score
    • recall
    • roc_auc
    • root_mean_squared_error
    • sum_on_target
    • token_count
    • true_negatives
    • true_positives

    For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data.

    Important 

    If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment.

    To add a custom AI metric mapping:

    1. Click Add Custom AI Metrics Mappings.
    2. In Metric, select the custom metric from the list of available Databricks AI metrics.
    3. In Attribute, select the attribute in which you want to see the value.
      Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
    Exclude system AI ModelsAI models

    Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models.

    • By default, No is selected, and all accessible AI models are integrated.
    • If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration.

    For more information about these pretrained Databricks AI models, go to the Databricks documentation.

    Important When you integrate a data source without applying Include or Exclude Mappings rules, and then later exclude a registered asset using an Include or Exclude Mapping during resynchronization, the related assets receive the Missing from Source status.
  8. Click Save.
  9. In the Synchronization Schedule section, click the Add synchronization schedule icon.
  10. Enter the information.
    FieldDescription
    RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
    Cron

    The Quartz Cron expression that determines when the synchronization takes place.

    This field is only visible if you select Cron expression in the Repeat field.

    Every

    The day on which you want to synchronize, for example, Sunday.

    This field is only visible if you select Weekly in the Repeat field.

    Every first

    The day of the month on which you want to synchronize, for example, Tuesday.

    This field is only visible if you select Monthly in the Repeat field.

    At

    The time at which you want to synchronize automatically, for example, 14:00.

    • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
    • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
    Time zoneThe time zone for the schedule.
  11. Click Save

What's next

Depending on your selection, the synchronization job integrates the AI models, metadata of the databases, schemas, tables, and columns, or both.
After the synchronization: