Ways to work with Databricks

Combining the ways of working with Databricks

The 2 possibilities don't cancel each other out. You can use both ways to show the information you want in Collibra Platform. You can use the integration of Databricks Unity Catalog to quickly get an overview of all your Databricks databases in Collibra Platform. Once you have a better view on the important databases, you can register them individually via the JDBC driver.

Important Use the same System asset for the integration and the registration. Otherwise, assets will be duplicated.

Switching to working exclusively with the Databricks Unity Catalog integration (beta)

If you previously used both the Databricks Unity Catalog integration and the Databricks JDBC synchronization for some databases, and you want to switch to using only the Databricks Unity Catalog integration, complete the following steps:

1. Update the existing Databricks Unity Catalog synchronization capability.

   1. Go to the existing Databricks Unity Catalog synchronization capability.
   2. Click Edit.
   3. Add the existing Databricks JDBC connection to the JDBC Databricks Connection field.
   4. Click Save.
2. Re-synchronize the Databricks Unity Catalog integration.
   Show how to synchronize Databricks Unity Catalog

   1. On the main toolbar, click → Catalog.
      The Data Catalog Home opens.
   2. In the tab bar, click Integrations.
      The Integrations page opens.
   3. Click the Integration Configuration tab.
   4. On the main toolbar, click .
      The Create dialog box appears.
   5. In the Register with Edge section of the Create dialog box, click Register a data sourceIntegration Configuration.
      The Register contentIntegration Configuration tab page opens.
   6. Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Data sources/Capabilities column.
      The synchronization configuration page opens.
   7. In the Configuration SectionSynchronization Configuration section, click Add Configuration.
   8. In the Configuration SectionSynchronization Configuration section, click the Edit icon.
   9. In Ingestion Type, select what you want to integrate.
      The available options are: metadata, AI models, and metadata and AI models.

      Depending on your selection, extra fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.
      

   10. Complete the fields as needed.

       Field	Available if you integrate	Action
       System	Metadata	In System, select the System asset in which you want to link the Databricks assets.
       Default Asset Status	Metadata AI models	In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are: Implemented: Implemented means that all assets receive the Implemented status. No Status: No status means that newly created assets receive the first status listed in your operating model statuses, and that existing assets keep their assigned status. Example  In these examples, you register a data source with assets A, B, and C. Show example 1 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Implemented Asset B: In Review Asset C: Implemented Asset D: Candidate, because this status is the first one listed in your operating model. Show example 2 During the first integration, you set the Default Asset Status field to No status. As a result, A, B, and C receive the Candidate status, because this status is the first one listed in your operating model. You change the status of asset B to Implemented. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to No Status. As a result, the assets receive the following status: Asset A: Candidate Asset B: Implemented Asset C: Candidate Asset D: Candidate Show example 3 During the first integration, you set the Default Asset Status field to Implemented. As a result, A, B, and C receive the Implemented status. You change the status of asset B to In Review. You register the same data source, which now has assets A. B, C, and D. During the second integration, you set the Default Asset Status field to Implemented. As a result, the assets receive the following status: Asset A: Implemented Asset B: Implemented Asset C: Implemented Asset D: Implemented
       Domain Include Mappings	Metadata	Optionally, in Domain Include Mappings, specify which databases and schemas you want to integrate and optionally the Collibra domains where they need to be added. This means you can use this field to limit the databases and schemas you integrate and to define where they need to be added. Important  If you don't define include any mappings, the integration automatically creates new domains for each Database and Schema asset in the same community as the System asset. For more information, go to Integrated Databricks Unity Catalog data. If you include a path but don't define a domain, the integration automatically creates new domains in the same community as the System asset. If you add a domain include mapping for the database but not for a related schema, the automatically created domain for the schema is added in the same community as the domain of the database. A match with a schema has priority over a match with a database. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. Tip  You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account. Optionally, in Domain, select the Collibra domain in which you want to integrate the metadata. If you don't define a domain, the integration automatically creates new domains in the same community as the System asset. Example  Show examples Path Orders and domain Domain B In this case, the Orders Database asset and all its related assets will be integrated in Domain B. Path Orders > fk* and domain Domain B In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas that start with fk and their related assets will be integrated in Domain B. Path Orders > * and domain Domain B In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas in the Orders catalog and their related assets will be integrated in Domain B. Show full scenario You have a database Orders that includes multiple schemas. If you want to make sure that the Orders database and related schemas are added to domain B, add the following include mappings: Path Orders and domain Domain B, to make sure the Database asset is added to Domain B. Path Orders > * and domain Domain B, to make sure all Schema assets in Orders are added to Domain B. If you want to make sure that the Orders database and related schemas are added to domain B, except for the schemas that start with test_, add the following include mappings: Path Orders and domain Domain B, to make sure the Database asset is added to Domain B. Path Orders > test_* and domain Domain C, to make sure that all schemas in Order that start with test_ are added to Domain C. Path Orders > * and domain Domain B, to make sure all other Schema assets in Orders are added to Domain B.
       Domain Exclude Mappings	Metadata	Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. Tip You can use the ? and * wildcards in the catalog and schema names. For example:* > test.
       Extensible Properties Mappings	Metadata	Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects. Optionally, in Extensible Properties Mappings, specify which additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into Collibra. You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in Collibra assets. You do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute. Important  If you use this feature, make sure to add all required characteristics to the asset type assignments. The name of the property starts with the object type, for example catalogs.systemAttributes.metastore_id. catalogs refers to Database assets, schemas to Schema assets, table to Table assets, and views to Database View assets. The following system properties are supported:   Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by". Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by". Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Show steps to add a property mapping Click Add Another Mapping. In Property Name, do one of the following: To add a system attribute, select the Databricks Unity Catalog property name from the drop-down list. To add a custom attribute, type the name of the custom property manually. Use the following naming convention: [object type].customParameters.[name of parameter]. For example: catalogs.customParameters.Parameter1 schemas.customParameters.catalogAndNamespace.part.1 table.customParameters.view.catalogAndNamespace.part.2 views.customParameters.Paramerer2 In Attribute, select the attribute in which you want to see the value.
       Stop Compute Resource	Metadata	This field is important if the Compute Resource HTTP Path field was completed in the Databricks capability to allow for source tag integration. If this field is set to Yes, the compute resource in Databricks Unity Catalog will be stopped right after the source tags are extracted. If this field is set to No, the compute resource remains active. Tip To prevent clusters from running for the entire synchronization duration, you can also configure the Terminate after ... minutes of inactivity setting in Databricks. The setting ensures that clusters automatically stop after a period of inactivity. For more information, go to the Databricks documentation.
       Domain	AI models	In Domain, select the domain in which you want to add the Databricks AI Model assets.
       Custom AI Metrics Mappings	AI models	Optionally, in Custom AI Metrics Mappings, define which custom Databricks AI Model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data. Important  If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment. Show steps to add a custom AI metrics mapping Click Add Custom AI Metrics Mappings. In Metric, type the name of the custom metric manually. Use the exact name as in Databricks Unity Catalog. In Attribute, select the attribute in which you want to see the value. Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
       Exclude system AI Models	AI models	Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models. By default, No is selected, and all accessible AI models are integrated. If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration. For more information about these pretrained Databricks AI models, go to the Databricks documentation.

       Important 

       When you integrate a data source without applying Include or Exclude Mappings rules, and then later exclude a registered asset using an Include or Exclude Mapping during resynchronization, the related assets will receive the Missing from Source status.

   11. Click Save Configuration.
   12. Click Synchronize.
       A notification indicates the synchronization has started.