Ways to work with Databricks

Combining the ways of working with Databricks

The 2 possibilities don't cancel each other out. You can use both ways to show the information you want in Collibra Platform Self-Hosted. You can use the integration of Databricks Unity Catalog to quickly get an overview of all your Databricks databases in Collibra Platform Self-Hosted. Once you have a better view on the important databases, you can register them individually via the JDBC driver.

Important Use the same System asset for the integration and the registration. Otherwise, assets will be duplicated.

Switching to working exclusively with the Databricks Unity Catalog integration (in preview)

If you previously used both the Databricks Unity Catalog integration and the Databricks JDBC synchronization for some databases, and you want to switch to using only the Databricks Unity Catalog integration, complete the following steps:

1. Update the existing Databricks Unity Catalog synchronization capability.

   1. Go to the existing Databricks Unity Catalog synchronization capability.
   2. Click Edit.
   3. Add the existing Databricks JDBC connection to the JDBC Databricks Connection field.
   4. Click Save.
2. Re-synchronize the Databricks Unity Catalog integration.
   Show how to synchronize Databricks Unity Catalog

   1. On the main toolbar, click → Catalog.
      The Catalog homepage opens.
   2. In the tab bar, click Integrations.
      The Integrations page opens.
   3. Click the Integration Configuration tab.
   4. Locate the Databricks connection that you used when you added the Databricks Unity Catalog capability and click the link in the Capabilities column.
      The synchronization configuration page opens.
   5. In the Synchronization Configuration section, click the Edit icon.
   6. In Ingestion Type, select what you want to integrate.
      The available options are: metadata, AI models, and metadata and AI models.

      Depending on your selection, extra fields appear. Your selection will also impact the integrated Databricks Unity Catalog data.
      

   7. Complete the fields as needed.

      Field	Available if you integrate	Action
      System	Metadata	In System, select the System asset in which you want to link the Databricks assets. Important If you previously ran a Databricks Unity Catalog integration synchronization, use the same system to ensure a successful integration.
      Default Asset Status (Deprecated)	Metadata AI models	In Default Asset Status, select how you want to set the status of the synchronized assets. The possible values are: Implemented: Implemented means that all assets receive the Implemented status. No Status: No status means that newly created assets receive the first status listed in your Operating Model statuses, and that existing assets keep their assigned status. This field is deprecated and will be removed in the future. You can now define the default status in the capability configuration. Ensure that the value in this field matches the one in the capability configuration, as this field still takes precedence over the capability value.
      Version	Metadata	Select one of the following values to determine how schemas are integrated into domains: V0 All schemas are included in domain mappings. Schemas that are not explicitly mapped must be excluded manually using the domain exclude mapping. V1 Only schemas explicitly mapped are included. All other schemas are automatically excluded. With this option, you do not need to exclude schemas you do not want to integrate. Example  If you specify the following mappings in the Domain Include Mappings field: Path Orders and domain Domain A Path Orders > fk* and domain Domain B Order is the database, and fk* is any schema with a name that starts with fk. If you select V0, the Order database and all other schemas that do not start with fk are integrated into Domain A. All schemas starting with fk are integrated into Domain B. You must manually exclude any additional schemas you do not want. If you select V1, the Order database is integrated into Domain A, and all schemas starting with fk are integrated into Domain B. Schemas not explicitly mapped are automatically excluded, so you do not need to specify Domain Exclude Mappings.
      Domain Include Mappings	Metadata	Optionally, in Domain Include Mappings, specify which databases and schemas you want to integrate and optionally the CPSH domains where they need to be added. This means you can use this field to limit the databases and schemas you integrate and to define where they need to be added. Important  If you don't define include any mappings, the integration automatically creates new domains for each Database and Schema asset in the same community as the System asset. For more information, go to Integrated Databricks Unity Catalog data. If you include a path but don't define a domain, the integration automatically creates new domains in the same community as the System asset. If you selected V0 for the Version field and add a domain include mapping for the database but not for a related schema, the automatically created domain for the schema is added in the same community as the domain of the database. A match with a schema has priority over a match with a database. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the databases and schemas in Databricks Unity Catalog for which you want to integrate the metadata. Tip  You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account. Optionally, in Domain, select the CPSH domain in which you want to integrate the metadata. If you don't define a domain, the integration automatically creates new domains in the same community as the System asset. Example  Show examples Path Orders and domain Domain B In this case, the Orders Database asset and all its related assets will be integrated in Domain B. Path Orders > fk* and domain Domain B In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas that start with fk and their related assets will be integrated in Domain B. Path Orders > * and domain Domain B In this case, the Orders Database asset will be integrated in the same domain as the System asset. All schemas in the Orders catalog and their related assets will be integrated in Domain B. Show full scenario You have a database Orders that includes multiple schemas. If you want to make sure that the Orders database and related schemas are added to Domain B, add the following include mappings: Path Orders and domain Domain B, to make sure the Database asset is added to Domain B. Path Orders > * and domain Domain B, to make sure all Schema assets in Orders are added to Domain B. If you want to make sure that the Orders database and related schemas are added to domain B, except for the schemas that start with test_, add the following include mappings:  Path Orders and domain Domain B, to make sure the Database asset is added to Domain B. Path Orders > test_* and domain Domain C, to make sure that all schemas in Order that start with test_ are added to Domain C. Path Orders > * and domain Domain B, to make sure all other Schema assets in Orders are added to Domain B.
      Domain Exclude Mappings	Metadata	Optionally, in Domain Exclude Mappings, specify the path to databases and schemas in Databricks Unity Catalog that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to the databases and schemas in Databricks Unity Catalog that you want to exclude. Tip You can use the ? and * wildcards in the catalog and schema names. For example:* > test.
      Extensible Properties Mappings	Metadata	Via the Extensible Properties Mapping field, Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects. Optionally, in Extensible Properties Mappings, specify which additional default system properties or custom properties that you want to integrate from Databricks Unity Catalog into CPSH. You can integrate most values from the Details page from Catalog, Schema, Table, and View objects into specific attributes in CPSH assets. You do this by adding the mapping between the fields for the objects in Databricks Unity Catalog and the Collibra attribute. Important  If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment. The name of the property starts with the object type, for example catalogs.systemAttributes.metastore_id. catalogs refers to Database assets, schemas to Schema assets, table to Table assets, and views to Database View assets. The following system properties are supported:   Catalogs: "browse_only", "catalog_type", "connection_name", "created_at", "created_by", "isolation_mode", "metastore_id", "provider_name", "provisioning_info", "securable_kind", "securable_type", "share_name", "storage_location", "storage_root", "updated_at" , and "updated_by". Schemas: "catalog_type", "created_at", "created_by", "metastore_id", "securable_type", "securable_kind", "storage_location", "storage_root", "updated_at", and "updated_by". Table: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Views: "access_point", "catalog_name", "created_at", "created_by", "data_access_configuration_id", "data_source_format", "deleted_at", "metastore_id", "schema_name", "securable_type", "securable_kind", "sql_path", "storage_credential_name", "storage_location", "table_type", "updated_at", "updated_by", and "view_definition". Show steps to add a property mapping Click Add Another Mapping. In Property Name, do one of the following: To add a system attribute, select the Databricks Unity Catalog property name from the drop-down list. To add a custom attribute, type the name of the custom property manually. Use the following naming convention: [object type].customParameters.[name of parameter]. For example: catalogs.customParameters.Parameter1 schemas.customParameters.catalogAndNamespace.part.1 table.customParameters.view.catalogAndNamespace.part.2 views.customParameters.Paramerer2 In Attribute, select the attribute in which you want to see the value.
      Stop Compute Resource	Metadata	This field is important if the Compute Resource HTTP Path field was completed in the Databricks capability to allow for source tag integration. If this field is set to Yes, the compute resource in Databricks Unity Catalog will be stopped right after the source tags are extracted. If this field is set to No, the compute resource remains active. Tip To prevent clusters from running for the entire synchronization duration, you can also configure the Terminate after ... minutes of inactivity setting in Databricks. The setting ensures that clusters automatically stop after a period of inactivity. For more information, go to the Databricks documentation.
      Domain	AI models	In Domain, select the domain in which you want to add the Databricks AI Model assets.
      Custom AI Metrics Mappings	AI models	Optionally, in Custom AI Metrics Mappings, define which custom Databricks AI Model metrics you want to integrate. You do this by adding the mapping between the custom metric and the Collibra attribute. Show available custom metrics accuracy_score, exact_match example_count f1_score f1_score_micro f1_score_macro false_negatives false_positives log_loss max_error mean_absolute_error mean_absolute_percentage_error mean_on_target mean_squared_error precision precision_recall_auc r2_score recall roc_auc root_mean_squared_error sum_on_target token_count true_negatives true_positives For an overview of the out-of-the-box metrics we integrate by default, go to Integrated Databricks Unity Catalog data. Important  If you use this feature, make sure to add any custom attributes/characteristics, as needed, to the asset type assignment. Show details Click Add Custom AI Metrics Mappings. In Metric, select the custom metric from the list of available Databricks AI metrics. In Attribute, select the attribute in which you want to see the value. Make sure to select an attribute that is included in the Databricks AI Model asset type assignment.
      Exclude system AI Models	AI models	Optionally, in Exclude system AI models, indicate that you don't want to integrate the pretrained Databricks AI models. By default, No is selected, and all accessible AI models are integrated. If you select Yes, the AI models in the "system" Databricks catalog will be excluded from the integration. For more information about these pretrained Databricks AI models, go to the Databricks documentation.

      Important 

      When you integrate a data source without applying Include or Exclude Mappings rules, and then later exclude a registered asset using an Include or Exclude Mapping during resynchronization, the related assets will receive the Missing from Source status.

   8. Click Save.
   9. Click Synchronize.
      A notification indicates the synchronization has started.