Create a Databricks connection to an Edge or Collibra Cloud site

Before you can synchronize Databricks Unity Catalog via Edge, you need to prepare your Edge or Collibra Cloud site. If you have defined an outbound (forward) proxy on your Edge site, the integration considers that configuration when connecting to the data source.

Ensure that you have completed all the prerequisites in your Databricks Unity Catalog and Collibra environments. Once you have created a connection, you can then proceed with metadata ingestion.

Do you use a vault?

You can use a vault to add your data source information to your Edge site connection.

Check the connection property table below to see which information is available for your vault.

Vaults are not available for Collibra Cloud site sites.

No vault

AWS Secrets Manager

Azure Key Vault

CyberArk Vault

Google Secret Manager

HashiCorp Vault

I don't use a vault
AWS Secrets Manager
Azure Key Vault
CyberArk Vault
Google Secret Manager
HashiCorp Vault

How to use your vault...

To use your vault, do the following:

In the Value Type field, select Vault Key.
Enter the query value to identify the secret in your vault.
Example
Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>
If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets
For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

To use your vault, do the following:

In the Value Type field, select Vault Key.
Enter the required information:
Secret Engine Type
Select one of the following:
- Key Value
- Database
Engine Path
The engine path to your vault where the value is stored.
Secret Path
The secret path to your vault where the value is stored.
Field
If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
Role
If your Secret Engine Type is Database, enter the role specified in the Database engine.
Example

To use your vault, do the following:

In the Value Type field, select Vault Key.
Enter the required information:
Vault Name
The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
Secret Name
The name of the secret in your vault where the value is stored.
Example

To use your vault, do the following:

In the Value Type field, select Vault Key.
Enter the required information:
Secret Name
The name of the secret in your vault where the value is stored.
Field
If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
Example

To use your vault, do the following:

In the Value Type field, select Vault Key.
Enter the name of the secret in your vault where the value is stored.
Example

Prerequisites

In your Collibra environment

You either created and installed an Edge site or were granted a Collibra Cloud site.

You have added a vault to your Edge site.

Note Vaults are not supported on Collibra Cloud sites.

If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

In your Databricks environment

Your Databricks access token or OAuth client must have the BROWSE permission on the catalogs in Databricks Unity Catalog from which you want to integrate metadata. For more information on the BROWSE permission, go to the Databricks documentation.
If you want to integrate source tags, additional permissions are needed.
- The metadata synchronization for Databricks Unity Catalog uses compute clusters (SQL query compute warehouse) to collect source tags. To allow this, grant the following permissions:
  - CAN ATTACH TO
  - CAN RESTART
  During the synchronization configuration, you can define that the compute clusters must stop after the source tags are extracted.
- To integrate source tags from specific tables in system.information_schema, grant the following permissions:
  - USE CATALOG permission on system catalog
  - USE SCHEMA permission on system.information_schema
- SELECT permission on the following:
  - system.information_schema.catalog_tags
  - system.information_schema.schema_tags
  - system.information_schema.table_tags
  - system.information_schema.column_tags
If you want to integrate Databricks AI models, ensure that your Databricks access token or OAuth client also has the following permissions:
- EXECUTE permission on the registered model.
- USE CATALOG permission on the parent catalog.
- USE SCHEMA permission on the parent schema.

Steps

Open a site.
1. On the main toolbar, click → Settings.
  The Settings page opens.
2. In the tab pane, click Edge.
  The Sites tab opens and shows a table with an overview of your sites.
3. In the table, click the name of the site whose status is Healthy.
  The site page opens.
In the Connections section, click Create connection.
Select Databricks to connect to Databricks.
The Create connection page appears.

Enter the required information.

Field	Description	Required	Available for Vaults?
Name	The name of the Edge or Collibra Cloud site connection for Databricks.	Yes	No
Description	The description of the connection.	No	No
Vault	The vault where you store your data source values.	No	No
Workspace URL	Enter the URL of any Databricks workspace connected to Unity Catalog that you want to integrate. To retrieve the URL, log into Databricks and copy the URL. For example: https://123.cloud.databricks.com.	Yes	Yes
Authentication Type	Select the type of authentication that you want to apply. You can select any of the following values: Personal Access Token OAuth For information on OAuth-based authentication in Databricks Unity Catalog, go to the Databricks documentation. Microsoft Entra ID For information, go to MS Entra service principal authentication in the Azure Databricks documentation.	Yes	No
Access Token	The security token that was generated in Databricks for the workspace. The access token must be a personal access token (PAT). It is possible to generate a PAT for service principals. For information on the service principal token, go to the Databricks documentation. Note Ensure that your Databricks access token has been granted the required permissions in your Databricks environment.	Yes, if you select `Personal Access Token` as the authentication type.	Yes
Client ID	The client ID for OAuth-based authentication in Databricks, or the client ID of the Microsoft Entra ID service principal. For information on OAuth-based authentication in Databricks Unity Catalog, go to the Databricks documentation. For information on the Microsoft Entra ID service principal, go to Microsoft Entra service principal authentication in the Azure Databricks documentation. Note Ensure that your Databricks OAuth client or Microsoft Entra ID service principal has been granted the required permissions in your Databricks environment.	Yes, if you select `OAuth` or `Microsoft Entra ID` as the authentication type.	No
Client Secret	The client secret generated for the OAuth-based authentication on Databricks, or the client secret of the Microsoft Entra ID service principal.	Yes, if you select`OAuth` or `Microsoft Entra ID` as the authentication type.	No
Tenant ID	The Directory (tenant) ID for the related application registered in Microsoft Entra ID. For information, go to MS Entra service principal authentication in the Azure Databricks documentation.	Yes, if you select `Microsoft Entra ID` as the authentication type.	No

Click Create.
The connection is added to the Edge or Collibra Cloud site.

Collibra validates the credentials when synchronizing Databricks Unity Catalog.

What's next

If you want to allow for sampling, profiling, and classification, create a Databricks JDBC connection. If you created a Databricks JDBC connection, you can use that JDBC connection when you configure the synchronization page. You can then add the Databricks Unity Catalog capability to an Edge or Collibra Cloud site.

To create technical lineage for Databricks Unity Catalog, go to Create a technical lineage via Edge.