Create a Databricks connection to an Edge or Collibra Cloud site

Before you can synchronize Databricks Unity Catalog via Edge, you need to prepare your Edge or Collibra Cloud site. If you have defined an outbound (forward) proxy on your Edge site, the integration considers that configuration when connecting to the data source.

Ensure that you have completed all the prerequisites in your Databricks Unity Catalog and Collibra environments. Once you have created a connection, you can then proceed with metadata ingestion.

Do you use a vault?

You can use a vault to add your data source information to your Edge site connection.

Check the connection property table below to see which information is available for your vault.

Vaults are not available for Collibra Cloud site sites.
No vault
AWS Secrets Manager
Azure Key Vault
CyberArk Vault
Google Secret Manager
HashiCorp Vault
  
To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the query value to identify the secret in your vault.
    Example 
    Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

    If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

    For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Secret Engine Type
    Select one of the following:
    • Key Value
    • Database
    Engine Path
    The engine path to your vault where the value is stored.
    Secret Path
    The secret path to your vault where the value is stored.
    Field
    If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
    Role
    If your Secret Engine Type is Database, enter the role specified in the Database engine.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Vault Name
    The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
    Secret Name
    The name of the secret in your vault where the value is stored.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Secret Name
    The name of the secret in your vault where the value is stored.
    Field
    If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
    Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the name of the secret in your vault where the value is stored.
    Example 

Prerequisites

In your Collibra environment

  • You either created and installed an Edge site or were granted a Collibra Cloud site.
  • You have added a vault to your Edge site.
    Note  Vaults are not supported on Collibra Cloud sites.
  • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.
  • You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.

In your Databricks environment

  • Your Databricks access token or OAuth client must have the BROWSE permission on the catalogs in Databricks Unity Catalog from which you want to integrate metadata. For more information on the BROWSE permission, go to the Databricks documentation.
  • If you want to integrate source tags, additional permissions are needed.
    • The metadata synchronization for Databricks Unity Catalog uses compute clusters (SQL query compute warehouse) to collect source tags. To allow this, grant the following permissions:
      • CAN ATTACH TO
      • CAN RESTART
      During the synchronization configuration, you can define that the compute clusters must stop after the source tags are extracted.
    • To integrate source tags from specific tables in system.information_schema, grant the following permissions:
      • USE CATALOG permission on system catalog
      • USE SCHEMA permission on system.information_schema
    • SELECT permission on the following:
      • system.information_schema.catalog_tags
      • system.information_schema.schema_tags
      • system.information_schema.table_tags
      • system.information_schema.column_tags
  • If you want to integrate Databricks AI models, ensure that your Databricks access token or OAuth client also has the following permissions:
    • EXECUTE permission on the registered model.
    • USE CATALOG permission on the parent catalog.
    • USE SCHEMA permission on the parent schema.

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the table, click the name of the site whose status is Healthy.
      The site page opens.
  2. In the Connections section, click Create connection.
  3. Select Databricks to connect to Databricks.
    The Create connection page appears.
  4. Enter the required information.
    FieldDescriptionRequiredAvailable for Vaults?
    Name

    The name of the Edge or Collibra Cloud site connection for Databricks.

    		 Yes No
    Description

    The description of the connection.

    		 No No
    Vault The vault where you store your data source values. No No
    Workspace URL

    Enter the URL of any Databricks workspace connected to Unity Catalog that you want to integrate.
    To retrieve the URL, log into Databricks and copy the URL. For example: https://123.cloud.databricks.com.

    		 Yes Yes
    Authentication Type

    Select the type of authentication that you want to apply. You can select any of the following values:

    		 Yes No
    Access Token

    The security token that was generated in Databricks for the workspace. The access token must be a personal access token (PAT).
    It is possible to generate a PAT for service principals. For information on the service principal token, go to the Databricks documentation.

    Note Ensure that your Databricks access token has been granted the required permissions in your Databricks environment.

    		 Yes, if you select Personal Access Token as the authentication type. Yes
    Client ID

    The client ID for OAuth-based authentication in Databricks, or the client ID of the Microsoft Entra ID service principal.

    For information on OAuth-based authentication in Databricks Unity Catalog, go to the Databricks documentation.

    For information on the Microsoft Entra ID service principal, go to Microsoft Entra service principal authentication in the Azure Databricks documentation.

    Note Ensure that your Databricks OAuth client or Microsoft Entra ID service principal has been granted the required permissions in your Databricks environment.

    		 Yes, if you select OAuth or Microsoft Entra ID as the authentication type. No
    Client Secret

    The client secret generated for the OAuth-based authentication on Databricks, or the client secret of the Microsoft Entra ID service principal.

    		 Yes, if you selectOAuth or Microsoft Entra ID as the authentication type. No
    Tenant ID

    The Directory (tenant) ID for the related application registered in Microsoft Entra ID.

    For information, go to MS Entra service principal authentication in the Azure Databricks documentation.

    		Yes, if you select Microsoft Entra ID as the authentication type. No
  5. Click Create.
    The connection is added to the Edge or Collibra Cloud site.

Collibra validates the credentials when synchronizing Databricks Unity Catalog.

What's next

If you want to allow for sampling, profiling, and classification, create a Databricks JDBC connection. If you created a Databricks JDBC connection, you can use that JDBC connection when you configure the synchronization page. You can then add the Databricks Unity Catalog capability to an Edge or Collibra Cloud site.

To create technical lineage for Databricks Unity Catalog, go to Create a technical lineage via Edge.