Create a Databricks connection to an Edge site

Important 

In Collibra 2024.02, we've launched a new user interface (UI) in beta for Collibra Data Intelligence Platform! You can learn more about this latest UI in the UI overview.

Use the following options to see the documentation in the latest UI or in the previous, classic UI:

Before you begin

  • You have created and installed an Edge site.
  • The integration of Databricks via Edge has been enabled.
  • You have given the Edge Site role the required permissions.
  • Your Databricks access token has the following minimum permissions:
    • USE CATALOG permission for catalogs.
    • USE SCHEMA permission for schemas.
    • SELECT permission for tables/views belonging to the catalogs and schemas.

Required permissions

Steps

  1. Open an Edge site.
    1. On the main toolbar, click Products icon, and then click Cogwheel icon Settings.
      The Collibra settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of the Edge sites.
    3. In the table, click the name of the Edge site whose status is Healthy.
      The Edge site page opens.
  2. In the Connections section, click Create connection.
    The Create connection page appears.
  3. Enter the required information.
    FieldDescriptionRequired

    Connection settings

    This section contains the general settings of your connection.

    Name

    The name of the Edge connection for Databricks.

    		 Yes
    Description

    The description of the connection.

    		 No
    Connection provider

    The connection provider, which determines the available connection parameters.

    Select Databricks to connect to Databricks.

    		Yes

    Connection parameters

    		This section contains the settings to connect to your data source.
    Workspace URL

    Enter the URL of any Databricks workspace connected to Unity Catalog that you want to integrate.
    To retrieve the URL, log into Databricks and copy the URL. For example: https://123.cloud.databricks.com.

    		 Yes
    Access Token

    The security token that was generated in Databricks for the workspace.

    The access token can be a personal access token or service principle token. We recommend using a service principle token. For information on the service principle token, go to the Databricks documentation.

    Note 

    The user / token must have the following minimum permissions:

    • USE CATALOG permission for catalogs.
    • USE SCHEMA permission for schemas.
    • SELECT permission for tables/views belonging to the catalogs and schemas.
    		 Yes
    Encryption options

    Select the type of encryption used to store the Secret Access Key.

    Default: To be encrypted by Edge management server.

    		 Yes
  4. Click Create.
    The connection is added to the Edge site.

What's next?

You can now add the Databricks Unity Catalog capability to an Edge site.

Available vaults

Tip 

You can use a vault to add your data source information to your Edge site connection.
None
AWS Secrets Manager
Azure Key Vault
CyberArk Vault
Google Secret Manager
HashiCorp Vault
  

Before you begin

  • You have created and installed an Edge site.
  • The integration of Databricks via Edge has been enabled.
  • You have given the Edge Site role the required permissions.
  • You have added a vault to your Edge site.
  • Your Databricks access token has the following minimum permissions:
    • USE CATALOG permission for catalogs.
    • USE SCHEMA permission for schemas.
    • SELECT permission for tables/views belonging to the catalogs and schemas.

Required permissions

Steps

  1. Open an Edge site.
    1. On the main toolbar, click Products icon, and then click Cogwheel icon Settings.
      The Collibra settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of the Edge sites.
    3. In the table, click the name of the Edge site whose status is Healthy.
      The Edge site page opens.
  2. In the Connections section, click Create connection.
  3. Select Databricks to connect to Databricks.
    The Create connection page appears.
  4. Enter the required information.
    FieldDescriptionRequired
    Name

    The name of the Edge connection for Databricks.

    		 Yes
    Description

    The description of the connection.

    		 No
    Vault The vault where you store your data source values. No
    Workspace URL

    Enter the URL of any Databricks workspace connected to Unity Catalog that you want to integrate.
    To retrieve the URL, log into Databricks and copy the URL. For example: https://123.cloud.databricks.com.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Secret Engine Type

      Select one of the following:

      • Key Value
      • Database
      Engine PathThe engine path to your vault where the value is stored.
      Secret PathThe secret path to your vault where the value is stored.
      Field

      The name of the field to your vault where the value is stored.

      Note Only available if you selected Key Value in the Secret Engine Type field.

      Role

      The role specified in the Database engine.

      Note Only available if you selected Database in the Secret Engine Type field.

      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Vault NameThe name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret NameThe name of the secret in your vault where the value is stored.
      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Secret NameThe name of the secret in your vault where the value is stored.
      Field

      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.

      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.

      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 
    		 Yes
    Access Token

    The security token that was generated in Databricks for the workspace.

    The access token can be a personal access token or service principle token. We recommend using a service principle token. For information on the service principle token, go to the Databricks documentation.

    Note 

    The user / token must have the following minimum permissions:

    • USE CATALOG permission for catalogs.
    • USE SCHEMA permission for schemas.
    • SELECT permission for tables/views belonging to the catalogs and schemas.
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Secret Engine Type

      Select one of the following:

      • Key Value
      • Database
      Engine PathThe engine path to your vault where the value is stored.
      Secret PathThe secret path to your vault where the value is stored.
      Field

      The name of the field to your vault where the value is stored.

      Note Only available if you selected Key Value in the Secret Engine Type field.

      Role

      The role specified in the Database engine.

      Note Only available if you selected Database in the Secret Engine Type field.

      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Vault NameThe name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret NameThe name of the secret in your vault where the value is stored.
      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      NameDescription
      Secret NameThe name of the secret in your vault where the value is stored.
      Field

      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.

      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.

      Example 
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 
    		 Yes
  5. Click Create.
    The connection is added to the Edge site.

What's next?

You can now add the Databricks Unity Catalog capability to an Edge site.