Create a dbt connection

For CollibraData Lineage to connect to and retrieve metadata from dbt Cloud, create a dbt connection.

Before you begin

  • Create an Edge site on K3S.
  • You have added a vault to your Edge site.
    Note  Vaults are not supported on Collibra Cloud sites.
  • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

Prerequisites

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the site overview, click the name of a site.
      The site page appears.
  2. In the Connections section, click Create Connection.
    The Create Connection dialog box appears.
  3. Select dbt connection.
  4. Enter the connection information.
    FieldDescriptionRequired

    Name

    		The name of the connection.

    Yes

    Description

    		 The description of the connection. This field is also visible when you register content.

    No

    Vault The vault where you store your data source values. No

    Admin URL

    The endpoint of the dbt Cloud Administrative API that Collibra Data Lineage uses to download job artifacts. This is typically the Access URL associated with your dbt Cloud account.

    1. Log in to dbt Cloud as an Administrator.
    2. Navigate to Account Settings.
    3. In the Access URLs section, look for the Access URL field.

    The default value for this field is https://cloud.getdbt.com/api/v2. This value applies only to the North America multi-tenant region. For more information, go to dbt Cloud API v2 in the dbt documentation.

    This field is used only if you do not enter a value for the Environment Ids field in the Technical Lineage for dbt Cloud capability.

    If you enter values for both the Admin URL and Environment Ids fields, the Environment Ids field takes precedence.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
      Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

      If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

      For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Engine Type
      Select one of the following:
      • Key Value
      • Database
      Engine Path
      The engine path to your vault where the value is stored.
      Secret Path
      The secret path to your vault where the value is stored.
      Field
      If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
      Role
      If your Secret Engine Type is Database, enter the role specified in the Database engine.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Vault Name
      The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret Name
      The name of the secret in your vault where the value is stored.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Name
      The name of the secret in your vault where the value is stored.
      Field
      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 

    No

    Metadata URL

    The endpoint of the dbt Cloud Discovery API. This is typically the Semantic Layer GraphQL API URL associated with your dbt Cloud account.

    1. Log in to dbt Cloud as an Administrator.
    2. Navigate to Account Settings.
    3. In the Access URLs section, look for the Semantic Layer GraphQL API URL field.

    The default value is https://metadata.cloud.getdbt.com/graphql. For details, go to Query the Discovery API in dbt documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
      Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

      If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

      For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Engine Type
      Select one of the following:
      • Key Value
      • Database
      Engine Path
      The engine path to your vault where the value is stored.
      Secret Path
      The secret path to your vault where the value is stored.
      Field
      If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
      Role
      If your Secret Engine Type is Database, enter the role specified in the Database engine.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Vault Name
      The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret Name
      The name of the secret in your vault where the value is stored.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Name
      The name of the secret in your vault where the value is stored.
      Field
      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 

    No

    Token Name

    The name of the service token. It can be any unique meaningful name.

    1. Generate a Service token and ensure that you set the Read-Only permissions for CollibraData Lineage to work properly.
    2. Copy the token value when you save the service token.

    For details, go to Generating service account tokens in dbt documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
      Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

      If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

      For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Engine Type
      Select one of the following:
      • Key Value
      • Database
      Engine Path
      The engine path to your vault where the value is stored.
      Secret Path
      The secret path to your vault where the value is stored.
      Field
      If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
      Role
      If your Secret Engine Type is Database, enter the role specified in the Database engine.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Vault Name
      The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret Name
      The name of the secret in your vault where the value is stored.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Name
      The name of the secret in your vault where the value is stored.
      Field
      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 
    		 Yes

    Token Value

    Enter the service token.

    Tip You can select To be encrypted by Edge management server or Encrypted with public key to indicate the encryption method.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
      Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

      If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

      For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Engine Type
      Select one of the following:
      • Key Value
      • Database
      Engine Path
      The engine path to your vault where the value is stored.
      Secret Path
      The secret path to your vault where the value is stored.
      Field
      If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
      Role
      If your Secret Engine Type is Database, enter the role specified in the Database engine.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Vault Name
      The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret Name
      The name of the secret in your vault where the value is stored.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Name
      The name of the secret in your vault where the value is stored.
      Field
      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 
    		 Yes
  5. Click Create.

What's next

Add the Technical Lineage for dbt Cloud capability to your Edge or Collibra Cloud site.