Create a technical lineage via Edge

Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

You can also use the Collibra Catalog Cloud Ingestions API to create or update a technical lineage capability and start or schedule a synchronization to create a technical lineage. For more information about using APIs, go to Collibra Developer Portal.

Tip 

Select a data source and the connection type if needed to see the related information.

Currently, the information is shown for:

X
Important 
  • Metadata and asset details are ingested, but technical lineage is not yet available.
  • This integration is in preview. If you’d like to test the new integration, go to the Collibra Preview Portal and get started.

Do you use a vault?

You can use a vault to add your data source information to your Edge site connection.

Check the connection property table below to see which information is available for your vault.

Vaults are not available for Collibra Cloud site sites.
No vault
AWS Secrets Manager
Azure Key Vault
CyberArk Vault
Google Secret Manager
HashiCorp Vault
  
To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the query value to identify the secret in your vault.
    Example 
    Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

    If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

    For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Secret Engine Type
    Select one of the following:
    • Key Value
    • Database
    Engine Path
    The engine path to your vault where the value is stored.
    Secret Path
    The secret path to your vault where the value is stored.
    Field
    If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
    Role
    If your Secret Engine Type is Database, enter the role specified in the Database engine.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Vault Name
    The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
    Secret Name
    The name of the secret in your vault where the value is stored.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the required information:
    Secret Name
    The name of the secret in your vault where the value is stored.
    Field
    If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
    Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
    Example 

To use your vault, do the following:
  1. In the Value Type field, select Vault Key.
  2. Enter the name of the secret in your vault where the value is stored.
    Example 

Note If you're using a Collibra Cloud site, go the Collibra Cloud site documentation to check if your data source is supported.

Important Collibra Data Lineage support for Databricks Unity Catalog leverages the system tables feature in Databricks Unity Catalog. The system tables feature is in Public Preview. For details, go to Databricks Previews support & details in Databricks documentation.

Before you begin

Requirements and permissions

The following requirements and permissions are needed for the technical lineage process. Additional, Edge-related roles and resources, are mentioned in each of the specific steps.

Steps

Tip Be sure to check out our Extract Tableau Technical Lineage (Latest UI) course in Collibra University. This course guides you through the steps of this procedure.
Tip Be sure to check out our Extract Power BI Technical Lineage (Latest UI) course in Collibra University. This course guides you through the steps of this procedure.
  1. Enable technical lineage via Edge.
  2. Set up Tableau.
  3. Set up Power BI.
  4. Set up Looker.
  5. Set up SSRS-PBRS.
  6. Set up MicroStrategy.

  7. Use the following procedure to set up Fluentd.

      1. Determine the host and port where the Fluentd collector will run. This daemon needs to be up and running continuously to collect metadata.
      2. Open a port for HTTP so that the data source can send REST API events to this location. Ensure that the port is open for listening, and document the port number and host.

        You can use HTTPS for improved security. Generate an SSL certificate and configure Fluentd when using HTTPS. For more information, go to http in Flentd documentation.

      3. Download the Fluent package by following the guide for your system. For example, if you use Red Hat Linux, go to Install by RPM Package (Red Hat Linux) in the Fluentd documentation.
      4. Start the service and check the status. For example, you can enter the following command:
        - sudo systemctl start fluentd.service; sudo systemctl status fluentd.service
      5. Check the logs to ensure the service is running correctly. For example, if you use Red Hat Linux, you can find the log files at /var/log/fluent/fluentd.log.
      6. Test basic functionality by completing the following steps. Red Hat Linux is used as an example.
        1. The default configuration is set to receive logs at an HTTP endpoint on port 8888 and route them to stdout. On Red Hat Linux, the default configuration is at /etc/fluent/fluentd.conf.
        2. Enter the following command to test:
          curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/debug.test
        3. Check the outcome by viewing the latest log entry:
          tail -n 1 /var/log/fluent/fluentd.log
      7. Configure Fluentd for OpenLineage.

        The following sample configuration shows how to collect messages on port 8888 and save them to files grouped by 5-minute windows. For details, use Configuration in the Flentd documentation.

        <source>
  @type http
  port 8888
  json_array true
</source>
<match openlineage>
  @type file
  path /fluentd/logs/openlineage
  path_suffix .jsonl
  json_array true						
  <buffer>
    timekey 5m
    timekey_use_utc true
    timekey_wait 5m
  </buffer>
  append true
  <format>
    @type json
  </format>
</match>

        You can test the result after configuration by entering the following command:
        curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/openlineage

        And then check the outcome by entering the following command:
        ls -l /var/log/fluent/collect_packets

  8. Use this procedure to configure your software to emit OpenLineage messages and save the resulting files to a location accessible by Collibra.

    1. To install and configure the OpenLineage integration in Airflow, follow this guideline in the Airflow documentation: Using OpenLineage integration.

      You can use the following configuration as an example:

      [openlineage]

transport='{"type":"http", "url": "http://HOST_OR_URL_WHERE_FLUENTD_IS:8888/openlineage'

namespace = 'airflow'
    2. After Fluentd saves the lineage messages to files, on your Edge site, create a directory and note its path.
    3. Copy the files in OpenLineage format to the directory that you created in the previous step.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.
    4. Copy the files in OpenLineage format to the relevant directory in your cloud-based storage system. The files must be in one of the following:
      • An AWS S3 bucket.
      • An Azure Data Lake Storage container.
      • A Google Cloud Storage bucket.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.

  9. Use the following information to set up Fluentd.

      1. Determine the host and port where the Fluentd collector will run. This daemon needs to be up and running continuously to collect metadata.
      2. Open a port for HTTP so that the data source can send REST API events to this location. Ensure that the port is open for listening, and document the port number and host.

        You can use HTTPS for improved security. Generate an SSL certificate and configure Fluentd when using HTTPS. For more information, go to http in Flentd documentation.

      3. Download the Fluent package by following the guide for your system. For example, if you use Red Hat Linux, go to Install by RPM Package (Red Hat Linux) in the Fluentd documentation.
      4. Start the service and check the status. For example, you can enter the following command:
        - sudo systemctl start fluentd.service; sudo systemctl status fluentd.service
      5. Check the logs to ensure the service is running correctly. For example, if you use Red Hat Linux, you can find the log files at /var/log/fluent/fluentd.log.
      6. Test basic functionality by completing the following steps. Red Hat Linux is used as an example.
        1. The default configuration is set to receive logs at an HTTP endpoint on port 8888 and route them to stdout. On Red Hat Linux, the default configuration is at /etc/fluent/fluentd.conf.
        2. Enter the following command to test:
          curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/debug.test
        3. Check the outcome by viewing the latest log entry:
          tail -n 1 /var/log/fluent/fluentd.log
      7. Configure Fluentd for OpenLineage.

        The following sample configuration shows how to collect messages on port 8888 and save them to files grouped by 5-minute windows. For details, use Configuration in the Flentd documentation.

        <source>
  @type http
  port 8888
  json_array true
</source>
<match openlineage>
  @type file
  path /fluentd/logs/openlineage
  path_suffix .jsonl
  json_array true						
  <buffer>
    timekey 5m
    timekey_use_utc true
    timekey_wait 5m
  </buffer>
  append true
  <format>
    @type json
  </format>
</match>

        You can test the result after configuration by entering the following command:
        curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/openlineage

        And then check the outcome by entering the following command:
        ls -l /var/log/fluent/collect_packets

  10. Use this procedure to set up the Apache Spark integration to emit OpenLineage messages to Fluentd and save the resulting files to a location accessible by Collibra. For more information, go to Apache Spark in the OpenLineage documentation.

    1. Download and install OpenLineage Spark integration:

      1. Download the JAR file from Maven Central, and select the latest available version of the openlineage-spark package.
      2. Install the JAR file by adding it to the list of dependent JARs in your AWS Glue job.
    2. Set the properties in the SparkSession. You can set the properties in different ways. For details, go to Apache Spark Configuration Usage and Spark Config Parameters.
      Note Avoid using the file transport type to save data to an S3 bucket. S3 buckets store immutable objects, not files, which means that Java file operations will not function.
    3. Verify the integration. To ensure that the Apache Spark integration works, you can send events to console and view them in CloudWatch.

      The following example shows the configuration to send the data to the console. When the configuration is set, AWS Glue jobs are mapped to a log group under the CloudWatch service. Each job execution generates a job run ID. The logs associated with the job run can be found in this log group. If the integration works, the logs will contain JSON with OpenLineage information.

      import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.conf import SparkConf
from awsglue.context import GlueContext
from awsglue.job import Job
							
conf = SparkConf()
conf.set("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener")\
    .set("spark.jars.packages", "io.openlineage:openlineage-spark:1.8.0")\
    .set("spark.openlineage.version", "v1")\
    .set("spark.openlineage.namespace", "OL_EXAMPLE")\
    .set("spark.openlineage.transport.type", "console")
							
sc = SparkContext.getOrCreate(conf=conf)
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

    4. Integrate with Fluentd. After you verify that the lineage events are generated in the console, update the configuration to direct the metadata to your Fluentd collector.
      For additional configuration patterns, go to Examples in the Transport topic in the OpenLineage documentation. Use the Spark Config examples and specify the URL and port you defined when setting up Fluentd.

    5. After Fluentd saves the lineage messages to files, on your Edge site, create a directory and note its path.
    6. Copy the files in OpenLineage format to the directory that you created in the previous step.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.
    7. After Fluentd saves the lineage messages to files, copy the files in OpenLineage format to the relevant directory in your cloud-based storage system. The files must be in one of the following:
      • An AWS S3 bucket.
      • An Azure Data Lake Storage container.
      • A Google Cloud Storage bucket.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.

  11. You can create technical lineage for OpenLineage integration by using the Shared Storage connection on Edge.

    Ensure that your tool is configured to emit OpenLineage messages. You can configure your tool to emit OpenLineage messages by using a REST API, or to write OpenLineage messages directly to disk.

  12. Use this procedure to configure your tool to emit OpenLineage messages and save the resulting files to a directory on the Edge site.

    1. If your tool emits OpenLineage messages by using a REST API, complete the following steps to install Fluentd to collect the messages.
      If your tool emits OpenLineage messages directly to disk, you can skip the following steps.
        1. Determine the host and port where the Fluentd collector will run. This daemon needs to be up and running continuously to collect metadata.
        2. Open a port for HTTP so that the data source can send REST API events to this location. Ensure that the port is open for listening, and document the port number and host.

          You can use HTTPS for improved security. Generate an SSL certificate and configure Fluentd when using HTTPS. For more information, go to http in Flentd documentation.

        3. Download the Fluent package by following the guide for your system. For example, if you use Red Hat Linux, go to Install by RPM Package (Red Hat Linux) in the Fluentd documentation.
        4. Start the service and check the status. For example, you can enter the following command:
          - sudo systemctl start fluentd.service; sudo systemctl status fluentd.service
        5. Check the logs to ensure the service is running correctly. For example, if you use Red Hat Linux, you can find the log files at /var/log/fluent/fluentd.log.
        6. Test basic functionality by completing the following steps. Red Hat Linux is used as an example.
          1. The default configuration is set to receive logs at an HTTP endpoint on port 8888 and route them to stdout. On Red Hat Linux, the default configuration is at /etc/fluent/fluentd.conf.
          2. Enter the following command to test:
            curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/debug.test
          3. Check the outcome by viewing the latest log entry:
            tail -n 1 /var/log/fluent/fluentd.log
        7. Configure Fluentd for OpenLineage.

          The following sample configuration shows how to collect messages on port 8888 and save them to files grouped by 5-minute windows. For details, use Configuration in the Flentd documentation.

          <source>
  @type http
  port 8888
  json_array true
</source>
<match openlineage>
  @type file
  path /fluentd/logs/openlineage
  path_suffix .jsonl
  json_array true						
  <buffer>
    timekey 5m
    timekey_use_utc true
    timekey_wait 5m
  </buffer>
  append true
  <format>
    @type json
  </format>
</match>

          You can test the result after configuration by entering the following command:
          curl -X POST -d 'json={"json":"Here is my message"}' http://localhost:8888/openlineage

          And then check the outcome by entering the following command:
          ls -l /var/log/fluent/collect_packets

    2. On your Edge site, create a directory and note its path.
    3. Copy the files in OpenLineage format to the directory on the Edge site.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.
    4. Copy the files in OpenLineage format to the relevant directory in your cloud-based storage system. The files must be in one of the following:
      • An AWS S3 bucket.
      • An Azure Data Lake Storage container.
      • A Google Cloud Storage bucket.
      Note Whenever you synchronize lineage, you must upload all source files you want to include in the technical lineage graph.

  13. For Collibra Data Lineage to connect to and retrieve metadata from your data source, create a JDBC connection.

    If you have set up a JDBC connection when you registered the data source via Edge, you can use the existing JDBC connection.

    For details about creating a JDBC connection, go to Create a JDBC connection.

  14. For Collibra Data Lineage to connect to and retrieve metadata from Google Dataplex, create a GCP connection.

    Do you use a vault?

    You can use a vault to add your data source information to your Edge site connection.

    Check the connection property table below to see which information is available for your vault.

    Vaults are not available for Collibra Cloud site sites.
    No vault
    AWS Secrets Manager
    Azure Key Vault
    CyberArk Vault
    Google Secret Manager
    HashiCorp Vault
    		  
    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the query value to identify the secret in your vault.
      Example 
      Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

      If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

      For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Engine Type
      Select one of the following:
      • Key Value
      • Database
      Engine Path
      The engine path to your vault where the value is stored.
      Secret Path
      The secret path to your vault where the value is stored.
      Field
      If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
      Role
      If your Secret Engine Type is Database, enter the role specified in the Database engine.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Vault Name
      The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
      Secret Name
      The name of the secret in your vault where the value is stored.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the required information:
      Secret Name
      The name of the secret in your vault where the value is stored.
      Field
      If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
      Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
      Example 

    To use your vault, do the following:
    1. In the Value Type field, select Vault Key.
    2. Enter the name of the secret in your vault where the value is stored.
      Example 

    Prerequisites

    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the table, click the name of the site whose status is Healthy.
        The site page opens.
    2. In the Connections section, click Create connection.
      The Create connection page appears.
    3. Select the GCP connection to connect to Google Cloud Platform.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the Edge or Collibra Cloud site connection for Google Cloud Platform.

      		 Yes
      Description

      The description of the connection.

      		 No
      Vault The vault where you store your data source values. No
      Connection typeThe authentication method for your GCP connection. Select one of the following options:
      Service account
      Use a Google service account for authentication.
      Workload Identity Federation (WIF)
      Use Workload Identity Federation to authenticate without a service account key.
      Workload Identity Federation (WIF) using GKE
      Use Workload Identity Federation in Google Kubernetes Engine (GKE) to authenticate.
      Important 
      • When using a cloud provider for WIF, such as AWS or GCP, your Edge site must be deployed on the same cloud provider. For example, if the Edge site is installed on GCP, you cannot use AWS as the WIF provider.
      • Automatic token generation for WIF is supported if your Edge site is installed on AWS using K3s or GCP using GKE. If your Edge site is installed using K3s on any cloud provider other than AWS, automatic token generation is not supported.
      •  If you select Workload Identity Federation (WIF) using GKE, the following rules apply:
        • Only select this connection type if you have created a separate edge site on a GKE cluster in Google Cloud.
        • The Project IDs field is required when configuring synchronization.
        • Proxies are not supported.
        • Column-level lineage is not supported.
      		 Yes
      Service Account / Workload Identity Federation (WIF)Enter one of the following values:
      • For the Service Account authentication method, add the full content of the service account key JSON file. 
        {
        "type": "service_account",
        "project_id": "PROJECT_ID",
        "private_key_id": "KEY_ID",
        "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n",
        "client_email": "SERVICE_ACCOUNT_EMAIL",
        "client_id": "CLIENT_ID",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://accounts.google.com/o/oauth2/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL"}

        Ensure the service account has the required permissions, as defined in the prerequisites.
        For more information about service account keys, go to the Google documentation.

      • For the Workload Identity Federation (WIF) authentication method, enter the token URL or the token if you're using WIF with a file-based credential source.
      • For the Workload Identity Federation (WIF) using GKE authentication method, you can ignore this field.
      		 Yes if you selected the Service Account or Workload Identity Federation (WIF) authentication method
      Property

      If your connection to GCP requires any additional parameters, click Add Property.

      		No
    5. Click Create.
      The connection is added to the Edge or Collibra Cloud site.

  15. For Collibra Data Lineage to connect to and retrieve metadata from Azure Data Factory, create an Azure connection.

    Before you begin

    Create an Edge site on K3S.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select the Azure connection.
    4. Enter the connection information.
      FieldDescriptionRequired
      Name

      The name of the Edge connection for Azure Data Factory.

      		 Yes
      Description

      The description of the connection.

      		 No
      Vault The vault where you store your data source values. No
      Service Principal ID

      The Application account ID to connect to the Azure.
      For information on the Azure Service Principal user and the Application ID, go to the Azure documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Service Principal Secret

      If you want to use the Service Principal authentication type, enter the application secret for the Service Principal.
      For information on the application secret value, go to the Azure documentation.

      If you want to use the Resource Owner Password Credentials authentication type, enter the password.

      Ensure that you select the corresponding authentication type when you add the Technical Lineage for ADF capability.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes

      Tenant ID

      The directory ID of your Azure Data Factory instance.
      For information on the Directory (tenant) ID, go to the Azure documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

  16. If you integrated Databricks Unity Catalog, you had created a Databricks connection. You can use the Databricks connection when you add a technical lineage for Databricks Unity Catalog. If you registered your Databricks file system by using the JDBC connection instead, use this information to create a Databricks connection.

    Prerequisites

    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection and select Databricks connection in the Create Connection dialog box.
      The Create Connection dialog box for Databricks connection opens.
    3. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the Edge or Collibra Cloud site connection for Databricks.

      		 Yes
      Description

      The description of the connection.

      		 No
      Vault The vault where you store your data source values. No
      Workspace URL

      Enter the URL of any Databricks workspace connected to Unity Catalog that you want to integrate.
      To retrieve the URL, log into Databricks and copy the URL. For example: https://123.cloud.databricks.com.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Authentication Type

      Select the type of authentication that you want to apply. You can select any of the following values:

      		 Yes
      Access Token

      The security token that was generated in Databricks for the workspace. The access token must be a personal access token (PAT).
      It is possible to generate a PAT for service principals. For information on the service principal token, go to the Databricks documentation.

      Note Ensure that your Databricks access token has been granted the required permissions in your Databricks environment.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes, if you select Personal Access Token as the authentication type.
      Client ID

      The client ID for OAuth-based authentication in Databricks, or the client ID of the Microsoft Entra ID service principal.

      For information on OAuth-based authentication in Databricks Unity Catalog, go to the Databricks documentation.

      For information on the Microsoft Entra ID service principal, go to Microsoft Entra service principal authentication in the Azure Databricks documentation.

      Note Ensure that your Databricks OAuth client or Microsoft Entra ID service principal has been granted the required permissions in your Databricks environment.

      		 Yes, if you select OAuth or Microsoft Entra ID as the authentication type.
      Client Secret

      The client secret generated for the OAuth-based authentication on Databricks, or the client secret of the Microsoft Entra ID service principal.

      		 Yes, if you selectOAuth or Microsoft Entra ID as the authentication type.
      Tenant ID

      The Directory (tenant) ID for the related application registered in Microsoft Entra ID.

      For information, go to MS Entra service principal authentication in the Azure Databricks documentation.

      		Yes, if you select Microsoft Entra ID as the authentication type.
    4. Click Create.
      The connection is added to the Edge or Collibra Cloud site.

  17. For CollibraData Lineage to connect to and retrieve metadata from dbt Cloud, create a dbt connection.

    Before you begin

    • Create an Edge site on K3S.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select dbt connection.
    4. Enter the connection information.
      FieldDescriptionRequired

      Name

      		The name of the connection.

      Yes

      Description

      		 The description of the connection. This field is also visible when you register content.

      No

      Vault The vault where you store your data source values. No

      Admin URL

      The dbt Cloud Administrative API that Collibra Data Lineage uses to download job artifacts.

      The default value is https://cloud.getdbt.com/api/v2. This default value applies only to the North America multi-tenant region. If your dbt Cloud instance is hosted in a different region or in a single-tenant environment, update this URL to match your deployment. For more information, go to dbt Cloud API v2 and dbt Semantic Layer GraphQL API in the dbt documentation.

      This field is used only if you do not enter a value for the Environment Ids field in the Technical Lineage for dbt Cloud capability.

      If you enter values for both the Admin URL and Environment Ids fields, the Environment Ids field takes precedence.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 

      No

      Metadata URL

      The dbt Cloud Discovery API. The default value is https://metadata.cloud.getdbt.com/graphql.

      For details, go to Query the Discovery API in dbt documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 

      No

      Token Name

      The name of the service token. It can be any unique meaningful name.

      1. Generate a Service token and ensure that you set the Read-Only permissions for CollibraData Lineage to work properly.
      2. Copy the token value when you save the service token.

      For details, go to Generating service account tokens in dbt documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes

      Token Value

      Enter the service token.

      Tip You can select To be encrypted by Edge management server or Encrypted with public key to indicate the encryption method.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

  18. For CollibraData Lineage to connect to and retrieve metadata from Informatica Intelligent Cloud Services, create an Informatica Intelligent Cloud Services (IICS) connection.

    Before you begin

    • To create and use Informatica Intelligent Cloud Services (IICS) connection, use Collibra Platform 2023.03 or later.
    • Create an Edge site on K3S.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Informatica Intelligent Cloud Services (IICS) connection.
    4. Enter the connection information.
      FieldDescriptionRequired

      Name

      		The name of the connection.

      Yes

      Description

      		 The description of the connection. This field is also visible when you register content.

      No

      Vault The vault where you store your data source values. No

      IICS URL

      The URL of the Informatica Intelligent Cloud Services environment sign-in page. For example: https://dm-us.informaticaintelligentcloud.com.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes

      Username

      The username that you use to sign in to Informatica Intelligent Cloud Services.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes

      Password

      The password that you use to sign in to Informatica Intelligent Cloud Services.

      Tip You can select To be encrypted by Edge management server or Encrypted with public key to indicate the encryption method.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

  19. To retrieve data from Tableau, you have to connect to Tableau via the Edge or Collibra Cloud site.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have a resource role with the Configure external system resource permission, for example, Owner.
    • If you connect to Tableau Online, you have a Tableau user with at least Viewer rights.
    • If you connect to Tableau Server, you have a Tableau user with access to at least one site.
    • You have the necessary Tableau permissions.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Tableau connection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the Tableau connection field, when adding the Technical Lineage for Tableau capability to the Edge or Collibra Cloud site.

      		 Yes
      DescriptionThe description of the connection. No
      Vault The vault where you store your data source values. No
      URL

      The URL of your Tableau server.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Authentication Type

      The authentication type for your connection to the Tableau server.

      		 Yes
      Username/Token Name
      • If you selected authentication type username, enter the username of the Tableau user.
      • If you selected authentication type token, enter the personal access token name of the Tableau user.
      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Password/Token Secret
      • If you selected authentication type username, enter the password of the Tableau user.
      • If you selected authentication type token, enter the personal access token secret of the Tableau user.
      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Custom Certificate

      Important This field will soon be deprecated. For guidance on uploading a custom certificate to an Edge site, refer to the "Optionally, use a custom certificate to allow Edge to connect to your data source" content in the "Before you begin" section of this topic.

      Optional field for uploading a custom server certificate, to connect to Tableau. Self-signed certificates are also supported.

      Click Upload and search for your custom server certificate.

      If you specify a certificate during Edge installation, that certificate is used and the certificate you specify here is ignored.

      You don't need to add the certificate to the Java Truststore. Edge stores the certificate as it does any other input parameter, and automatically uses it when connecting to Tableau.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 No
    5. Click Create.

    What's next

    Add the Technical Lineage for Tableau capability to the Edge or Collibra Cloud site.

  20. To retrieve data from Looker, you have to connect to Looker via the Edge site.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have a resource role with the Configure external system resource permission, for example, Owner.
    • You have either a Looker user with the Admin role, or a Looker user with a custom role that has the permissions mentioned in Set up Looker.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Looker connection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the Looker connection field, when adding the Technical Lineage for Looker capability to the Edge site.

      		 Yes
      DescriptionThe description of the connection. No
      Vault The vault where you store your data source values. No
      Looker URL

      The URL to your Looker API.

      Tip There are two ways to find the Looker API URL:
      • In the API Host URL field in the Looker Admin menu. If this field is empty, you can use the default Looker API URL which you can find in the interactive API documentation.
      • In the interactive API documentation URL. It is the part of the URL before /api-docs/.

      Note Looker 3.1 APIs are deprecated; however, the API3 credentials for authorization and access control remain valid.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Client ID

      The username you use to access the Looker API.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Client secret key

      The secret key you use to access the Looker API.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

    What's next

    Add the Technical Lineage for Looker capability to the Edge site.

  21. To retrieve data from SSRS-PBRS, you have to connect to SSRS-PBRS via the Edge site.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have a resource role with the Configure external system resource permission, for example, Owner.

    • You need the following roles, with user access to the server from which you want to ingest:

      • A system-level role that is at least a System user role.
      • An item-level role that is at least a Content Manager role.

      We recommend that you use SQL Server 2019 Reporting Services or newer. We can't guarantee that older versions will work.

    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Microsoft SSRS/PBRS connection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the Microsoft SSRS/PBRS connection field, when adding the Technical Lineage for SSRS-PBRS capability to the Edge site.

      		 Yes
      DescriptionThe description of the connection. No
      Vault The vault where you store your data source values. No
      Microsoft SSRS-PBRS URL

      The URL to the server's web portal. By default, the URL is http://<computer-name>/reports. For example, "http://1.23.45.678/PowerBIReports".

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Username

      The username you use to sign in to the web portal.

      Tip If you use NTLM authentication, your username also contains the NTLM domain name. For example MyDomain\\username.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Password

      The password you use to sign in to the web portal.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

    What's next

    Add the Technical Lineage for SSRS-PBRS capability to the Edge site.

  22. To retrieve data from Power BI, you have to connect to Power BI via the Edge or Collibra Cloud site.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have a resource role with the Configure external system resource permission, for example, Owner.
    • You have the necessary Power BI permissions.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Power BI connection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the Power BI connection field, when adding the Technical Lineage for Power BI capability to the Edge site.

      		 Yes
      DescriptionThe description of the connection. No
      Vault The vault where you store your data source values. No
      Tenant Domain

      The domain associated with the Microsoft Azure tenant. This domain is either a default domain or a custom domain. For example, collibrapowerbi.onmicrosoft.com.

      Note Usually, you can find a list of Power BI tenant or server domains in your Azure Active Directory or in the top right menu.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Authentication type

      The authentication type for your connection to Power BI. Enter one of the following:

      • Service Principal
      • Resource Owner Password Credentials
      		 Yes
      Application ID

      The unique ID of the Microsoft Azure Application (client) ID.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Username

      The email address of your Azure Active Directory user.

      This field only applies if you entered Resource Owner Password Credentials in the Authentication type field.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 No
      Password/Secret key

      Your password (if you entered Resource Owner Password Credentials in the Authentication type field) or your secret key (if you entered Service Principal in the Authentication type field), for your Azure Active Directory user.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Custom certificate

      Important This field will soon be deprecated. For guidance on uploading a custom certificate to an Edge site, refer to the "Optionally, use a custom certificate to allow Edge to connect to your data source" content in the "Before you begin" section of this topic.

      Optional field for uploading a custom server certificate, to connect to Power BI. Self-signed certificates are also supported.

      Click Upload and search for your custom server certificate.

      If you specify a certificate during Edge installation, that certificate is used and the certificate you specify here is ignored.

      You don't need to add the certificate to the Java Truststore. Edge stores the certificate as it does any other input parameter, and automatically uses it when connecting to Power BI.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 No
    5. Click Create.

  23. To retrieve data from Qlik, you have to connect to Qlik via the Edge or Collibra Cloud site.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Qlikconnection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the Qlik connection field, when adding the Technical Lineage for Qlik capability to the Edge site.

      		 Yes
      DescriptionThe description of the connection. No
      Qlik URLThe base URL of your Qlik instance, for example https://customer.eu.qlikcloud.com. Yes
      Client ID

      Your client ID for OAuth authentication.

      This field only applies if you entered Resource Owner Password Credentials in the Authentication type field.

      		 Yes
      Client secret key

      Your client secret key for OAuth authentication.

      		 Yes
    5. Click Create.

  24. Harvested metadata is sent to the Collibra Data Lineage service instances, for processing. You can connect to the Collibra Data Lineage service instances via OAuth authentication. To do so, you have to create a Technical Lineage Admin connection.

    Important OAuth authentication is not yet available for Collibra Platform for Government customers.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select the Technical Lineage Admin connection.
    4. Enter the connection information.
      FieldDescriptionRequired
      Name

      A name for the Edge connection.

      		 Yes
      Description

      A description of the connection.

      		 No

      Authentication Type

      The authentication method you use to connect to Collibra Data Lineage:

      • Basic Authentication
        If you choose this method, ignore the rest of the fields.
      • OAuth
        If you choose this method, you must use the following fields to provide a client ID and client secret. This authentication method is recommended for enhanced security.
        Important OAuth authentication is not yet available for Collibra Platform for Government customers.
      		 Yes

      Client ID

      Your client ID for OAuth authentication.

      1. In Collibra Settings, click OAuth Applications.
      2. Click Register New Application.
        The Register New Application dialog box appears.
      3. Enter the following information:
        1. For the Application Type, select Platform.
        2. Provide a name for the application.
        3. In the Integration Type drop-down list, select Technical Lineage.
      4. Click Register.
      5. Copy and safely store the Client ID and Client Secret.

        Important This is the only time you are able to see the client secret.

      For complete information, go to OAuth Applications.

      Yes

      Client Secret

      Your client secret for OAuth authentication.

      1. In Collibra Settings, click OAuth Applications.
      2. Click Register New Application.
        The Register New Application dialog box appears.
      3. Enter the following information:
        1. For the Application Type, select Platform.
        2. Provide a name for the application.
        3. In the Integration Type drop-down list, select Technical Lineage.
      4. Click Register.
      5. Copy and safely store the Client ID and Client Secret.

        Important This is the only time you are able to see the client secret.

      For complete information, go to OAuth Applications.

      Yes

      .

    5. Click Create.

  25. To retrieve data from MicroStrategy, you have to connect to MicroStrategy via the Edge site.

    Prerequisites

    • You have a global role with the Product Rights > System administration global permission.
    • You have a global role that has the Manage Edge sites global permission.
    • You have a global role that has the Manage connections and capabilities global permission.
    • You have a resource role with the Configure external system resource permission, for example, Owner.
    • You have the necessary MicroStrategy permissions.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select MicroStrategy connection.
    4. Enter the required information.
      FieldDescriptionRequired
      Name

      The name of the connection. The name can be anything, as long as it is unique.

      Tip The name that you provide here is the name you have to select in the MicroStrategy connection field, when adding the Technical Lineage for MicroStrategy capability to the Edge site.

      		 Yes
      DescriptionThe description of the connection. No
      Vault The vault where you store your data source values. No

      MicroStrategy URL

      The URL of your MicroStrategy Intelligence Server.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Username

      The username that you use to sign in to MicroStrategy.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
      Password

      The password that you use to sign in to MicroStrategy.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		 Yes
    5. Click Create.

    What's next

    Add the Technical Lineage for MicroStrategy capability to the Edge site.

  26. For Collibra Data Lineage to connect to and retrieve metadata from Matillion, create a Matillion connection.

    Before you begin

    • To create and use Matillion connection, use Collibra Platform 2023.03 or later.
    • Create an Edge site on K3S.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    2. In the Connections section, click Create Connection.
      The Create Connection dialog box appears.
    3. Select Matillion connection.
    4. Enter the connection information.
      FieldDescriptionRequired

      Name

      		The name of the connection.

      Yes

      Description

      		 The description of the connection. This field is also visible when you register content.

      No

      Vault The vault where you store your data source values. No

      Matillion URL

      The URL of your Matillion environment. For example, https://<domain name> or https://<IP address>.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 

      Yes

      Authentication Type

      The authentication details for signing in to Matillion. You can select one of the following values:

      Credentials
      Use the username and password authentication type.
      Token
      Use the token-based authentication type.

      Yes

      Username

      The username that you use to sign in to Matillion.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 
      		This field is required if you set the Authentication type field to Credentials.

      Password/Token secret

      The password that you use to sign in to Matillion.

      Tip You can select To be encrypted by Edge management server or Encrypted with public key to indicate the encryption method.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the query value to identify the secret in your vault.
        Example 
        Note The Query must be a string containing the properties required to identify the secret. Each property must be separated by a semicolon (;). For example: Safe=<SafeName>;Folder=<FolderName>;Object=<ObjectName>

        If a property is a folder with sub-folders, use a backslash (\) to define the folder path. For example: Folder=Root\Top Secrets\More Secrets

        For more information about query formats and supported properties, go to the CyberArk Credential Provider documentation.

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Engine Type
        Select one of the following:
        • Key Value
        • Database
        Engine Path
        The engine path to your vault where the value is stored.
        Secret Path
        The secret path to your vault where the value is stored.
        Field
        If your Secret Engine Type is Key Value, enter the name of the field to your vault where the value is stored.
        Role
        If your Secret Engine Type is Database, enter the role specified in the Database engine.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Vault Name
        The name of your Azure Key Vault in your Azure Key Vault service where the value is stored.
        Secret Name
        The name of the secret in your vault where the value is stored.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the required information:
        Secret Name
        The name of the secret in your vault where the value is stored.
        Field
        If the secret stored in your AWS Secrets Manager is a JSON value, for example {"pass1": "my-password", "pass2": "my-password2"}, then you need to specify the Field to point to the exact JSON value that should be used. For example, Secret Name: edge-db-customer; Field: pass.
        Note If the secret stored in your AWS Secrets Manager is a plain string value, for example my-password, then you do not need to specify the Field.
        Example 

      To use your vault, do the following:
      1. In the Value Type field, select Vault Key.
      2. Enter the name of the secret in your vault where the value is stored.
        Example 

      Yes

    5. Click Create.

  27. A Shared Storage connection allows you to grant your capabilities access to files from a shared folder. This connection is especially useful for capabilities with large files, as you don't need to manually upload the files directly to Edge. Instead, you define the path to the files when creating the new connection.

    You can upload files that are up to 1 GB in size. Each Edge site allows a total of 6 GB of uploads. This means, for example, you could upload 6 files that are 1 GB each to the Shared Storage connection on your Edge site.

    Note  If you want to create technical lineage for different projects, create a Shared Storage connection and a Technical Lineage for dbt capability for each project.

    Before you begin

    • You have created an Edge site.
    • You have installed an Edge site.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Prerequisites

    Steps

    1. Run one of the following commands in the Edge CLI tool included with the Edge site installer to create a new shared folder and upload your source files:
      Note If your Edge site is installed on a bundled k3s cluster, you must add sudo to the beginning of any command run in the Edge CLI. For example:
      sudo ./edgecli objects folder-upload -h
      • Upload a shared folder which contains multiple data source files:
        Copy
        ./edgecli objects folder-upload \
            --source <source-string> \
            --target <target-string> \
            --ttlSeconds <time>
        KeyDefinition
        <source-string>The data source file you want to upload for your Shared Storage connection.
        <target-string>The folder in your Edge site where you want to store the shared folder and files.
        Note This folder does not have to already exist in your Edge site. If it does not, a folder with the name entered here will be created in your Edge site, containing the folder and files you have uploaded.
        <time> (optional)The number of seconds that the uploaded files will be available before being evicted, the default is 15552000 seconds (180 days).

        Example

        Let's say we have a shared folder /tmp/folder-a, with the following directory structure:

        • /tmp/folder-a/file-a.txt
        • /tmp/folder-a/sub-folder-1/file-b.txt
        • /tmp/folder-a/sub-folder-1/file-c.txt

        Let's upload everything in /tmp/folder-a to a shared folder named shared-folder-1. To do so, run the following command:

        ./edgecli objects folder-upload \
--source /tmp/folder-a \
--target shared-folder-1 \
--ttlSeconds 3600

        The resulting output is the following:

        Uploaded 3 files:
target=shared-folder-1 key=file-a.txt: size=14 ttlSeconds=3600
target=shared-folder-1 key=sub-folder-1/file-b.txt: size=14 ttlSeconds=3600
target=shared-folder-1 key=sub-folder-1/file-c.txt: size=14 ttlSeconds=3600
Total of 42 bytes uploaded
        Note 
        • If you repeat the command, all existing objects that are present in the shared folder will be overwritten.
        • If you remove some files in the source directory, they will not be removed from the shared folder.
        • If you want file parity with your source folder, use the folder-delete command, and then use the folder-upload command.
        • If you need help with the command parameters, run the help command in the Edge CLI.
          ./edgecli objects folder-upload -h
      • Upload a single shared data source file:
        Copy
        ./edgecli objects file-upload \
            --source <source-string> \
            --target <target-string> \
            --key <key-string> \
            --ttlSeconds <time>
        KeyDefinition
        <source-string>The data source file you want to upload for your Shared Storage connection.
        <target-string>The folder in your Edge site where you want to store the shared file.
        Note This folder does not have to already exist in your Edge site. If it does not, a folder with the name entered here will be created in your Edge site, containing the file you have uploaded.
        <key-string> (optional)The path of a specific file within a folder or nested within multiple folders. For example, you only want to upload the myFile.txt file, which is in the myFolders folder.

        If you do not specify this property, it will default to the file name.

        <time> (optional)The number of seconds that the uploaded file will be available before being evicted, the default is 15552000 seconds (180 days).

        Example

        Let's say we have a shared file called /tmp/data-source-file-to-upload/a.txt \, and we want to upload it to a folder called shared-folder-2 in our Edge site.

        To do so, run the following command:

        ./edgecli objects file-upload 
--source /tmp/data-source-file-to-upload/a.txt \
--target shared-folder-2 \
--key rekeyed-a.txt \
--ttlSeconds 8000

        The resulting output is the following:

        Uploaded 1 file:
target=shared-folder-2 key=rekeyed-a.txt: size=14 ttlSeconds=8000
        Note 
        • If you repeat the command, all existing objects that are present in the shared folder will be overwritten.
        • If you remove the file in the source directory, it will not be removed from the shared folder.
        • If you want file parity with your source folder, use the folder-delete command, and then use the folder-upload command.
        • If you need help with the command parameters, run the help command in the Edge CLI.
          ./edgecli objects file-upload -h
    2. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    3. In the Connections section, click Create Connection.
      The Create Connection dialog box opens.
    4. Select Shared storage connection.
      The Create Connection dialog box for Shared storage connection opens.
    5. Enter the connection information.
      FieldDescriptionRequired
      NameThe name of the connection. Yes
      DescriptionThe description of the connection. No
      Folder

      Enter the name of your folder created in step 1. Subfolders are not allowed, so create a separate folder for each Shared Storage connection.

      Note Edge lists the location of this file to the right of this field.      		 Yes
    6. Click Create.

  28. To create technical lineage for JDBC data sources by using the Shared Storage Connection, you must provide SQL files that include your SQL queries. CollibraData Lineage processes the metadata based on your queries to create the technical lineage.

    Prepare the SQL files and store them in your cloud-based storage system. The files must be in one of the following:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    Steps

    1. Create your SQL files. Ensure that the following requirements are met for the SQL files:

      • The SQL files must be UTF-8 encoded.
      • The SQL files can't have white spaces in their names.
      • For better ingestion, include one SQL statement in one SQL file.
      • SQL files that contain Data Definition Language (DDL) statements must be processed before SQL files that contain Data Manipulation Language (DML) statements.
        For the data sources that are listed in Supported SQL statements , Collibra Data lineage automatically detects DDL statements, regardless of the SQL file names.
        For other JDBC data sources, Collibra Data Lineage processes SQL files in alphanumeric order. To ensure that DDL statements are processed first, name SQL files that contain DDL statements so they are before files that contain DML statements.
      • The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. For more information, go to Add a technical lineage capability to your Edge site and Automatic stitching for technical lineage.
      • For Collibra Data Lineage to correctly highlight the transformation logic in the Source code pane, we strongly recommend that your SQL files have Unix line endings. Non-Unix line endings, for example Carriage Return (CR) and Line Feed (LF) line breaks, do not influence the extracted lineage and can result in incorrect highlighting.

      For more information, go to Supported SQL syntax.

    2. Store the SQL files in the folder that you created when you created the Shared Storage connection.
    3. Store the SQL files in your cloud-based storage system. The files must be in one of the following:
      • An AWS S3 bucket.
      • An Azure Data Lake Storage container.
      • A Google Cloud Storage bucket.

    Example 1 SQL statements do not include schema and database names

    This example shows the SQL files that include the queries on the Persons and JobInformation tables and the JobTitleView view. The SQL statements don't contain the database and schema values, so Collibra Data Lineage uses the values of the Database and Schema fields in the technical lineage for SqlDirectory capability for stitching. The SQL files are named in a way that ensures the DDL statements are processed before the DML statement.

    • The ddl-persons.sql file 
      CREATE TABLE Persons (
    PersonID int,
    LastName varchar(255),
    FirstName varchar(255),
    Address varchar(255),
    City varchar(255)
);
    • The ddl-jobinformation.sql file 
      CREATE TABLE JobInformation (
    PersonID int,
    Department varchar(255),
    Title varchar(255)
);
    • The view-jobtitle.sql file 
      CREATE VIEW JobTitleView AS
SELECT
    Persons.PersonID,
    Persons.FirstName,
    Persons.LastName,
    JobInformation.Title
from
    Persons
    INNER JOIN JobInformation ON Persons.PersonID = JobInformation.PersonId

    Example 2 SQL statements include schema and database names

    This example shows SQL files that include the queries on the Persons and JobInformation tables and the JobTitleView view. The SQL statements contain the database and schema names for each table and view, and Collibra Data Lineage uses them for stitching. The SQL files are named in a way that ensures the DDL statements are processed before the DML statement.

    • The ddl-db1-schemaA-persons.sql file 
      CREATE TABLE DB1.SchemaA.Persons (
    PersonID int,
    LastName varchar(255),
    FirstName varchar(255),
    Address varchar(255),
    City varchar(255)
);
    • The ddl-db2-schemaB-jobinformation.sql file 
      CREATE TABLE DB2.SchemaB.JobInformation (
    PersonID int,
    Department varchar(255),
    Title varchar(255)
);
    • The view-db2-schemaC-jobtitleview.sql file 
      CREATE VIEW DB2.SchemaC.JobTitleView AS
SELECT
    Persons.PersonID,
    Persons.FirstName,
    Persons.LastName,
    JobInformation.Title
from
   DB1.SchemaA.Persons
   INNER JOIN DB2.SchemaB.JobInformation ON Persons.PersonID = JobInformation.PersonId

  29. Prepare the data source files and store them in the folder that you created when you created the Shared Storage connection in the previous step.

    Prepare the data source files and store them in your cloud-based storage system. The files must be in one of the following:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    Prerequisites

    You have IBM InfoSphere Information Server version 11.5 or newer.

    Steps

    1. Export the DataStage project files (DSX) for which you want to create a technical lineage. Exclude executables when you export the files.

      Tip You can either export a DataStage project manually or automatically via command line.
    2. Store the DataStage files in your cloud-based storage system.

    3. Store the DataStage files in your Shared Storage connection folder.

      Note If you use Collibra Platform 2024.05 or newer, ensure that you use the Edge CLI tool for creating the Shared Storage connection and storing files in the Shared Storage connection folder. Go to the following Create a Shared Storage connection step for detailed instructions. For more information, go to Edge Command Line Interface (CLI).

    4. Optionally, if your DataStage project uses environment variables, manually export the environment files (ENV).

    5. Give the environment files the same name as the DataStage project files. For example, if your project file is named datastage-project-1.dmx, name your environment file datastage-project-1.env.

    6. Store the environment files in the same bucket or container in your cloud-based storage system.

      Important  
      • CollibraData Lineage only supports DSX and ENV files.
      • You can have one DSX file per DataStage project.
      • You can have more than one DSX file in the bucket or container in your cloud-based storage system.
      • You can have one or none ENV file per DSX file.
      • The name of the DSX file and the ENV file has to be the same.

    7. Store the environment files in the same Shared Storage connection folder.

      Important  
      • Collibra Data Lineage only supports DSX and ENV files.
      • You can have one DSX file per DataStage project.
      • You can have more than one DSX file in the Shared Storage connection folder.
      • You can have one or none ENV file per DSX file.
      • The name of the DSX file and the ENV file has to be the same.

  30. Prepare the data source files and store them in the folder that you created when you created the Shared Storage connection.

    Prepare the data source files and store them in your cloud-based storage system. The files must be in one of the following:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    Steps

    1. In the dbt Core environment, locate the target/ directory of your dbt project. The target/ directory must contain the manifest JSON file and compiled SQL files. This directory is created by running the dbt run and dbt compile commands.
    2. If the target/ directory does not exist or the directory does not contain the manifest JSON file or compiled SQL files, complete the following steps to create the directory and the required files:
      1. Set the profile of your dbt Core to the environment that you want to retrieve the lineage information from.
      2. Use the dbt compile command to generate SQL files. For details, go to About dbt compile command and Manifest JSON file in dbt documentation.
    3. Store the target/ directory in your Shared Storage connection folder and ensure that you maintain the folder structure. Your Shared Storage connection folder must contain all files and subdirectories, such as target/manifest.json and target/compiled/project-name/models/.
      Note If you use Collibra Platform 2024.05 or newer, ensure that you use the Edge CLI tool for creating the Shared Storage connection and storing files in the Shared Storage connection folder. Go to the following Create a Shared Storage connection step for detailed instructions. For more information, go to Edge Command Line Interface (CLI).
    4. Store the target/ directory in a bucket or directory in your cloud-based storage system and ensure that you maintain the folder structure. The bucket or container must contain all files and subdirectories, such as target/manifest.json and target/compiled/project-name/models/.

  31. Prepare the data source files and store them in the folder that you created when you created the Shared Storage connection in the previous step.

    Prepare the data source files and store them in your cloud-based storage system. The files must be in one of the following:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    Prerequisites

    You have Informatica PowerCenter version 9.6 or newer.

    Steps

    1. Export the Informatica objects or repository for which you want to create a technical lineage to the Shared Storage connection folder. Export the Informatica objects or repository for which you want to create a technical lineage, and then upload them to the bucket or container in your cloud-based storage system.Make sure to export all objects, parameter files, mappings and sessions at the same time.
      1. Open Informatica PowerCenter Repository Manager.
      2. Connect to your Informatica repository.
      3. In the navigation panel, navigate to the workflow that contains the Informatica objects that you want to export.
      4. Right-click on the workflow and click Dependencies.
      5. In the Dependencies dialog box, do the following:
        1. Select Primary/Foreign Key dependencies.
        2. Select Global Shortcut dependencies.
        3. In the Object Types selector, select all object types except User-Defined.
      6. Click OK.
        The Dependencies dialog box closes.
        A dialog box with all Informatica objects appears.
      7. Select all objects.
      8. In the toolbar, click (Export to XML).
      9. Save the resulting XML files in your local folder.
      1. In the Informatica PowerCenter Client or PowerCenter Services bin directories, open pmrep.
      2. Export Informatica PowerCenter repository objects.
        Note Make sure that you export the same Informatica PowerCenter repository objects as during a manual export.
      3. Save the resulting XML files in your local folder.
      Note 
      • If your folder contains previous versions of the parameter files, objects might be duplicated across different file versions. The duplicated objects cause Collibra Data Lineage to ignore some transformations, resulting in missing lineage and error messages. For example, if a parameter file is exported after a column was added to a table, duplicated objects exist if the previous version of the parameter file remains in the folder. To avoid missing lineage, export all objects and parameter files at the same time.
      • All XML and parameter files, for example PAR, TXT or PRM files in this folder and its subfolders are taken into account when you create a technical lineage, but Collibra Data Lineage only shows a technical lineage for workflows that have mappings with sources, transformations and targets. Collibra supports the most common Informatica PowerCenter transformations. For more information, see the Informatica PowerCenter documentation.
      • When you export a workflow, ensure that all dependencies – meaning referenced folders, mappings, shortcuts, and sessions – are included in the same export file. This applies whether you export the XML file manually or by using the command line. Collibra Data Lineage looks for a TASKINSTANCE in the workflows (and in worklets in workflows). The TASKINSTANCE points to the sessions, which are dependent on mappings. If a TASKINSTANCE can’t be found in the workflows or worklets, lineage cannot be extracted.
      • To create a technical lineage, the following tags must be present in your XML file:
        • <REPOSITORY>
        • <FOLDER>
        • <SOURCE> / <TARGET>
        • <SESSION>
        • <MAPPING> (that contains one or more <TRANSFORMATION> tags)
        • <WORKFLOW> (that contains one or more <TASK> tags)
      • If parameters are missing from the parameter files, an UNRESOLVED PARAMETERS analyze error is shown in the analysis results in the Sources tab page. For more information, go to Analyze errors and possible solutions in Technical lineage Sources tab page.

    2. In the Shared Storage connection folder, create a folder named techlin-param and put the parameter files in the techlin-param folder.

      Note If you use Collibra Platform 2024.05 or newer, ensure that you use the Edge CLI tool for creating the Shared Storage connection and storing files in the Shared Storage connection folder. Go to the following Create a Shared Storage connection step for detailed instructions. For more information, go to Edge Command Line Interface (CLI).

    3. In the bucket or container in your cloud-based storage system, create a folder named techlin-param and put the parameter files in that folder.

  32. Prepare the data source files and store them in the folder that you created when you created the Shared Storage connection in the previous step.

    Prepare the data source files and store them in your cloud-based storage system. The files must be in one of the following:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    Prerequisites

    • You have SQL Server Integration Services 2012 or newer with package format version 6 or newer.
    • You have Microsoft Visual Studio version 2012 or newer.

    Steps

    1. Export the SSIS files for which you want to create a technical lineage.

      Tip You can export them directly from the SQL Server Integration Services repository or via Microsoft Visual Studio. For more information, see the SQL Server Integration Services documentation.

    2. Store the SSIS files to the Shared Storage connection folder. Typically, the folder contains the following files:

      • SSIS package files (DTSX), containing the SQL Server Integration Services source code.
      • Connection manager files (CONMGR), containing environment and connection information.
      • Parameter files (PARAMS), if applicable.
      Note 
      • If you use Collibra Platform 2024.05 or newer, ensure that you use the Edge CLI tool for creating the Shared Storage connection and storing files in the Shared Storage connection folder. Go to the following Create a Shared Storage connection step for detailed instructions. For more information, go to Edge Command Line Interface (CLI).
      • All files in this folder and subfolders are taken into account when you create a technical lineage. Technical lineage via Edge automatically detects data sources in the SSIS files.
      • Not all SSIS files are processed and shown in the technical lineage. Technical lineage via Edge retrieves all of the SSIS package files from the server, but only the files that contain lineage information, meaning those that contain a data flow, or Pipeline, are processed.

    3. Upload the SSIS files to the bucket or container in your cloud-based storage system. Typically, the bucket or container should contain the following files:

      • SSIS package files (DTSX), containing the SQL Server Integration Services source code.
      • Connection manager files (CONMGR), containing environment and connection information.
      • Parameter files (PARAMS), if applicable.
      Note 
      • All files in this folder and subfolders are taken into account when you create a technical lineage. Technical lineage via Edge automatically detects data sources in the SSIS files.
      • Not all SSIS files are processed and shown in the technical lineage. Technical lineage via Edge retrieves all of the SSIS package files from the server, but only the files that contain lineage information, meaning those that contain a data flow, or Pipeline, are processed.

  33. A Cloud Storage connection allows you to grant your capabilities access to your files. Your files must be stored in one of the following cloud-based storage systems:

    • An AWS S3 bucket.
    • An Azure Data Lake Storage container.
    • A Google Cloud Storage bucket.

    For guidance on how to create a connection between your cloud-based storage system and Edge or Collibra Cloud site, go to the appropriate topic:

  34. A Shared Storage connection allows you to grant your capabilities access to files from a shared folder. This connection is especially useful for capabilities with large files, as you don't need to manually upload the files directly to Edge. Instead, you define the path to the files when creating the new connection.

    You can upload files that are up to 1 GB in size. Each Edge site allows a total of 6 GB of uploads. This means, for example, you could upload 6 files that are 1 GB each to the Shared Storage connection on your Edge site.

    Note  If you want to create technical lineage for different projects, create a Shared Storage connection and a Technical Lineage for dbt capability for each project.

    Before you begin

    • You have created an Edge site.
    • You have installed an Edge site.
    • You have added a vault to your Edge site.
      Note  Vaults are not supported on Collibra Cloud sites.
    • If your data source connection requires a file from your vault, the file must be encoded into Base64 and stored as a regular secret in your vault.

    Prerequisites

    Steps

    1. Run one of the following commands in the Edge CLI tool included with the Edge site installer to create a new shared folder and upload your source files:
      Note If your Edge site is installed on a bundled k3s cluster, you must add sudo to the beginning of any command run in the Edge CLI. For example:
      sudo ./edgecli objects folder-upload -h
      • Upload a shared folder which contains multiple data source files:
        Copy
        ./edgecli objects folder-upload \
            --source <source-string> \
            --target <target-string> \
            --ttlSeconds <time>
        KeyDefinition
        <source-string>The data source file you want to upload for your Shared Storage connection.
        <target-string>The folder in your Edge site where you want to store the shared folder and files.
        Note This folder does not have to already exist in your Edge site. If it does not, a folder with the name entered here will be created in your Edge site, containing the folder and files you have uploaded.
        <time> (optional)The number of seconds that the uploaded files will be available before being evicted, the default is 15552000 seconds (180 days).

        Example

        Let's say we have a shared folder /tmp/folder-a, with the following directory structure:

        • /tmp/folder-a/file-a.txt
        • /tmp/folder-a/sub-folder-1/file-b.txt
        • /tmp/folder-a/sub-folder-1/file-c.txt

        Let's upload everything in /tmp/folder-a to a shared folder named shared-folder-1. To do so, run the following command:

        ./edgecli objects folder-upload \
--source /tmp/folder-a \
--target shared-folder-1 \
--ttlSeconds 3600

        The resulting output is the following:

        Uploaded 3 files:
target=shared-folder-1 key=file-a.txt: size=14 ttlSeconds=3600
target=shared-folder-1 key=sub-folder-1/file-b.txt: size=14 ttlSeconds=3600
target=shared-folder-1 key=sub-folder-1/file-c.txt: size=14 ttlSeconds=3600
Total of 42 bytes uploaded
        Note 
        • If you repeat the command, all existing objects that are present in the shared folder will be overwritten.
        • If you remove some files in the source directory, they will not be removed from the shared folder.
        • If you want file parity with your source folder, use the folder-delete command, and then use the folder-upload command.
        • If you need help with the command parameters, run the help command in the Edge CLI.
          ./edgecli objects folder-upload -h
      • Upload a single shared data source file:
        Copy
        ./edgecli objects file-upload \
            --source <source-string> \
            --target <target-string> \
            --key <key-string> \
            --ttlSeconds <time>
        KeyDefinition
        <source-string>The data source file you want to upload for your Shared Storage connection.
        <target-string>The folder in your Edge site where you want to store the shared file.
        Note This folder does not have to already exist in your Edge site. If it does not, a folder with the name entered here will be created in your Edge site, containing the file you have uploaded.
        <key-string> (optional)The path of a specific file within a folder or nested within multiple folders. For example, you only want to upload the myFile.txt file, which is in the myFolders folder.

        If you do not specify this property, it will default to the file name.

        <time> (optional)The number of seconds that the uploaded file will be available before being evicted, the default is 15552000 seconds (180 days).

        Example

        Let's say we have a shared file called /tmp/data-source-file-to-upload/a.txt \, and we want to upload it to a folder called shared-folder-2 in our Edge site.

        To do so, run the following command:

        ./edgecli objects file-upload 
--source /tmp/data-source-file-to-upload/a.txt \
--target shared-folder-2 \
--key rekeyed-a.txt \
--ttlSeconds 8000

        The resulting output is the following:

        Uploaded 1 file:
target=shared-folder-2 key=rekeyed-a.txt: size=14 ttlSeconds=8000
        Note 
        • If you repeat the command, all existing objects that are present in the shared folder will be overwritten.
        • If you remove the file in the source directory, it will not be removed from the shared folder.
        • If you want file parity with your source folder, use the folder-delete command, and then use the folder-upload command.
        • If you need help with the command parameters, run the help command in the Edge CLI.
          ./edgecli objects file-upload -h
    2. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the site overview, click the name of a site.
        The site page appears.
    3. In the Connections section, click Create Connection.
      The Create Connection dialog box opens.
    4. Select Shared storage connection.
      The Create Connection dialog box for Shared storage connection opens.
    5. Enter the connection information.
      FieldDescriptionRequired
      NameThe name of the connection. Yes
      DescriptionThe description of the connection. No
      Folder

      Enter the name of your folder created in step 1. Subfolders are not allowed, so create a separate folder for each Shared Storage connection.

      Note Edge lists the location of this file to the right of this field.      		 Yes
    6. Click Create.
  35. Which custom lineage definition option are you using?

    • You have to define the technical lineage in a single JSON file. Use the following format and code examples to create the JSON file, and then store the file in the folder that you created when you created the Shared Storage connection, earlier in this procedure.

      If you opt for the single-file definition option, you use a lineage.json file to define the lineage between two or more data objects, and optionally include transformations details to create the custom technical lineage.

      The following sections in the JSON file define different parts in the resulting Collibra technical lineage graph:

      • tree, which defines the data object hierarchy. The data objects are shown as nodes in the technical lineage graph.
      • lineages, which defines the lineage relation. The lineage relations are shown as edges in the technical lineage graph. The edges represent the data flow from a source to a target.
      • codebase_files, which points to the source code files that include transformation details.

      To create a simple custom technical lineage, you need to include assets and lineages sections in your JSON file. You can add the transformation code in the lineages section.

      To create an advanced custom technical lineage, you need to include assets, lineages and codebase_files sections in your JSON file. You add references to the transformation code in source code files in the codebase_files section.

      Transformation code in both simple and advanced custom technical lineages is shown in the source code pane at the bottom part of the technical lineage graph.

      Requirements and restrictions

      The source code files must be in the same directory as the lineage.json file. Otherwise, an error occurs indicating that the lineage harvester (deprecated) cannot find the source code files.

      Sections

      Sections

      		Description

      version

      The version of the JSON architecture. Specify the value of 1.0, which is the only supported version.

      tree

      This section contains tree definitions of data objects between which lineages can be defined. The data objects are systems, databases, schemas, tables, views, columns, dashboards and reports.

      Each node of a tree contains the name, type and optionally children or leaves properties which form a hierarchy of data objects. You must define a node only once in this section. With the nested tree format, you can reuse the properties of one node for multiple children. For example, you can define a database once and use the children array to define multiple tables in the database.

      Tip Usually, the structure you map is the following: system > database > schema > table > column. The system is optional, unless the useCollibraSystemName property is set to true in your lineage harvester configuration file. Collibra Data Lineage can stitch these data objects to assets in Data Catalog. However, you can also map custom objects, for example dashboards and reports. Custom objects cannot be stitched to assets in Data Catalog.

      Important If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in this section, or else stitching will fail.

      lineages

      This section contains the path from a source to a target and defines the transformation code or transformation references to be processed by the Collibra Data Lineage service.

      Important If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in this section, or else stitching will fail.

      codebase_files

      This optional section defines the reference to source code files. Store the source code files that contain the transformation code in the same directory as the lineage.json file.

      Include this section only when you create an advanced custom technical lineage.

      tree section properties

      Properties

      		Description
      name

      The name of your data object. Specify this property with the system name, database name, schema name, table name, view name or column name.

      The following rules apply when you specify this property:

      • The names are case-sensitive. You cannot, however, have two nodes with the same name, but different case, under the same parent node.
      • The names of children and leaves can be identical if the children and leaves with the same names are in different parent nodes.
      type

      The type of your data object. You can specify one of the following options: system, database, schema, table, view, column, dashboard or report.

      If the useCollibraSystemName property in your lineage harvester (deprecated) configuration file is set to true, the system data object is used to stitch to the System asset in Data Catalog. If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in this section, or else stitching will fail.
      children

      The sub-objects that have a hierarchical relation to the defined data object.

      Each child can contain children properties, except for the penultimate child. The penultimate children property must contain the leaves property. The leaves property cannot contain a children property.

      For example, you can use the children property to define a table and use the leaves properties to define columns that have a relation to the table node.

      Each child and leave have the name and type properties and the optional catalog_fullname, catalog_domain_id, catalog_asset_type_name and catalog_asset_type_uuid properties.

      leaves

      The sub-objects of an object that is defined in a children property, but cannot have sub-objects of their own.

      A technical lineage is defined as relations between leaf nodes of the tree.

      The value of the type property of the leaves property must be column or report. Indirect and table-level technical lineages are not supported. For the workarounds to create a table level or indirect technical lineage, see Programming considerations.

      lineage section properties

      Properties

      		RequiredDescription
      src_path
      		Yes

      The hierarchical path to the source data object. This data object is defined as a leaf in the tree section.

      This property represents where the data comes from for a transformation.

      trg_path
      		Yes

      The hierarchical path to the target data object. This data object is defined as a leaf in the tree section.

      This property represents where the data flows to.

      <data objects>
      		Yes

      An ordered array of data object names. This array is required to define the sub-objects of the src_path and trg_path properties.

      Specify the array with the data object names that start from the top of the tree section and finish at a leaf node.

      This example shows data objects that can be stitched: system > database > schema > table > column.

      This example shows data objects that cannot be stitched: dashboard > report > column.

      If the useCollibraSystemName property in your lineage harvester (deprecated) configuration file is set to true, the system data object is used to stitch to the System asset in Data Catalog. If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in this section, or else stitching will fail.
      mapping

      Yes

      Simple custom technical lineage only

      The mapping name. This property specifies a name for the transformation code.

      source_code

      Yes

      Simple custom technical lineage only

      The transformation code, which determines how the technical lineage is constructed.

      The transformation code can be a descriptive string or a SQL statement that manipulates data.

      mapping_ref

      No

      Advanced custom technical lineage only

      This property contains the name of the mapping reference to the transformation code in source code files. This property also contains the position and length of the transformation code to be highlighted in the technical lineage graph.

      source_code

      No

      Advanced custom technical lineage only

      The name of the source code file that contains the transformation code. The transformation code can be a SQL statement, code that manipulates data or a descriptive string.

      The source code file must be in the same folder as the lineage.json file.

      mapping

      No

      Advanced custom technical lineage only

      The unique descriptor of a part of transformation code in a source code file that is in the same directory as the lineage.json file.

      A source code file can contain different parts of transformation code that represent different data flows. This property indicates the referenced data flow.

      The value of this property is the same as the value of the mapping_refs property in the codebase_files section.

      codebase_pos

      No

      Advanced custom technical lineage only

      The positions indicate a string of the transformation code in a source code file to be highlighted in the bottom part of the Collibra technical lineage graph. The whole lines that include the transformation code are highlighted.

      The string must be a subset of the string of the transformation code that is defined by the pos_start and pos_len properties of the mapping_refs property in the codebase_files section.

      pos_start

      No

      Advanced custom technical lineage only

      The start position of the string of the transformation code to be highlighted. The start position is in characters, not bytes.

      The value must be equal to or greater than the value of the pos_start property of the mapping_refs property in the codebase_files section.

      pos_len

      No

      Advanced custom technical lineage only

      The length of the string of the transformation code to be highlighted. The length is in characters, not bytes.

      Specify a value in the following range:

      • Equal to or greater than 1.
      • Less than or equal to the length of the string that is defined by the pos_len property of the mapping_refs property in the the codebase_files section.

      For example, if you specify "pos_start": 10 and "pos_len": 160 in the codebase_files section, specify a value for this property in the range of 0 - 149.

      codebase_files section properties

      Properties

      		Description
      <source code path>

      The file path to source code files that contain the transformation code. The transformation code can be a SQL statement or code that manipulates data.

      The source code file must be in the same directory as the lineage.json file.

      mapping_refs

      The mapping of the transformation code and the position of the transformation code that is shown in the bottom part of the technical lineage graph.

      This property defines a string of the transformation code in the source code file to be shown in the technical lineage graph. The string must include the string that is defined by the pos_start and pos_len properties of the mapping property in the lineage section.

      <mapping>

      The unique descriptor of a part of transformation code in a source code file that is in the same directory as the lineage.json file.

      A source code file can contain different parts of transformation code that represent different data flows. This property indicates the referenced data flow.

      The value must match the value of the mapping property in the lineage section.

      pos_start

      The start position of the string of the transformation code. The start position is in characters, not bytes.

      Specify a value in the following range:

      • Equal to or greater than 0.
      • Less than or equal to the value of the pos_start property in the mapping property in the lineage section.
      pos_len

      The length of the string of the transformation code. The length is in characters, not bytes.

      Specify a value in the following range:

      • Greater than or equal to 1.
      • Less than or equal to the length of the source code file minus the start position.

      For example, if you specify "pos_start": 10 and the file length is 160 characters, specify a value for this property in the range of 1 - 150.

      Programming considerations

      ConsiderationDetails
      Indirect and table-level lineage

      Currently, there is no native support for indirect and table-level lineages. As a workaround, you can specify "type": "column" and "name": "*" for the leaves property to create a table level or indirect technical lineage. With this specification, the indirect technical lineage is shown as a solid line instead of a dashed line in the Collibra technical lineage graph, and is always shown, regardless of whether or not the Show indirect dependencies option is enable or disabled.

      Many-to-one column lineage

      The fundamental unit of definition within the custom technical lineage JSON file is a single source-to-target mapping. Collibra Data Lineage requires a one-to-one relationship definition between the source asset and the target asset for each entry.

      To represent a many-to-one relationship, such as when multiple source columns contribute to a single target column (for example: Source A, Source B, Source C → Target Column D), you must define individual lineage entries for each source-to-target pair.

      Example

      For some example JSON files, go to Custom technical lineage JSON file examples.

      This section shows some example lineage.json files for simple custom technical lineage and advanced custom technical lineage.

      Each example can be used to generate technical lineage graphs in Collibra to represent the IOT_JSON and IOT_DEVICES_PER_COUNTRY tables with the following columns:

      IOT_JSON

      		IOT_DEVICES_PER_COUNTRY

      CCA3

      COUNTRY

      DEVICE_ID

      NUMBER_DEVICES

      Example JSON file for a simple custom technical lineage

      In the following example, the tree section defines the IOT_JSON and IOT_DEVICES_PER_COUNTRY tables and columns. The tables are in a schema named COLLIBRA. The COLLIBRA schema is in a database named COLLIBRA and a system named Databricks.

      Important If you define the System asset in your lineage.json file, the useCollibraSystemName property in your lineage harvester configuration file (deprecated) must be set to true; otherwise, relations will not be created between the relevant assets in Collibra and stitching will fail.

      To show the transformation code at the bottom of the technical lineage graph, specify the mapping and source_code properties in the lineages section.

      Copy
      { 
        "version": "1.0",
        "tree": [
          { 
              "name": "Databricks", 
                 "type": "system",
              "children": [
                 { 
                 "name": "COLLIBRA", 
                 "type": "database",
                 "children": [
                       { 
                            "name": "COLLIBRA", 
                            "type": "schema",
                            "children": [
                           { 
                               "name": "IOT_JSON", 
                               "type": "table",
                               "leaves": [
                                  { 
                                  "name": "CCA3", 
                                  "type": "column"
                              },
                              { 
                                  "name": "DEVICE_ID", 
                                  "type": "column"
                              }
                           ]
                           },
                           { 
                               "name": "IOT_DEVICES_PER_COUNTRY",
                           "type": "table",
                           "leaves": [
                              { 
                                   "name": "COUNTRY", 
                                   "type": "column"
                              },
                              { 
                                  "name": "NUMBER_DEVICES",  
                                  "type": "column"
                              }
                          ] 
                              }
                      ]
                  }
                    ]
                 }
                 ]
             } 
        ],
        "lineages": [
           {
               "src_path": [
               {
                   "system": "Databricks"
               },
               {
                   "database": "COLLIBRA"
                  },
               {
                   "schema": "COLLIBRA"
               },
               {
                   "table": "IOT_JSON"
               },
               {
                   "column": "CCA3"
               }
            ],
            "trg_path": [
               {
                   "system": "Databricks"
               },
               {
                   "database": "COLLIBRA"
               },
               {
                   "schema": "COLLIBRA"
               },
               {
                   "table": "IOT_DEVICES_PER_COUNTRY"
               },
               {
                   "column": "COUNTRY"
               }
            ],
            "mapping": "dev_no_bat_per_country_view",
            "source_code": "INSERT INTO ... SELECT CCA3 AS COUNTRY...FROM IOT_JSON"
            }
        ]
      }

      Example JSON file for an advanced custom technical lineage

      In the following example, the tree section defines the IOT_JSON and IOT_DEVICES_PER_COUNTRY tables and columns. The tables are in a schema named COLLIBRA. The COLLIBRA schema is in a database named COLLIBRA and a system named Databricks.If you define the System asset in your lineage.json file, the useCollibraSystemName property in your lineage harvester configuration file (deprecated) must be set to true; otherwise, relations will not be created between the relevant assets in Collibra and stitching will fail.

      Copy
      {
        "version": "1.0",
        "tree": [
           { 
               "name": "Databricks", 
            "type": "system",
            "children": [
               { 
                   "name": "COLLIBRA", 
                   "type": "database",
                   "children": [
                         { 
                          "name": "COLLIBRA", 
                          "type": "schema",
                          "children": [
                             {
                             "name": "IOT_JSON",
                             "type": "table",
                             "leaves": [
                                { 
                                    "name": "CCA3", 
                                "type": "column"
                            },
                            { 
                                "name": "DEVICE_ID", 
                                "type": "column"
                            }
                         ] 
                     },
                     { 
                         "name": "IOT_DEVICES_PER_COUNTRY", 
                         "type": "table",
                         "leaves": [
                            { 
                                "name": "COUNTRY",
                                "type": "column"
                            },
                            { 
                                "name": "NUMBER_DEVICES", 
                                "type": "column"
                            }
                            ] 
                               }
                           ]
                        }
                     ] 
                  }
               ] 
            }
        ],
        "lineages": [
           {
               "src_path": [
               {
                      "system": "Databricks"
                  },
               {
                   "database": "COLLIBRA"
               },
               {
                   "schema": "COLLIBRA"
               },
               {
                   "table": "IOT_JSON"
               },
               {
                   "column": "CCA3"
               }
            ],
            "trg_path": [
               {
                   "system": "Databricks"
               },
               {
                   "database": "COLLIBRA"
               },
               {
                   "schema": "COLLIBRA"
               },
               {
                   "table": "IOT_DEVICES_PER_COUNTRY"
               },
               {
                   "column": "COUNTRY"
               }
           ],
           "mapping_ref": 
              {
                  "source_code": "transforms.sql",
                  "mapping": "dev_no_bat_per_country_view",
                  "codebase_pos": [
                     { 
                        "pos_start": 71, "pos_len": 69
                     } 
                     ]
                 } 
            }
        ],
        "codebase_files": 
          {
             "transforms.sql": 
             {
                 "mapping_refs": 
                    {
                        "dev_no_bat_per_country_view": 
                    {
                        "pos_start": 0,
                        "pos_len": 246
                    }
                 }
             }
          }
        }

      Example technical lineage graphs

      Both example lineage.json files generate the following technical lineage graph, which contains 2 nodes and 1 edge.

      Note This example image is from the classic Technical lineage viewer UI. The latest viewer UI displays elements differently.

      The following technical lineage graph is generated by using the example lineage.json file for an advanced custom technical lineage. The bottom part shows the transformation code that generated the data flow.

      In the lineages section, the pos_start property is specified with 71 and the pos_len property is specified with 69. The specifications indicate that the transformation code that starts at position 71 and the following 69 characters are highlighted in blue. Line 2 in the technical lineage graph contains the highlighted transformation code.

      Note This example image is from the classic Technical lineage viewer UI. The latest viewer UI displays elements differently.

    • You can define the custom technical lineage in any number of JSON files. Use the following format and code examples to create the JSON files, and then store the files in the folder that you created when you created the Shared Storage connection, earlier in this procedure.

      If you opt for the batch definition option, you need to create a folder with all of your JSON files and specify the folder in your lineage harvester configuration file. The harvester then accesses the folder, zips the content and ingests it for processing.

      Which files you need in your batch folder

      Let's say that you create a folder and name it custom-lineage. In this folder, you need the following:

      • Exactly one metadata file, to provide the JSON architecture version, the data source type, and asset type UUIDs of the assets you want to include in the technical lineage.
      • Optionally, one or more asset files, to provide a list of data objects you want to include in the technical lineage and define the data object hierarchy to achieve stitching.
      • One or more lineage files, to define the lineage relation between two or more data objects.
      • Optionally, a subfolder of source code files that contain the transformation code.
      Example 
      __CUSTOM-LINEAGE__
    ├── assets-domain1.json
    ├── assets1.json
    ├── lineage.json
    ├── lineage-extra.json
    ├── metadata.json
    └── source_codes
        ├── sc1.sql
        └── sc2.py

      Metadata file

      Your metadata file has to be named metadata.json. Format the file as shown in the following image:

      Example 
      Copy
      {
        "version": 3, 
        "application_name": "databricks",
        "asset_types":{
          "Column":{"uuid": "00000000-0000-0000-0000-000000031008"},
          "Table":{"uuid": "00000000-0000-0000-0000-000000031007"},
          "Database":{"uuid": "00000000-0000-0000-0000-000000031006"},
          "Schema":{"uuid": "00000000-0000-0000-0001-000400000002"}
        }
      }

       

      Tip 
      Copy
      {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties": {
          "version": {
            "anyOf": [
              {
                "type": "integer"
              },
              {
                "type": "string",
                "pattern": "^\\d+$"
              }
            ]
          },
          "application_name": {
            "type": "string"
          },
          "asset_types": {
            "type": "object",
            "additionalProperties": {
              "type": "object",
              "properties": {
                "uuid": {
                  "type": "string"
                }
              }
            }
          }
        },
        "required": [
          "application_name",
          "version"
        ]
      }
      SectionDescription

      version

      The version of the JSON architecture. For batch-file instruction, the value must be 3.

      application_name

      The type of data source for which you are creating a technical lineage.

      This helps us to better understand your needs and make more informed decisions concerning future integrations.

      asset_types

      The asset types and UUIDs of the asset types you want to include in the technical lineage.

      Important If you choose to include asset files in your batch definition, the values (meaning the asset types) that you specify in this property must match the values that you specify in the type properties in your asset files. Likewise, the values that you specify in this property must match the asset types that you mention in your lineage files.

      Assets files

      Optionally, you can include one or more assets files. You use asset files to provide the list of data objects you want to include in the technical lineage and define the data object hierarchy. The props property allows you to specify the full names and domain IDs of the assets.

      Tip 

      Don't use asset files in the following scenarios:

      • Your data source consists of the traditional (System) > Database > Schema > Table > Column asset types and hierarchy. In that case, full names are automatically, correctly constructed.
      • You are working with assets that are not part of that traditional asset hierarchy (in which case, you need to use the props property to achieve stitching) and you define props in one or more lineage files.

      The names of your assets files have to follow the format assets<something-unique>.json.

      Asset files can consist of nodes, parent, and leaf kinds of assets. In the following example code, we used the nodes property to specify the highest levels of the data object hierarchy that we want to view in the technical lineage: Database and Schema. We then used the parent and leaf properties to build out the lower levels of the data object hierarchy: Table and Column, respectively.

      parent assets represent what we traditionally refer to as the table-level lineage. leaf assets represents what we traditionally refer to as the column-level lineage.

      Keep in mind that the property names nodes, parent, and leaf are designed to be non-restrictive, so you can define a hierarchy to reflect the hierarchy of any asset types (similar to the database > schema > table > column hierarchy), including your custom asset types.

      Tip For examples of how to configure the props property, as shown in the following code examples, see Using the props property.

      Tip 
      Copy
      {
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "$defs": {
              "assetData": {
                  "type": "object",
                  "properties": {
                      "name": {
                          "type": "string"
                      },
                      "type": {
                          "type": "string"
                      }
                  },
                  "required": [
                      "name",
                      "type"
                  ]
              },
              "props": {
                  "type": ["object", "null"],
                  "properties": {
                      "fullname": {
                          "type": "string"
                      },
                      "domain_id": {
                          "type": "string"
                      }
                  },
                  "required": [
                      "fullname"
                  ]
              }
          },
          "anyOf": [
              {
                  "type": "object",
                  "properties": {
                      "nodes": {
                          "type": "array",
                          "items": {
                              "$ref": "#/$defs/assetData"
                          }
                      },
                      "props": {
                          "$ref": "#/$defs/props"
                      },
                      "parent": {
                          "$ref": "#/$defs/assetData"
                      },
                      "leaf": {
                          "$ref": "#/$defs/assetData"
                      }
                  },
                  "required": [
                      "nodes",
                      "parent",
                      "leaf"
                  ]
              },
              {
                  "type": "object",
                  "properties": {
                      "nodes": {
                          "type": "array",
                          "items": {
                              "$ref": "#/$defs/assetData"
                          }
                      },
                      "props": {
                          "$ref": "#/$defs/props"
                      },
                      "parent": {
                          "$ref": "#/$defs/assetData"
                      }
                  },
                  "required": [
                      "nodes",
                      "parent"
                  ]
              },
              {
                  "type": "object",
                  "properties": {
                      "nodes": {
                          "type": "array",
                          "items": {
                              "$ref": "#/$defs/assetData"
                          }
                      },
                      "props": {
                          "$ref": "#/$defs/props"
                      }
                  },
                  "required": [
                      "nodes"
                  ]
              }
          ]
      }

      Property

      		Description

      nodes

      A JSON element in which you specify the highest levels of the hierarchy. In the example code, the nodes specify the hierarchy of GCS File System > GCS Bucket.

      Example 
      Copy
      {
          "nodes": [
              {
                  "name": "GCS1",
                  "type": "GCS File System"
              }, 
              {
                  "name": "GCS-B1",
                  "type": "GCS Bucket"
              }
          ],
          "props": {
              "fullname": "<full name of the GCS Bucket asset>",
              "domain_id": "<domain of the GCS Bucket asset>"
          }
      }
      name

      The name of the node data object. The value is case-sensitive.

      The value of the name property is not case-sensitive for Database, Schema, Table and Column assets. For those assets, any capitalization discrepancies are rectified during processing, and the names always appear in uppercase in the technical lineage. However, assets files are not needed or recommended for these asset types.

      type

      The type of data object of the specified node, for example: System, Database, Dashboard, or Report. The value is case-sensitive.

      Important The values (meaning the asset types) that you specify for this property must match the values that you specify in the asset_types property in your metadata file.

      parent

      A lower-level data object in a hierarchy for which the highest levels are specified in the nodes section. The parent property represents what we traditionally refer to as the table-level lineage.

      When specifying parent data objects, you also have to include the nodes information, as shown in the following example code.

      Example 
      Copy
      {
          "nodes": [
              {
                  "name": "GCS1",
                  "type": "GCS File System"
              },
              {
                  "name": "GCS-B1",
                  "type": "GCS Bucket"
              }
          ],
          "parent": {
              "name": "DIR1",
              "type": "Directory"
          },
          "props": {
              "fullname": "<full name of the Directory asset>",
              "domain_id": "<domain of the Directory asset>"
          }
      }

      Important If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in this section, or else stitching will fail.

      Tip Each parent object can contain leaf data objects. For example, you can use the parent property to specify a table, and use the leaf properties to specify the columns in the table.

      name

      The name of the parent data object. The value is case-sensitive.

      The value of the name property is not case-sensitive for Database, Schema, Table and Column assets. For those assets, any capitalization discrepancies are rectified during processing, and the names always appear in uppercase in the technical lineage. However, assets files are not needed or recommended for these asset types.

      type

      The asset type of the parent data object, for example: Table, Directory, Dashboard, or Report. The value is case-sensitive.

      Important The values (meaning the asset types) that you specify for this property must match the values that you specify in the asset_types property in your metadata file.

      leaf

      The lowest level data object in your hierarchy. The leaf property represents what we traditionally refer to as the column-level lineage.

      When specifying leaf data objects, you also have to include the nodes and parent information, as shown in the following example code.

      The names of parents and leaf data objects can be identical if the data objects with the same names are sub-objects of different nodes data objects.

      Example 
      Copy
      {
          "nodes": [
              {
                  "name": "GCS1",
                  "type": "GCS File System"
              }, 
              {
                  "name": "GCS-B1",
                  "type": "GCS Bucket"
              }
          ],
          "parent": {
              "name": "DIR1",
              "type": "Directory"
              },
          "leaf": {
              "name": "data.xls",
              "type": "File"
              },
          "props": {
              "fullname": "<full name of the File asset>",
              "domain_id": "<domain of the File asset>"
          }
      }
      name

      The name of the leaf data object. The value is case-sensitive.

      The value of the name property is not case-sensitive for Database, Schema, Table and Column assets. For those assets, any capitalization discrepancies are rectified during processing, and the names always appear in uppercase in the technical lineage. However, assets files are not needed or recommended for these asset types.

      type

      The asset type of the leaf data object, for example: Column, Dashboard, or Report. The value is case-sensitive.

      Important The values (meaning the asset types) that you specify for this property must match the values that you specify in the asset_types property in your metadata file.

      props

      Use this property to manually stitch data objects to assets in Data Catalog by specifying their full name and domain ID.

      Important considerations for stitching
      • Use props for assets that do not follow the traditional System > Database > Schema > Table > Column hierarchy, such as GCS, S3, or local files. Traditional hierarchies are stitched automatically and do not require props.
      • All asset types and UUIDs must be specified in your metadata file.

      • If the Collibra system name setting is set to True when enabling technical lineage via Edge, the system data object is used to stitch to the System asset in Data Catalog. If False, do not specify the system object in the props section, or stitching will fail.

      • The props property applies only to the last asset in the chain; that is the leaf asset, if present, or the parent asset if no leaf exists. Stitching is not recursive. Stitching a child asset (for example, a column) does not automatically stitch its parent (for example, a table or file). To view lineage at both levels, provide a separate entry for each asset.

      File processing order and duplicate definitions

      Define props in only one location to avoid unexpected stitching results. This is important because:

      • Asset and lineage files are processed alphabetically, with all asset files processed before lineage files.
      • If the same asset is defined multiple times with the props property, the first definition encountered in the processing order is used.
      • If an asset is defined in both an asset file and a lineage file, the values in the asset file take precedence, because asset files are processed first.

      For examples of how to configure the props property and how to use it for a custom hierarchy, see Using the props property.

      fullname

      The full name of the asset in Collibra. The value is case-sensitive.

      domain_id

      The reference ID of the domain in which the asset exists in Collibra.

      Using the props property

      The following examples offer some guidance as to when to use the props property and how to configure it.

      First off, let's examine why you don't need to use the props property for the traditional Database > Schema > Table > Column hierarchy. Let's say you have an assets file in which you define a leaf kind of asset:

      Copy
      {
        "nodes": [{
          "name": "Snowflake",
          "type": "System"
        }, {
          "name": "DB1",
          "type": "Database"
        }, {
          "name": "PUBLIC",
          "type": "Schema"
        }],
        "parent": {
          "name": "T1",
          "type": "Table"
        },
        "leaf": {
          "name": "COL1",
          "type": "Column"
        }
      }

      In this case, the full name of the leaf asset (in this case a Column asset) is automatically and correctly constructed as: "snowflake>DB1>PUBLIC>T1>COL1". In fact, for the traditional database type of hierarchy, you don't even need to use asset files, much less the props property.

      However, for the following custom hierarchy, you can use the props property to specify the correct full name of the leaf asset, in this case a File asset.

      Copy
      {
      "nodes": [{
        "name": "gcs",
        "type": "GCS File System"
      }, {
        "name": "bucket1",
        "type": "GCS Bucket"
      }, {
        "name": "/",
        "type": "Directory"
      }],
      "parent": {
        "name": "examples",
        "type": "Directory"
      },
      "leaf": {
        "name": "data.xls",
        "type": "File"
      },
      "props": {
        "fullname": "gcs > bucket1/examples/data.xls",
        "domain_id": "<domain in which the file asset resides>"
      }

      If you don't provide the full name of the leaf asset, it will be constructed using the default traditional formatting (system) > database > schema > table > column. The result would be the full name: "gcs > bucket1 > / > examples > data.xls". However, this is not the correct construction for File assets. The full name provided in the example above ensures the correct construction, so that stitching is achieved.

      • A props property in a nodes section describes the last asset in the array. In this example, you are specifying the full name and domain ID of the Schema asset "SCH1", not the Database asset "DB1".
      • Copy
        {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "props": {
                "fullname": "<full name of the Schema asset>",
                "domain_id": "<domain of the Schema asset>"
                }
        }
      • If you also want to specify the full name and domain ID of the Database asset, you need two entries, as follows:
        Copy
        {
            "nodes": [{"name":"DB1", "type": "Database"}],
            "props": {
                "fullname": "<full name of the Database asset>",
                "domain_id": "<domain of the Database asset>"
                    }
        }

      • To specify the full name and domain ID of a parent asset, use the props property as follows:
        Copy
        {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB1", "type": "Table"},
            "props": {
                "fullname": "<full name of the Table asset>",
                "domain_id": "<domain of the Table asset>"
                }
        }
      • To specify the full name and domain ID of a leaf asset, use the props property as follows:
        Copy
        {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB1", "type": "Table"},
            "leaf": {"name": "COL1", "type": "Column"},
            "props": {
                "fullname": "<full name of the Column asset>",
                "domain_id": "<domain of the Column asset>"
                }
        }
      • To stitch multiple levels of a hierarchy, use the props property on the final asset in the full name. Stitching a leaf asset, for example, a column, does not automatically stitch its parent, for example, a table or file. To ensure lineage is visible and navigable at both levels in the Data Catalog, define each level explicitly.
        Copy
        {
            "comment": "Entry 1: Stitches the File (Parent) so lineage appears on the File asset page.", 
            "nodes": [ { "name": "gcs", "type": "GCS File System" }, { "name": "bucket1", "type": "GCS Bucket" } ], 
            "parent": { "name": "examples", "type": "Directory" }, 
            "leaf": { "name": "data.xls", "type": "File" }, 
            "props": {
                "fullname": "gcs > bucket1/examples/data.xls",  
                "domain_id": "00000000-0000-0000-0000-000000000001"
              } 
        },
        { 
            "comment": "Entry 2: Stitches the Column (Leaf) for column-level lineage.", 
            "nodes": [ { "name": "gcs", "type": "GCS File System" }, { "name": "bucket1", "type": "GCS Bucket" }, { "name": "examples", "type": "Directory" } ],
            "parent": { "name": "data.xls", "type": "File" }, 
            "leaf": { "name": "Column1", "type": "Column" }, 
            "props": { 
                 "fullname": "gcs > bucket1/examples/data.xls > Column1", 
                 "domain_id": "00000000-0000-0000-0000-000000000001" } 
        }

      There are two ways to find this information.

      Via API

      You can use the Find assets API, as documented in the Collibra Developer Portal. In the body of the response, look for the following details in the results array:

      • name: This is the full name of the asset.
      • domain > id: This is the unique reference ID of the domain in which the asset is located.
      Via Data Catalog
      • To find the full name, open the relevant asset page, and then click Actions > Edit. The Edit dialog box appears. The full name is shown in the Name field.
      • To find the domain ID, open the relevant domain in Collibra. In the following example URL, the reference ID is in bold:
        https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001.

      Lineage files

      You can have one or more lineage files in the folder. The names of your lineage files have to follow the format lineage<something-unique>.json.

      You use the lineage file to define the lineage relation between two or more data objects. The lineage relations are shown as edges in the technical lineage graph. The edges represent the data flow from a source to a target.

      This section contains the path from a source to a target and defines the transformation code or transformation references to be processed by the Collibra Data Lineage service.

      Note  If the useCollibraSystemName property in your lineage harvester (deprecated) configuration file is set to true, the system data object is used to stitch to the System asset in Data Catalog. If the useCollibraSystemName property is set to false in your lineage harvester (deprecated) configuration file, do not specify the system data object in these files, or else stitching will fail.
      Tip 
      Copy
      {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "$defs": {
          "assetData": {
            "type": "object",
            "properties": {
              "name": {
                "type": "string"
              },
              "type": {
                "type": "string"
              }
            },
            "additionalProperties": false,
            "required": [
              "name",
              "type"
            ]
          },
          "props": {
            "type": [
              "object",
              "null"
            ],
            "properties": {
              "fullname": {
                "type": "string"
              },
              "domain_id": {
                "type": "string",
                "format": "uuid"
              }
            },
            "additionalProperties": false,
            "required": [
              "fullname"
            ]
          }
        },
        "type": "object",
        "properties": {
          "src": {
            "anyOf": [
              {
                "type": "object",
                "properties": {
                  "nodes": {
                    "type": "array",
                    "items": {
                      "$ref": "#/$defs/assetData"
                    }
                  },
                  "parent": {
                    "$ref": "#/$defs/assetData"
                  },
                  "leaf": {
                    "$ref": "#/$defs/assetData"
                  },
                  "props": {
                    "$ref": "#/$defs/props"
                  }
                },
                "additionalProperties": false,
                "required": [
                  "nodes",
                  "parent",
                  "leaf"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "nodes": {
                    "type": "array",
                    "items": {
                      "$ref": "#/$defs/assetData"
                    }
                  },
                  "parent": {
                    "$ref": "#/$defs/assetData"
                  },
                  "props": {
                    "$ref": "#/$defs/props"
                  }
                },
                "additionalProperties": false,
                "required": [
                  "nodes",
                  "parent"
                ]
              }
            ]
          },
          "trg": {
            "anyOf": [
              {
                "type": "object",
                "properties": {
                  "nodes": {
                    "type": "array",
                    "items": {
                      "$ref": "#/$defs/assetData"
                    }
                  },
                  "parent": {
                    "$ref": "#/$defs/assetData"
                  },
                  "leaf": {
                    "$ref": "#/$defs/assetData"
                  },
                  "props": {
                    "$ref": "#/$defs/props"
                  }
                },
                "additionalProperties": false,
                "required": [
                  "nodes",
                  "parent",
                  "leaf"
                ]
              },
              {
                "type": "object",
                "properties": {
                  "nodes": {
                    "type": "array",
                    "items": {
                      "$ref": "#/$defs/assetData"
                    }
                  },
                  "parent": {
                    "$ref": "#/$defs/assetData"
                  },
                  "props": {
                    "$ref": "#/$defs/props"
                  }
                },
                "additionalProperties": false,
                "required": [
                  "nodes",
                  "parent"
                ]
              }
            ]
          },
          "source_code": {
            "type": "object",
            "properties": {
              "path": {
                "type": "string"
              },
              "highlights": {
                "type": [
                  "array",
                  "null"
                ],
                "items": {
                  "type": "object",
                  "properties": {
                    "start": {
                      "type": "integer"
                    },
                    "len": {
                      "type": "integer"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "len",
                    "start"
                  ]
                }
              },
              "transformation_display_name": {
                "anyOf": [
                  {
                    "type": "integer"
                  },
                  {
                    "type": "string"
                  }
                ]
              }
            },
            "additionalProperties": false,
            "required": [
              "path"
            ]
          }
        },
        "additionalProperties": false,
        "required": [
          "src",
          "trg"
        ]
      }
      Example 
      Copy
      [
        {
          "src": {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB1", "type": "Table"},
            "leaf": {"name": "COL1", "type": "Column"},
            "props": {
            "fullname": "<full name of the leaf asset>",
            "domain_id": "<domain of the leaf asset>"
            },
          },
          "trg": {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB2", "type": "Table"},
            "props": {
            "fullname": "<full name of the parent asset>",
            "domain_id": "<domain of the parent asset>"
          },
          "source_code" : {
            "path": "<folder name>/sc1.sql", 
            "highlights": [{"start": 71, "len": 69 }, ...],
            "transformation_display_name": "middle bubble"
          }
        }
       }
      ]

      Properties

      		Description
      src

      The hierarchical path to the source data object. This property represents where the data comes from for a transformation.

      Important The source of a lineage can only be a parent or a leaf.

      Example 
      Copy
      {
          "src": {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB1", "type": "Table"},
            "leaf": {"name": "COL1", "type": "Column"}
      }
      trg

      The hierarchical path to the target data object. This property represents where the data flows to.

      Important The target can be a parent or a leaf; however, if the source is a parent, the target must be a parent.

      Tip If the target asset is a parent asset and the source asset is a leaf asset, we refer to the lineage as "indirect lineage". If the target asset is a parent asset and the source asset is a parent asset, we refer to the lineage as "table-level lineage".

      Example 
      Copy
      {
          "trg": {
            "nodes": [{"name":"DB1", "type": "Database"}, {"name": "SCH1", "type": "Schema"}],
            "parent": {"name": "TB2", "type": "Table"}
      }
      props

      An optional property that allows you to specify the full name and domain of an asset, for the purpose of stitching.

      This property is not required for Database, Schema, Table and Column asset types.

      File processing order and duplicate definitions

      Define props in only one location to avoid unexpected stitching results. This is important because:

      • Asset and lineage files are processed alphabetically, with all asset files processed before lineage files.
      • If the same asset is defined multiple times with the props property, the first definition encountered in the processing order is used.
      • If an asset is defined in both an asset file and a lineage file, the values in the asset file take precedence, because asset files are processed first.

      source_code

      The transformation code that determines how the technical lineage is constructed. This can be a descriptive string or a SQL statement that manipulates data.

      This section is optional.

      path
       

      The path and name of the source code file that contains the transformation code. The path relative to the source_codes folder, which is in the same folder as the lineage JSON files.

      highlights

      This optional property identifies a string of transformation code in a source code file to be highlighted in the source code pane at the bottom part of the technical lineage graph. The entire lines that include the transformation code are highlighted.

      The string must be a subset of the string of transformation code that is defined by the start and len properties.

      start

      The start position of the string of the transformation code to be highlighted. The start position is in characters, not bytes.

      len

      The length of the string of the transformation code to be highlighted. The length is in characters, not bytes.

      transformation_display_name

      The name of the transformation when looking at the transformations view in the technical lineage viewer.

      Source codes subfolder and files

      You can provide a subfolder of source code files that define the transformation details. The source code folder and your JSON files must be in the CUSTOM_LINEAGE folder, along with the JSON files. If it's not, an error occurs indicating that the lineage harvester cannot find the source code files.

      The source code paths are relative to the CUSTOM_LINEAGE folder.

      Example 
      • source_codes/sc1.sql
      • source_codes/another-subfolder/sc2.sql
      Important Paths must not contain occurrences of ./. The following will fail:
      • source_codes/./sc1.sql

      What happens if you choose not to provide source code files

      If you are using the lineage harvester and there are no source code files to analyze, the batch stats are empty, as shown below. The lineage relations are still created, but because batch stats are directly linked to the source codes, if source code files are not provided, this is expected. 

      Batch stats:
	Parsing errors: 0
	Analysis errors: 0
	Done: 1

      The Done: 1 result is a dummy entry, so that the source appears in the Sources tab page.

      Example JSON files

      For some example JSON files, go to Custom technical lineage JSON file examples.

      In this section, we provide three detailed examples of how to configure your metadata, assets and lineage JSON files for the batch definition option.

      Example 1 is the least complex. It shows a configuration for working with data sources that conform to the traditional (System) > Database > Schema > Table > Column hierarchy. Examples 2 and 3 are more complex examples.

      Helpful considerations to keep in mind

      • Don't use asset files in the following scenarios:

        • Your data source consists of the traditional (System) > Database > Schema > Table > Column asset types and hierarchy. In that case, full names are automatically, correctly constructed.
        • You are working with assets that are not part of that traditional asset hierarchy (in which case, you need to use the props property to achieve stitching) and you define props in one or more lineage files.
      • Don't use the props property for the traditional (System) > Database > Schema > Table > Column hierarchy.

        First off, let's examine why you don't need to use the props property for the traditional database > schema > table > column hierarchy. Let's say you have an assets file in which you define a leaf kind of asset:

        Copy
        {
          "nodes": [{
            "name": "Snowflake",
            "type": "System"
          }, {
            "name": "DB1",
            "type": "Database"
          }, {
            "name": "PUBLIC",
            "type": "Schema"
          }],
          "parent": {
            "name": "T1",
            "type": "Table"
          },
          "leaf": {
            "name": "COL1",
            "type": "Column"
          }
        }

        In this case, the full name of the leaf asset (in this case a Column asset) is automatically and correctly constructed as: "snowflake>DB1>PUBLIC>T1>COL1".

        However, for the following custom hierarchy, you can use the props property to specify the correct full name of the leaf asset, in this case a File asset.

        Copy
        {
        "nodes": [{
          "name": "gcs",
          "type": "GCS File System"
        }, {
          "name": "bucket1",
          "type": "GCS Bucket"
        }, {
          "name": "/",
          "type": "Directory"
        }],
        "parent": {
          "name": "examples",
          "type": "Directory"
        },
        "leaf": {
          "name": "data.xls",
          "type": "File"
        },
        "props": {
          "fullname": "gcs > bucket1/examples/data.xls",
          "domain_id": "<domain in which the file asset resides>"
        }

        If you don't provide the full name of the leaf asset, it will be constructed using the default traditional formatting (system) > database > schema > table > column. The result would be the full name: "gcs > bucket1 > / > examples > data.xls". However, this is not the correct construction for File assets. The full name provided in the example above ensures the correct construction, so that stitching is achieved.

      Examples

      Scenario

      In this example, we are working with a single data source with the traditional (System) > Database > Schema > Table > Column asset types and hierarchy. Therefore, we don't need to include an asset file or use the props property.

      Which files to include

      We are including the following files:

      __CUSTOM-LINEAGE__
    ├── lineage.json
    ├── metadata.json
    └── source_codes
        ├── source_code_view_1.txt
        └── source_code_view_2.txt
      • A metadata JSON file.
        Tip You always need exactly one metadata JSON file.
      • One lineage JSON file, to define the lineage relationships and to show:
        • Column-level lineage.
        • Column-level lineage from the same source (src), which is achieved by adding another entry with the same source.
        • Indirect lineage, in other words the target (trg) is a parent asset.
        • Table-level lineage, in other words both the source and the target are a parent asset.
      • Two source code files in a source_code directory.
        Tip Source code files are always optional.
      The metadata JSON file
      Copy
      {
        "version": 3,
        "application_name": "custom lineage batch example 1",
        "asset_types": {
          "System": {
            "uuid": "00000000-0000-0000-0000-000000031302"
          },
          "Column": {
            "uuid": "00000000-0000-0000-0000-000000031008"
          },
          "Table": {
            "uuid": "00000000-0000-0000-0000-000000031007"
          },
          "Database": {
            "uuid": "00000000-0000-0000-0000-000000031006"
          },
          "Schema": {
            "uuid": "00000000-0000-0000-0001-000400000002"
          }
        }
      }
      The lineage JSON file
      Copy
      [{
          "src": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "T1",
              "type": "Table"
            },
            "leaf": {
              "name": "col1",
              "type": "Column"
            }
          },
          "trg": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "VIEW_1",
              "type": "Table"
            },
            "leaf": {
              "name": "col1",
              "type": "Column"
            }
          }
        }, {
          "src": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "T1",
              "type": "Table"
            },
            "leaf": {
              "name": "col1",
              "type": "Column"
            }
          },
          "trg": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "VIEW_1",
              "type": "Table"
            },
            "leaf": {
              "name": "col2",
              "type": "Column"
            }
          },
          "source_code": {
            "path": "source_codes/source_code_view_1.txt",
            "highlights": [{
              "start": 0,
              "len": 43
            }],
            "transformation_display_name": "view_1 creation"
          }
        }, {
          "src": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "T1",
              "type": "Table"
            },
            "leaf": {
              "name": "col2",
              "type": "Column"
            }
          },
          "trg": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "VIEW_1",
              "type": "Table"
            }
          }
        },
        {
          "src": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "T2",
              "type": "Table"
            }
          },
          "trg": {
            "nodes": [{
              "name": "snowflake",
              "type": "System"
            }, {
              "name": "DB1",
              "type": "Database"
            }, {
              "name": "PUBLIC",
              "type": "Schema"
            }],
            "parent": {
              "name": "VIEW_2",
              "type": "Table"
            }
          },
          "source_code": {
            "path": "source_codes/source_code_view_2.txt",
            "highlights": [{
              "start": 0,
              "len": 39
            }],
            "transformation_display_name": "view_2 creation"
          }
        }
      ]
      Scenario

      In this example, we are working with two data sources:

      • A source system with the following asset types and hierarchy: GCS File System > GCS Bucket > Directory > File.
      • A target system with the traditional (System) > Database > Schema > Table > Column asset types and hierarchy.

      To achieve stitching, we need to use the props property to define the correct full-name construction for the File asset in the source system. We can define props in a lineage file. By doing so, we don't need to include any asset files.

      Which files to include

      We are including the following files:

      __CUSTOM-LINEAGE__
    ├── lineage.json
    ├── metadata.json
      • A metadata JSON file.
        Tip You always need exactly one metadata JSON file.
      • One lineage JSON file, with the props property, to define the correct full-name construction for the File asset in the source system and achieve stitching.
        Tip By defining the props in the lineage file, we don't need to include an assets file. If, for example, we had several lineage files, it would be easier to include an assets file and define the props there once, rather than define the props in each lineage file.

      We are not including source code files. They are always optional.

      The metadata JSON file
      Copy
      {
        "version": 3,
        "application_name": "custom lineage batch example 2",
        "asset_types": {
          "System": {
            "uuid": "00000000-0000-0000-0000-000000031302"
          },
          "Column": {
            "uuid": "00000000-0000-0000-0000-000000031008"
          },
          "Table": {
            "uuid": "00000000-0000-0000-0000-000000031007"
          },
          "Database": {
            "uuid": "00000000-0000-0000-0000-000000031006"
          },
          "Schema": {
            "uuid": "00000000-0000-0000-0001-000400000002"
          },
          "File": {
            "uuid": "00000000-0000-0000-0000-000000031304"
          },
          "Directory": {
            "uuid": "00000000-0000-0000-0000-000000031303"
          },
          "GCS Bucket": {
            "uuid": "00000000-0000-0000-0001-002700000002"
          },
          "GCS File System": {
            "uuid": "00000000-0000-0000-0001-002700000001"
          }
        }
      }
      The lineage JSON file
      Copy
      [{
        "src": {
          "nodes": [{
            "name": "gcs",
            "type": "GCS File System"
          }, {
            "name": "catingestiontest",
            "type": "GCS Bucket"
          }, {
            "name": "/",
            "type": "Directory"
          }, {
            "name": "ingestion-test",
            "type": "Directory"
          }],
          "parent": {
            "name": "ingestion copy",
            "type": "Directory"
          },
          "leaf": {
            "name": "mytest.csv",
            "type": "File"
          },
          "props": {
            "fullname": "f13bf705-13a4-44c9-843e-f341feccfb6e > catingestiontest/ingestion-test/ingestion copy/mytest.csv/1611609340099809",
            "domain_id": "fea1b0b0-705f-4e0d-b5eb-1f21132cc718"
          }
        },
        "trg": {
          "nodes": [{
            "name": "snowflake",
            "type": "System"
          }, {
            "name": "DB1",
            "type": "Database"
          }, {
            "name": "PUBLIC",
            "type": "Schema"
          }],
          "parent": {
            "name": "T1",
            "type": "Table"
          },
          "leaf": {
            "name": "col1",
            "type": "Column"
          }
        }
      }]
      Scenario

      In this example, we are working with the same two data sources as in Example 2:

      • A source system with the following asset types and hierarchy: GCS File System > GCS Bucket > Directory > File.
      • A target system with the traditional (System) > Database > Schema > Table > Column asset types and hierarchy.

      To achieve stitching, we need to use the props property to define the correct full-name construction for the File asset in the source system. However, this time we have multiple lineage files. Instead of defining props in each lineage file, let's include an asset file. That way we only have to define props there once.

      Which files to include

      We are including the following files:

      __CUSTOM-LINEAGE__
    ├── assets.json
    ├── lineage-1.json
    ├── lineage-2.json
    ├── metadata.json
      • A metadata JSON file.
        Tip You always need exactly one metadata JSON file.
      • An assets JSON file, in which provide the list of data objects we want to include in the technical lineage, and define props .
      • Two lineage JSON files, to define the lineage relationships.

      We are not including source code files. They are always optional.

      Metadata JSON file
      Copy
      {
        "version": 3,
        "application_name": "custom lineage batch example 2",
        "asset_types": {
          "System": {
            "uuid": "00000000-0000-0000-0000-000000031302"
          },
          "Column": {
            "uuid": "00000000-0000-0000-0000-000000031008"
          },
          "Table": {
            "uuid": "00000000-0000-0000-0000-000000031007"
          },
          "Database": {
            "uuid": "00000000-0000-0000-0000-000000031006"
          },
          "Schema": {
            "uuid": "00000000-0000-0000-0001-000400000002"
          },
          "File": {
            "uuid": "00000000-0000-0000-0000-000000031304"
          },
          "Directory": {
            "uuid": "00000000-0000-0000-0000-000000031303"
          },
          "GCS Bucket": {
            "uuid": "00000000-0000-0000-0001-002700000002"
          },
          "GCS File System": {
            "uuid": "00000000-0000-0000-0001-002700000001"
          }
        }
      }
      Assets JSON file
      Copy
      [{
        "nodes": [{
          "name": "gcs",
          "type": "GCS File System"
        }, {
          "name": "catingestiontest",
          "type": "GCS Bucket"
        }, {
          "name": "/",
          "type": "Directory"
        }, {
          "name": "ingestion-test",
          "type": "Directory"
        }],
        "parent": {
          "name": "ingestion copy",
          "type": "Directory"
        },
        "leaf": {
          "name": "mytest.csv",
          "type": "File"
        },
        "props": {
          "fullname": "f13bf705-13a4-44c9-843e-f341feccfb6e > catingestiontest/ingestion-test/ingestion copy/mytest.csv/1611609340099809",
          "domain_id": "fea1b0b0-705f-4e0d-b5eb-1f21132cc718"
        }
      }, {
        "nodes": [{
          "name": "snowflake",
          "type": "System"
        }, {
          "name": "DB1",
          "type": "Database"
        }, {
          "name": "PUBLIC",
          "type": "Schema"
        }],
        "parent": {
          "name": "T1",
          "type": "Table"
        },
        "leaf": {
          "name": "col3",
          "type": "Column"
        }
      }]
      Lineage JSON file (1 of 2)
      Copy
      [{
        "src": {
          "nodes": [{
            "name": "gcs",
            "type": "GCS File System"
          }, {
            "name": "catingestiontest",
            "type": "GCS Bucket"
          }, {
            "name": "/",
            "type": "Directory"
          }, {
            "name": "ingestion-test",
            "type": "Directory"
          }],
          "parent": {
            "name": "ingestion copy",
            "type": "Directory"
          },
          "leaf": {
            "name": "mytest.csv",
            "type": "File"
          }
        },
        "trg": {
          "nodes": [{
            "name": "snowflake",
            "type": "System"
          }, {
            "name": "DB1",
            "type": "Database"
          }, {
            "name": "PUBLIC",
            "type": "Schema"
          }],
          "parent": {
            "name": "T1",
            "type": "Table"
          },
          "leaf": {
            "name": "col1",
            "type": "Column"
          }
        }
      }]
      Lineage JSON file (2 of 2)
      Copy
      [{
        "src": {
          "nodes": [{
            "name": "gcs",
            "type": "GCS File System"
          }, {
            "name": "catingestiontest",
            "type": "GCS Bucket"
          }, {
            "name": "/",
            "type": "Directory"
          }, {
            "name": "ingestion-test",
            "type": "Directory"
          }],
          "parent": {
            "name": "ingestion copy",
            "type": "Directory"
          },
          "leaf": {
            "name": "mytest.csv",
            "type": "File"
          }
        },
        "trg": {
          "nodes": [{
            "name": "snowflake",
            "type": "System"
          }, {
            "name": "DB1",
            "type": "Database"
          }, {
            "name": "PUBLIC",
            "type": "Schema"
          }],
          "parent": {
            "name": "T1",
            "type": "Table"
          },
          "leaf": {
            "name": "col2",
            "type": "Column"
          }
        }
      }]

  36. Complete the following steps to add the technical lineage capability to your Edge or Collibra Cloud site. If you want to create technical lineage for different projects, create a Shared Storage connection and a Technical Lineage for dbt capability for each project.

    Tip If your data source allows for system mapping, database mapping, schema mapping, or filtering, you can enter those configurations, in JSON format, in the Source Configuration field in the capability template. If you previously used the CLI lineage harvester and a <source ID> configuration file for those configurations, you can copy and paste the JSON code from your <source ID> file into the Source Configuration field.

    Prerequisites

    Steps

    1. Open a site.
      1. On the main toolbar, click Products icon → Cogwheel icon Settings.
        The Settings page opens.
      2. In the tab pane, click Edge.
        The Sites tab opens and shows a table with an overview of your sites.
      3. In the table, click the name of the site whose status is Healthy.
        The site page opens.
    2. In the Capabilities section, click Add capability.
      The Add capability page appears.
    3. Select the relevant capability template.
      Tip 
      • If you're using a JDBC connection, select Technical Lineage for Amazon RedshiftTechnical Lineage for AzureTechnical Lineage for Db2Technical Lineage for BigQueryTechnical Lineage for GreenplumTechnical Lineage for HiveTechnical Lineage for MySQLTechnical Lineage for NetezzaTechnical Lineage for OracleTechnical Lineage for PostgreSQLTechnical Lineage for SAP HANATechnical Lineage for SnowflakeTechnical Lineage for Spark SQLTechnical Lineage for SQL ServerTechnical Lineage for SybaseTechnical Lineage for Teradata.
      • If you're using a Shared Storage connection to access files in your Edge site, select Technical Lineage for SqlDirectory.
      • If you're using a Cloud Storage connection to access files in an AWS S3 bucket, a Google Cloud Storage bucket, or Azure Data Lake Storage container, select Technical Lineage for SqlDirectory (Cloud).
    4. Select the relevant capability template.
      Tip 
      • If you're using a JDBC connection, for a single database (dedicated SQL pool), select Technical Lineage for Azure.
      • If you're using a JDBC connection, for multiple databases (serverless SQL pool), select Technical Lineage for Azure (multi-DB).
      • If you're using a shared storage connection, select Technical Lineage for SqlDirectory.
    5. Select the relevant capability template.
      Tip 
      • If you're using a Shared Storage connection to access files in your Edge site, select : Technical Lineage for DataStage: Technical Lineage for Informatica PowerCenter: Technical Lineage for SQL Server Integration Services (SSIS): Technical Lineage for dbt: Technical Lineage for Custom Technical Lineage: Technical Lineage for Airflow - OpenLineage: Technical Lineage for AWS Glue - OpenLineage: Technical Lineage for OpenLineage: SQL Server Integration Services (SSIS)
      • If you're using a Cloud Storage connection to access files in an AWS S3 bucket, a Google Cloud Storage bucket, or a Azure Data Lake Storage container, select : Technical Lineage for DataStage (Cloud): Technical Lineage for Informatica PowerCenter (Cloud): Technical Lineage for SQL Server Integration Services (SSIS) (Cloud): Technical Lineage for dbt (Cloud): Technical Lineage for Custom Technical Lineage (Cloud): Technical Lineage for Airflow - OpenLineage (Cloud): Technical Lineage for AWS Glue - OpenLineage (Cloud): Technical Lineage for OpenLineage (Cloud): SQL Server Integration Services (SSIS) (Cloud)
    6. Select the relevant capability template : Technical Lineage for ADF: Technical Lineage for Databricks Unity Catalog: Technical Lineage for Informatica Intelligent Cloud Services (IICS): Technical Lineage for Matillion: Technical Lineage for dbt: Technical Lineage for dbt Cloud: Google Dataplex: Technical Lineage for Looker: Technical Lineage for MicroStrategy: Technical Lineage for PowerBI: Technical Lineage for Qlik (Preview): Technical Lineage for SSRS/PBRS: Technical Lineage for Tableau.
    7. Enter the required information.
      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Authentication Type

      The authentication details for signing in to Azure Data Factory. You can select one of the following values:

      Service Principal
      When you select this authentication type, ensure that you entered the application secret for the Service Principal in the Service Principal Secret field when you created the Azure connection.
      Resource Owner Password Credentials
      When you select this authentication type, ensure that you specify the username field in this capability and also entered the password in the Service Principal Secret field when you created the Azure connection.

      Yes

      ADF Connection

      The Azure connection that you created.

      Important If you used the TechLin Admin Connection field to specify an Edge connection, do not use this field to specify another Edge connection.

      Yes

      Username

      The email address of your Azure Active Directory user.

      This field applies only when you selected Resource Owner Password Credentials for the field.

      No

      Resource Group Name

      The name of the resource group that the data factory belongs to.

      Yes

      Subscription ID

      The subscription ID of the resource group.

      Yes

      Factories

      The Azure Data Factory factories that Collibra Data Lineage collects and processes. Specify this property with an array of Azure Data Factory factory names. This property is optional.

      The following rules apply when you specify this property:

      • Enter the factory names in square brackets ([ ]), enclose each factory name in double quotes (" "), and separate them by a comma, for example, ["MyFirstFactory", "MySecondFactory"].
      • The factory name is not case-sensitive. For example, the MyFactory and myfactory factories are considered the same by Azure Data Factory and Collibra Data Lineage.
      • If you do not specify any factory name, Collibra Data Lineage collects and processes all factories that have datasets and piplelines in them.

      No

      Pipeline Runs Days To Look Back

      The number of days of pipeline run metadata that Collibra Data Lineage retrieves and processes. For example, if the value is 30, Collibra Data Lineage analyzes pipeline runs from the past 30 days.

      You can specify an integer up to 365. The default value is 0, which means no pipeline run metadata is queried or used. To generate lineage from manual runs of parameterized pipelines, which is useful if you set parameters during manual runs, set this property to a value greater than 0.

      No

      Source Configuration

      The source configuration for database mapping, system mapping, schema mapping, and filtering. Specify the following properties in JSON format and enter the content in this field.

      If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <sourceId>.conf file in this field.

      Property

      		Description

      Mandatory?

      found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name> | found_dbname=<datafactory_name>_<linkedservice_name>;found_hostname=*

      The information of the supported data sources in Azure Data Factory to be collected by Collibra Data Lineage. You can specify any of the following values for the found_dbname property:

      • A database name. And then you can specify the following properties:
        • found_hostname=<server name>, where <server name> is the name of the server that the database is running on.
        • found_schema=<schema name>, where <schema name> is the name of the schema. This property is optional.
      • The combination of <datafactory_name>_<linkedservice_name>, where <datafactory_name> is a data factory name and <linkedservice_name> is a linked service name. If you use this combination, specify * for the found_hostname property.

      You can use wildcards to capture multiple connection string combinations:

      PatternDescription
      *Matches everything.
      ?Matches any single character.
      [seq]Matches any character in "seq".
      [!seq]Matches any character not in "seq".

      Yes

      dbname

      The name of the database asset in Data Catalog. Specify this property with the database name that you created when you registered the data source.

      No

      schema

      The name of the schema asset in Data Catalog. Specify this property with the schema name that you created when you registered the data source.

      If the Collibra Data Lineage fails to find the schema that you specify, it uses the default schema.

      No

      dialect

      If you specify a database name for the found_dbname property, select one of the following dialects. If you specify a linked service name for the found_dbname property, ignore this property.

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for a IBM Db2 data source.
      • generic, for any data source.
      • greenplum, for a Greenplum data source.
      • hana, for a SAP HANA data source.
      • hive, for a Hive data source.
      • impala, for a Impala data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for an PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      • vertica, for a Vertica data source.

      No

      collibraSystemName

      The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      If you don't specify a value for this property, DEFAULT is shown in the technical lineage.

      How to configure this property if you have two databases with the same name

      If you have two databases named Customers, and you created a System asset for each of these databases in Data Catalog, Customers-Europe and Customers-USA, you can configure this property as follows.

      Copy
      "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
          "dbname": "Customers",
          "schema": "mssql-schema-name",
          "dialect": "mssql",
          "collibraSystemName": "Customers-Europe"
      },
      "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
          "dbname": "Customers",
          "schema": "oracle-schema-name",
          "dialect": "oracle",
          "collibraSystemName": "Customers-USA"
      },

      No

      Example 
      Copy
      {
           "found_dbname=databasename1;found_hostname=server-name.onmicrosoft.com;found_schema=schema1": {
               "dbname": "mssql-database-name",
               "schema": "mssql-schema-name",
               "dialect": "mssql",
               "collibraSystemName": "mssql-system-name"
           },
           "found_dbname=datafactory_linkedservice;found_hostname=*": {
               "dbname": "linkedservice-dbname",
               "schema": "linkedservice-schema",
               "collibraSystemName": "linkedservice-system-name"
           }
      }

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      GCP Connection

      The GCP connection that you created.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Save Input Metadata

      Select the checkbox if you want to save the input metadata extracted from the data source in ZIP files. The files can be useful for troubleshooting. Select this option only on request of Collibra Support. If this option is selected, you can download the files from the Synchronization Result dialog box once the synchronization activity is completed.

      No

      Logging configuration, Memory (MiB), and JVM arguments

      These fields contain configuration options that can help when investigating issues with the capability.

      Important Only complete these fields on request of or together with Collibra Support.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Synonyms

      This query retrieves the alternative names for the database objects.

      Views

      This query retrieves the view definitions.

      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Synonyms

      This query retrieves the alternative names for the database objects.

      Views

      This query retrieves the view definitions.

      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Synonyms

      This query retrieves the alternative names for the database objects.

      Views

      This query retrieves the view definitions.

      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      In this capability, you use the Database name section to specify multiple databases.

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Synonyms

      This query retrieves the alternative names for the database objects.

      Views

      This query retrieves the view definitions.

      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Databricks Connection

      The Databricks connection that you created.

      Yes

      Compute Resource HTTP Path

      The HTTP path of the compute resource in Databricks Unity Catalog that Collibra Data Lineage collects and processes to create technical lineage.

      You can find the HTTP path in the connection details of your cluster as shown in the following graphic. For details, go to Get connection details for a cluster in Databricks documentation.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Time Frame

      Specify the duration for data collection. You can enter any of the following values:

      • A number of days.
        The default value is 365, which means that Collibra Data Lineage collects data of the past 365 days.
        If a negative number or 0 is entered, the default time frame of the past 365 day is used.
      • A date range:
        • YYYY-MM-DD YYYY-MM-DD. Collibra Data Lineage collects data from the specified start date to the specified end date.
        • YYYY-MM-DD now. Collibra Data Lineage collects data from the specified start date to the current date.
        • now YYYY-MM-DD. Collibra Data Lineage collects data the current date to the specified end date.

        The start date must be earlier than the end date and at least one day apart.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Save Input Metadata

      Select the checkbox if you want to save the input metadata extracted from the data source in ZIP files. The files can be useful for troubleshooting. Select this option only on request of Collibra Support. If this option is selected, you can download the files from the Synchronization Result dialog box once the synchronization activity is completed.

      No

      Ingest lineage from external tables

      Select this option to ingest lineage from external delta tables. Selecting this option can cause longer synchronization times.

      Clear the checkbox to exclude lineage from external delta tables.

      No

      Also ingest lineage from table_lineage

      Select this option to create both table-level and column-level lineage. In addition to the lineage from the system.access.column_lineage table, Collibra Data Lineage also ingests lineage that exists only in the system.access.table_lineage table when this option is selected. Selecting this option can cause longer synchronization times.

      To create only column-level lineage, clear the checkbox.

      No

      (Deprecated) Filters
      Note This field is deprecated. Use the Include Filter and Exclude Filter fields on the Synchronization page to specify which lineage events to include or exclude in technical lineage. If you specify this field and also the Include Filter and Exclude Filter fields, the Include Filter and Exclude Filter fields take precedence.

      Use this section to include or exclude databases and schemas to be ingested. Enter the filters in JSON format. If you used filters when you integrated Databricks Unity Catalog, you can enter in this field the content from the Filters and Domain Mapping field in the Databricks Unity Catalog capability. Noted that Collibra Data Lineage ignores the UUIDs that are specified in the content.

      Text in JSON format to include or exclude databases and schemas, and to configure domain mappings.

      • The text must be in JSON format and can contain an include and an exclude block. You can use any JSON validator to verify the format. Collibra is not responsible for the privacy, confidentiality, or protection of the data you submit to such JSON validators, and has no liability for such use.
      • In the include block, you can specify the domain in which specific catalogs or schemas must be ingested. The format is: “Catalog/Database > schema ”: “domain ID”. For example, "HR > address-schema": "30000000-0000-0000-0000-000000000000".
      • In the exclude block, you can specify the catalogs or schemas that you don't want to ingest. For example, "* > test".
      • The exclude block has priority over the include block.
      • If the include block is not present, we ingest all assets into the same domain as the System asset.
      • If there is no explicit domain mapping for a schema, we use the domain specified for the database.
      • You can use the keyword default as a domain ID. In that case, the catalog or schema will be ingested in the same domain as the System asset.
      • A match with a database has priority over a match with a schema.
      • The integration fails before the synchronization starts, if one or more domain IDs specified in the include block don't exist.
      • The integration fails before the synchronization starts if a domain ID is left empty in the include block.
      • You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account.
      Example 
      {
      "include": {
      "HR": "20000000-0000-0000-0000-000000000000",
      "HR > address-schema": "30000000-0000-0000-0000-000000000000",
      "Orders > fk*": "40000000-0000-0000-0000-000000000000",
      "Orders > *": "50000000-0000-0000-0000-000000000000",
      "* > profiling": "60000000-0000-0000-0000-000000000000",
      "sales": "default"
      },
      "exclude": [
      "testDB",
      " * > information_schema"
      ]
      }

      In this example:

      • Assets from the "HR" database will be ingested into the domain with ID "20000000-0000-0000-0000-000000000000". However, all assets from the "HR > address-schema" schema will be ingested into the domain with id "30000000-0000-0000-0000-000000000000".
      • All assets from the "Orders” database with schemas starting with fk (fk*) will be ingested into the domain with ID "40000000-0000-0000-0000-000000000000", and all other assets from the "Orders” database will be ingested into the domain with ID "50000000-0000-0000-0000-000000000000".
      • All assets from the "sales" database will be ingested in the same domain as the System asset.
      • Assets from the "profiling" schema will be ingested into the domain with ID "60000000-0000-0000-0000-000000000000". However, the "profiling" schema in the database "Orders" will be ingested in the domain with ID "50000000-0000-0000-0000-000000000000" because a database match has priority over a schema match.
      • All assets from the "testDB” database will be excluded.
      • All assets from the “information_schema” schema in all databases will be excluded.

      No

      Logging configuration

      Memory (MiB)

      JVM arguments

      These fields are configuration options that can help when investigating issues with the capability.

      Important Use these fields only at the request of Collibra Support.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

      If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <sourceId>.conf file in this field.

      Property

      		Description

      OdbcDataSources

      Open Database Connectivity data sources in IBM InfoSphere DataStage for which you want to create a technical lineage.

      <data-source-name>

      The ODBC data source name that you use in your DataStage projects.

      This section contains the properties to translate the database, schema and dialect.

      dbname
      		The name of your database, to which the ODBC data source connection refers.
      schema

      The name of your schema, to which the ODBC data source connection refers.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      NonOdbcConnectors

      Other data source connectors in IBM InfoSphere DataStage for which you want to create a technical lineage. For example, DB2, Oracle or Netezza.

      Note This section is optional.

      <data-source-connector-ID>

      The data source username and database of the connector that you use in your DataStage projects. This usually looks like for example admin@database-name. The combination of the username and database name should be unique.

      The following section contains the properties to translate the database, schema and dialect.

      dbname
      		The name of your database, to which the data source connection refers.
      schema

      The name of your schema, to which the data source connection refers.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      Jobs

      The jobs that you want technical lineage via Edge to collect and process to create the technical lineage.

      This section is optional. The following rules apply when you specify this section:

      • Specify jobs that are executed so that the technical lineage graph does not include any job parameters with undefined values.
      • Specify only the first and parent jobs in a sequence of executed jobs. Technical lineage via Edge automatically collects all jobs that are called by the parent jobs.

      For details about how CollibraData Lineage parses DataStage jobs and resolves parameters, see Transformation logic and common errors for DataStage.

      JobParameters
      		The runtime parameters that are not in the DSX and ENV files. You can specify multiple job parameters.
      name
      		The name of the job parameter. You can specify any of the following values:
      • A parameter name
      • A user variable
      • A parameter set

      Important Do not enclose the name between "#" characters, for example "name": "#name#"

      value

      The value of the job parameter. You can specify one of the following values, depending on the value of the name property:

      • If a parameter name is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter reference
      • If a user variable is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter set reference
      • If a parameter set is specified for the name property, specify this property with a value file name.

      For details about how the values are resolved, see the Parameter resolution section in Transformation logic and common errors for DataStage.

      perJobParameters

      The parameters of a specific job. For example, you ingest multiple jobs where the parameters have the same name, but different values.

      Note  This value takes precedence over the values specified in the JobParameters property. Otherwise, the original jobParameters field is used as the “default” option.
      jobID
      		The ID of the job.
      name
      		 The name of the job parameter. You can specify any of the following values:
      • A parameter name
      • A user variable
      • A parameter set

      Important Do not enclose the name between "#" characters, for example "name": "#name#"

      value
      The value of the job parameter. You can specify one of the following values, depending on the value of the name property:
      • If a parameter name is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter reference
      • If a user variable is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter set reference
      • If a parameter set is specified for the name property, specify a value file name as the value.

      For details about how the values are resolved, see the Parameter resolution section in Transformation logic and common errors for DataStage.

      Copy
      {
        "OdbcDataSources": {
          "oracle-data-source": {
            "dbname": "my-oracle-database",
            "schema": "my-oracle-schema",
            "dialect": "oracle",
            "collibraSystemName": "my-system"
          },
          "mssql-data-source": {
            "dbname": "my-mssql-database",
            "schema": "my-mssql-schema",
            "dialect": "mssql",
            "collibraSystemName": "my-system"
          }
        },
        "NonOdbcConnectors": {
          
          "admin@database-name": {
            "dbname": "my-netezza-database",
            "schema": "my-netezza-schema",
            "dialect": "netezza",
            "collibraSystemName": "my-system"
          },
          "admin@second-database-name": {
            "dbname": "my-second-netezza-database",
            "schema": "my-second-netezza-schema",
            "dialect": "netezza",
            "collibraSystemName": "my-system"
          }
        },   
        "jobs": [
          "my_job_1",
          "my_job_2"
        ],
        "jobParameters": [
          {
            "name": "parameter_value_1"
            "value": "parameter_value_1"
          },
          {
            "name": "parameter_name_2",
            "value": "parameter_value_2"
          }
        ]
        "perJobParameters": {
            "jobId1": [
                {
                    "name": "parameter_name_1",
                    "value": "parameter_value_3"
                }
             ],
             "jobId2": [
                 {
                     "name": "parameter_name_1",
                     "value": "parameter_value_4"
                 }
             ]
        }
      }

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

      If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <sourceId>.conf file in this field.

      Property

      		Description

      OdbcDataSources

      Open Database Connectivity data sources in IBM InfoSphere DataStage for which you want to create a technical lineage.

      <data-source-name>

      The ODBC data source name that you use in your DataStage projects.

      This section contains the properties to translate the database, schema and dialect.

      dbname
      		The name of your database, to which the ODBC data source connection refers.
      schema

      The name of your schema, to which the ODBC data source connection refers.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      NonOdbcConnectors

      Other data source connectors in IBM InfoSphere DataStage for which you want to create a technical lineage. For example, DB2, Oracle or Netezza.

      Note This section is optional.

      <data-source-connector-ID>

      The data source username and database of the connector that you use in your DataStage projects. This usually looks like for example admin@database-name. The combination of the username and database name should be unique.

      The following section contains the properties to translate the database, schema and dialect.

      dbname
      		The name of your database, to which the data source connection refers.
      schema

      The name of your schema, to which the data source connection refers.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      Jobs

      The jobs that you want technical lineage via Edge to collect and process to create the technical lineage.

      This section is optional. The following rules apply when you specify this section:

      • Specify jobs that are executed so that the technical lineage graph does not include any job parameters with undefined values.
      • Specify only the first and parent jobs in a sequence of executed jobs. Technical lineage via Edge automatically collects all jobs that are called by the parent jobs.

      For details about how CollibraData Lineage parses DataStage jobs and resolves parameters, see Transformation logic and common errors for DataStage.

      JobParameters
      		The runtime parameters that are not in the DSX and ENV files. You can specify multiple job parameters.
      name
      		The name of the job parameter. You can specify any of the following values:
      • A parameter name
      • A user variable
      • A parameter set

      Important Do not enclose the name between "#" characters, for example "name": "#name#"

      value

      The value of the job parameter. You can specify one of the following values, depending on the value of the name property:

      • If a parameter name is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter reference
      • If a user variable is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter set reference
      • If a parameter set is specified for the name property, specify this property with a value file name.

      For details about how the values are resolved, see the Parameter resolution section in Transformation logic and common errors for DataStage.

      perJobParameters

      The parameters of a specific job. For example, you ingest multiple jobs where the parameters have the same name, but different values.

      Note  This value takes precedence over the values specified in the JobParameters property. Otherwise, the original jobParameters field is used as the “default” option.
      jobID
      		The ID of the job.
      name
      		 The name of the job parameter. You can specify any of the following values:
      • A parameter name
      • A user variable
      • A parameter set

      Important Do not enclose the name between "#" characters, for example "name": "#name#"

      value
      The value of the job parameter. You can specify one of the following values, depending on the value of the name property:
      • If a parameter name is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter reference
      • If a user variable is specified for the name property, specify one of the following values:
        • A parameter value
        • A parameter set reference
      • If a parameter set is specified for the name property, specify a value file name as the value.

      For details about how the values are resolved, see the Parameter resolution section in Transformation logic and common errors for DataStage.

      Copy
      {
        "OdbcDataSources": {
          "oracle-data-source": {
            "dbname": "my-oracle-database",
            "schema": "my-oracle-schema",
            "dialect": "oracle",
            "collibraSystemName": "my-system"
          },
          "mssql-data-source": {
            "dbname": "my-mssql-database",
            "schema": "my-mssql-schema",
            "dialect": "mssql",
            "collibraSystemName": "my-system"
          }
        },
        "NonOdbcConnectors": {
          
          "admin@database-name": {
            "dbname": "my-netezza-database",
            "schema": "my-netezza-schema",
            "dialect": "netezza",
            "collibraSystemName": "my-system"
          },
          "admin@second-database-name": {
            "dbname": "my-second-netezza-database",
            "schema": "my-second-netezza-schema",
            "dialect": "netezza",
            "collibraSystemName": "my-system"
          }
        },   
        "jobs": [
          "my_job_1",
          "my_job_2"
        ],
        "jobParameters": [
          {
            "name": "parameter_value_1"
            "value": "parameter_value_1"
          },
          {
            "name": "parameter_name_2",
            "value": "parameter_value_2"
          }
        ]
        "perJobParameters": {
            "jobId1": [
                {
                    "name": "parameter_name_1",
                    "value": "parameter_value_3"
                }
             ],
             "jobId2": [
                 {
                     "name": "parameter_name_1",
                     "value": "parameter_value_4"
                 }
             ]
        }
      }

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Copy
      {
          "collibraSystemNames":{
              "projects":[
              {"collibraSystemName":"SystemName"}
              ]
          },
          "materializedMapping":{
              "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
          }
      }

      Property

      		Description

      Required?

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      No

      projects

      This section contains the Collibra system names.

      No

      collibraSystemName

      The system or server name of the data source. This is also the name of your System asset in Data Catalog:

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      In the following code example, the project is stitched to the systemname1 System asset in Data Catalog. 

      {
   "collibraSystemNames":{
	"projects":[
	    {"collibraSystemName":"systemname1"}
	]
   },
}
      		No
      materializedMapping

      Indicates how materializations in dbt are mapped. If you do not specify this property, CollibraData Lineage maps materializations to tables by default. You can change the mapping of a materialization to view.

      In the following example, the ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES materialization is mapped to a view. 

      	"materializedMapping":{
	    "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
	}
      		No

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Copy
      {
          "collibraSystemNames":{
              "projects":[
              {"collibraSystemName":"SystemName"}
              ]
          },
          "materializedMapping":{
              "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
          }
      }

      Property

      		Description

      Required?

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      No

      projects

      This section contains the Collibra system names.

      No

      collibraSystemName

      The system or server name of the data source. This is also the name of your System asset in Data Catalog:

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      In the following code example, the project is stitched to the systemname1 System asset in Data Catalog. 

      {
   "collibraSystemNames":{
	"projects":[
	    {"collibraSystemName":"systemname1"}
	]
   },
}
      		No
      materializedMapping

      Indicates how materializations in dbt are mapped. If you do not specify this property, CollibraData Lineage maps materializations to tables by default. You can change the mapping of a materialization to view.

      In the following example, the ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES materialization is mapped to a view. 

      	"materializedMapping":{
	    "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
	}
      		No

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      dbt Connection

      The dbt connection that you created.

      Yes

      Environment Ids

      The IDs of the environments that Collibra Data Lineage uses to download job artifacts.

      Enter an array of environment IDs, for example [123456, 987654]. This field is required if you do not enter a value for the Admin URL field in the dbt connection.

      If you enter values for both the Admin URL and Environment Ids fields, the Environment Ids field takes precedence.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be downloaded and enhance the performance of CollibraData Lineage in the following ways:

      • Filter the projects and jobs to be downloaded. Include projects and jobs to be downloaded by specifying the filter property.
      • Specify different Collibra system names for different projects by specifying the collibraSystemNames property .
      • Map a materialization as a view instead of a table by specifying the materializedMapping property.

      Specify the following properties in JSON format and enter the content in this field.

      Copy
      {
          "collibraSystemNames":{
              "projects":[
              {
                  "project_id":654321,
                  "collibraSystemName":"SystemName"
              }
             ]
          },
          "filter":{
              "jobIds":[1234],
              "projectIds":[654321]
          },
          "materializedMapping":{
              "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
          }
      }

      Property

      		Description

      Required?

      collibraSystemNames

      You can use this section to specify the Collibra System Name for each project.

      No

      projects

      This section contains the project names and the Collibra system names.

      No

      project_id

      Your project ID. You can find the project ID in the dbt URL right after projects. For example, if your dbt URL is https://cloud.getdbt.com/develop/54321/projects/12345 , your project_id is 12345.

      No

      collibraSystemName

      The system or server name of the data source. This is also the name of your System asset in Data Catalog:

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      In the following code example, the project with the 12345 project ID is stitched to the systemname1 System asset in Data Catalog. 

      {
   "collibraSystemNames":{
	"projects":[
	    {"project_id":"12345","collibraSystemName":"systemname1"}
	]
   },
}
      		No
      filter

      You can use this section to include projects and jobs to be downloaded. Collibra Data Lineage downloads and processes only the specified jobs and projects.

      In the following code example, the job with the 1234 job ID and the projects with the 98 and 5678 project IDs are downloaded. 

      {
       "filter": {
	   "jobIds": [
	       1234
	   ],
	   "projectIds": [
	       98,
	       5678
	   ]
	}
}

      No

      jobIds

      The job IDs of the jobs that you want to include.

      Specify an integer. Do not specify a string.

      To get your job ID, in your dbt, select Deploy and then Jobs. Select a job and you can find your job ID in the URL. For example, if your URL is cloud.getdbt.com/deploy/65432/projects/23456/jobs/123456, 123456 is your job ID.

      No

      projectIds

      The project IDs of the projects that you want to include.

      Specify an integer. Do not specify a string.

      You can find the project ID in the dbt URL right after projects. For example, if your dbt URL is https://cloud.getdbt.com/develop/54321/projects/12345 , your project_id is 12345.

      		No
      materializedMapping

      Indicates how materializations in dbt are mapped. If you do not specify this property, CollibraData Lineage maps materializations to tables by default. You can change the mapping of a materialization to view.

      In the following example, the ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES materialization is mapped to a view. 

      "materializedMapping":{
    "ELS_MATERIALIZE_MULTIPLE_EXTERNAL_TABLES":"VIEW"
}
      		No
      Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <sourceId>.conf file in this field.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Billing ID

      Important This field is currently optional. In a future version of Collibra it will become mandatory.

      The billing ID is a JDBC connection parameter that is required to execute the SQL statements to harvest the metadata. Enter the project ID of any single project for which you want to harvest metadata.

      Tip You can then use the Project ID field to specify all of the other projects from which you want to harvest metadata.

      No

      Project ID

      Use this field to specify (by project ID) the project or projects from which you want to harvest metadata. Leave this field empty if you want to harvest the metadata from all projects for which the service account has permissions.

      Tip Each field can only contain a single project ID. To list mutiple project IDs, click Add property, and then add the next project ID.

      Yes

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription

      Columns

      This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Columns Tail

      This query retrieves all columns tails.

      Views

      This query retrieves the view definitions.

      Other Queries

      This query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      External Database Name

      The database value to be used as the database name in the full path (system -> database -> schema -> table). Use this field to ensure successful stitching for a database-less data source. You can specify one of the following values:

      • CData, which CDATA drivers returned as a placeholder. Use this value if you did not create a custom database name by using the CustomizedDefaultCatalogName property when you registered your data source.
      • The custom database name that you specified for the CustomizedDefaultCatalogName property when you registered your data source.

      When you registered a HiveQL data source on Edge and did not specify the CustomizedDefaultCatalogName property, the following full path to a column in Data Catalog is created and CData is used as the database name: 

      Hive_123 (system) > CData (database) > Hive_ABC (schema) > Table > Column

      When Collibra Data Lineage collects metadata from the HiveQL data source, the value of the Database Name field is used as the schema name because HiveQL is database-less.

      If you do not enter a value for the External Database Name field, the value of the Collibra System Name field is used as the database name. If you set the Collibra system name setting to true, the value of the Collibra System Name field is also used as the system name. In this case, the full path is shown as follows and does not match the full path in Data Catalog:

      Hive_123 (system) > Hive_123 (database) > Hive_ABC (schema) > Table > Column

      In this case, the stitching is missing because of the mismatch of the full paths. To match the full path and preserve stitching, you must specify CData for the External Database Name field.

      No

      Database Name

      The name of the database or schema (these terms are synonymous for Hive) from which you want to harvest metadata.

      Click + Add Database Name, to add another database name.

      You must use either this field or the Database name JSON field.

      Database name JSON

      This field provides an alternative method for providing multiple database or schema (these terms are synonymous for Hive) names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription

      Columns

      This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Object Names

      		This query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      IICS connection

      The Informatica Intelligent Cloud Services (IICS) connection that you created.

      Note Collibra Platform 2023.03 or newer is required to use the Informatica Intelligent Cloud Services (IICS) connection.

      Yes

      Objects

      The objects that you want to retrieve.

      Each object requires a path and a type as shown in the following example, where,

      path
      The path to the object, which is relative to the Explore directory in IICS, for example, Sales.
      type

      The type of the object, for example, Taskflow.

      IICS scanner's starting point is a Taskflow or Linear Taskflow (Workflow). Therefore the only meaningful types to retrieve are: Taskflow, Workflow, Project and Folder.

      The types are not case sensitive.

      The following example retrieves the Project, Folder, Taskflow, and Workflow objects. 

      [
    {
	"path" : "Sales",
	"type" : "Project"
    }, 
    {
	"path" : "Finance/Task_Flows",
	"type" : "Folder"
    },
    {
	"path" : "Common/Task_Flows/tf_CalendarDimension",
	"type" : "Taskflow"
    },
    {
	"path" : "Common/Linear_Task_Flows/wf_StateProvinceDimension",
	"type" : "Workflow"
    } 
]

      Tip For more information about the objects that you can retrieve and the required information, go to the Informatica documentation.

      Yes

      Parameter Files

      Upload a ZIP file that contains Informatica Intelligent Cloud Services parameter files. You can name the ZIP file as you prefer. Ensure that the ZIP file contains all parameter files that you want Collibra Data Lineage to collect.

      Important The hierarchy of the files in the directory must be an exact match of the hierarchy of the files in your file system.
      1. Create a directory for your parameter files.
        For this example, let's name the directory my-parameter-files.
      2. Copy your parameter files to your parameter files directory.
        Be sure to preserve the full path for each of your parameter files. For example, for parameter file /root/child/child2/paramfile.txt, run the following commands:
        1. cd /full/path/<my-parameter-files>/
        2. mkdir -p root/child/child2/
        3. cp /root/child/child2/paramfile.txt root/child/child2/
      3. Compress the directory into a ZIP file and upload it to the Parameter Files field.

      No

      Source Configuration

      The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?

      collibraSystemNames

      This section contains the system information for Informatica Intelligent Cloud Services.

      connections

      This section contains the system connection information. This is required to reference to the system or server of the connection.

      connectionName

      The name of the connection. The name must match the System asset name in Data Catalog for stitching.

      		Yes
      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      		No

      connectionDefinitions

      This section contains the database, schema and dialect information for each connection in Informatica Intelligent Cloud Services.

      Note You can add connection information for each connection in the connections section.

      connectionName

      The name of the connection. The name must match with the name in a connection name in the connections section.

      This property is required.

      		Yes
      databaseName

      The name of your database. The name must match the Database asset name in Data Catalog for stitching.

      		Yes
      schemaName

      The name of your schema. The name must match the Schema asset name in Data Catalog for stitching.

      		Yes
      dialect

      The dialect of the connection. Specify this property for Collibra Data Lineage to properly extract and parse queries that are related to this connection.

      You can enter one of the following values:

      • bigquery
      • db2
      • hana
      • hive
      • greenplum
      • mssql
      • mysql
      • netezza
      • oracle
      • postgres
      • redshift
      • snowflake
      • spark
      • teradata
      		No
      Copy
      {
          "collibraSystemNames": {
              "connections": [
                  {
                      "connectionName": "DG_con_standby_cmdm_clientors",
                      "collibraSystemName": "PUBLIC"
                  },
                  {
                      "connectionName": "DG_con_dev_dg_dgiauser_su",
                      "collibraSystemName": "PUBLIC"
                  }
              ]
          },
          "connectionDefinitions": [
              {
                  "connectionName": "DG_con_standby_cmdm_clientors",
                  "databaseName": "main",
                  "schemaName": "dbo",
                  "dialect": "oracle"
              },
              {
                  "connectionName": "DG_con_dev_dg_dgiauser_su",
                  "databaseName": "main",
                  "schemaName": "dbo",
                  "dialect": "oracle"
              }
          ]
      }
      Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

      If the connection definitions are provided but certain properties are not specified, an analyze error called CONFIGURATION is displayed in the transformations table on the Sources tab page when the technical lineage is created. The unspecified properties are marked as UNDEFINED in the analyze error. For more information about the analyze errors, go to Analyze errors and possible solutions in Technical lineage Sources tab page.

      Property

      		Description

      connectionDefinitions

      This section contains the connection properties to a source in Informatica PowerCenter.

      <connectionName>

      The type of your source or target data source.

      This section contains the connection properties to a source or target in Informatica PowerCenter.

      Note Define a connection in the connection definitions only once; specifically, define a data source with the <connectionName> property specified only once in the connection definitions. If you define a connection multiple times, unexpected lineage and stitching issues might occur.
      dbname
      		The name of your source or target database.
      schema

      The name of your source or target schema.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      collibraSystemNames

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      databases

      This section contains the database information. This is required to connect directly to the system or server of the database.

      dbname
      		The name of the database. The database name is the same as the name you entered in the <connectionName> section.
      collibraSystemName

      The system or server name of the database.

      connections

      This section contains the connection information. This is required to reference to the system or server of the connection.

      connectionName

      The name of the connection.

      collibraSystemName

      The system or server name of the connection.

      Important If you are using variables in Informatica PowerCenter, add the value of the variable instead of the name in the connection definitions. For example, if the parameter file contains $DBConnection_dwh=DWH_EXPORT then you add the following connection definitions: 
      {
	"DWH_EXPORT":

		{ "dbname": "DWH", "schema": "DBO" }
}
      Copy
      {
          "connectionDefinitions": {
              "oracle_source": {
                  "dbname": "oracle-source-database-name1",
                  "schema": "my Oracle source schema",
                  "dialect": "oracle"
              },
              "oracle_target": {
                  "dbname": "oracle-target-database-name2",
                  "schema": "my other oracle target schema",
                  "dialect": "oracle"
              }
          },
          "collibraSystemNames": {
              "databases": [
                  {
                      "dbname": "oracle-source-database-name1",
                      "collibraSystemName": "oracle-system-name1"
                  },
                  {
                      "dbname": "oracle-target-database-name2",
                      "collibraSystemName": "oracle-system-name2"
                  }
              ],
              "connections": [
                  {
                      "connectionName": "oracle-source",
                      "collibraSystemName": "oracle-system-name1"
                  },
                  {
                      "connectionName": "oracle-target",
                      "collibraSystemName": "oracle-system-name2"
                  }
              ]
          }
      }
      Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

      If the connection definitions are provided but certain properties are not specified, an analyze error called CONFIGURATION is displayed in the transformations table on the Sources tab page when the technical lineage is created. The unspecified properties are marked as UNDEFINED in the analyze error. For more information about the analyze errors, go to Analyze errors and possible solutions in Technical lineage Sources tab page.

      Property

      		Description

      connectionDefinitions

      This section contains the connection properties to a source in Informatica PowerCenter.

      <connectionName>

      The type of your source or target data source.

      This section contains the connection properties to a source or target in Informatica PowerCenter.

      Note Define a connection in the connection definitions only once; specifically, define a data source with the <connectionName> property specified only once in the connection definitions. If you define a connection multiple times, unexpected lineage and stitching issues might occur.
      dbname
      		The name of your source or target database.
      schema

      The name of your source or target schema.

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      collibraSystemNames

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      This property is optional.

      databases

      This section contains the database information. This is required to connect directly to the system or server of the database.

      dbname
      		The name of the database. The database name is the same as the name you entered in the <connectionName> section.
      collibraSystemName

      The system or server name of the database.

      connections

      This section contains the connection information. This is required to reference to the system or server of the connection.

      connectionName

      The name of the connection.

      collibraSystemName

      The system or server name of the connection.

      Important If you are using variables in Informatica PowerCenter, add the value of the variable instead of the name in the connection definitions. For example, if the parameter file contains $DBConnection_dwh=DWH_EXPORT then you add the following connection definitions: 
      {
	"DWH_EXPORT":

		{ "dbname": "DWH", "schema": "DBO" }
}
      Copy
      {
          "connectionDefinitions": {
              "oracle_source": {
                  "dbname": "oracle-source-database-name1",
                  "schema": "my Oracle source schema",
                  "dialect": "oracle"
              },
              "oracle_target": {
                  "dbname": "oracle-target-database-name2",
                  "schema": "my other oracle target schema",
                  "dialect": "oracle"
              }
          },
          "collibraSystemNames": {
              "databases": [
                  {
                      "dbname": "oracle-source-database-name1",
                      "collibraSystemName": "oracle-system-name1"
                  },
                  {
                      "dbname": "oracle-target-database-name2",
                      "collibraSystemName": "oracle-system-name2"
                  }
              ],
              "connections": [
                  {
                      "connectionName": "oracle-source",
                      "collibraSystemName": "oracle-system-name1"
                  },
                  {
                      "connectionName": "oracle-target",
                      "collibraSystemName": "oracle-system-name2"
                  }
              ]
          }
      }
      Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Warning 
      • You can only specify one source ID per Looker instance. Ingesting the same Looker instance under different source IDs will fail.
      • Any single Looker instance can be ingested only once. If you create more than one connection for the same Looker instance, integration will fail. If you want to ingest from multiple unique Looker instances, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

      Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

      We highly recommend that you specify only one source ID per Looker service account.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Looker connection

      The Looker connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to Looker.

      Yes

      Domain ID

      The unique reference ID of the domain in Collibra Platform in which you want to ingest the Looker assets.

      This is the default domain.

      If you want to ingest the contents of specific Looker Folders into specific domains in Collibra, you specify the domain reference IDs in the filters section of your source configuration. See the Source Configuration field below.

      Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      		 Yes
      Paging limit

      Optional property for customizing the Looker API pagination settings. The default value of "50" is sufficient in most cases; however, you can decrease it to help mitigate node limit errors, or increase it to speed up API calls.

      Note The paging limit option is known to cause issues when used with Looker Core instances. If you experience issues, for example a Received RST_STREAM: Protocol error, we recommend disabling pagination by setting the value to "0".

      No

      Concurrency level

      This optional property is intended to help if you are experiencing HTTP 401 Unauthorized errors due to too many concurrent HTTP calls, using the same token. It allows you to specify the internal sizing, meaning the amount of tasks that can be executed at the same time.

      The default value is "15", meaning as many as 15 HTTP requests can take place in parallel. Consider reducing the value if you are experiencing HTTP 401 Unauthorized errors. Setting the value to "1" effectively disables the concurrency level, so that HTTP requests will be run in a synchronous manner, instead of in parallel.

      		 No
      Connection timeout

      This optional property is intended to help avoid timeout errors, when Edge attempts to connect to your Looker instance. The default value is "30", meaning a timeout error is thrown if a connection is not established within 30 seconds.

      If timeout errors persist, try setting the value to "60" or "90".

      		 No

      Source configuration

      This field allows you to provide JSON code, to:

      • Filter on the Looker folders from which you want to ingest metadata.
      • If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in Looker.
        Collibra Data Lineage uses the system names to match the structure of databases in Looker to assets in Data Catalog.

      If you previously integrated Looker via the lineage harvester, you can copy and paste in this field the JSON code from your Looker <source ID> configuration file.

      Property

      		Description

      Mandatory?

      Connections

      This section contains all Looker connections for which you want to create a technical lineage.

      Yes

      <connection name>

      The name of a connection object in Looker.

      Yes

      schema

      The name of the default schema of a supported data source in Looker.

      If the lineage harvester fails to find a specific schema, it uses the default schema.

      No

      dbname
      		The name of the database of a supported data source in Looker.

      No

      collibraSystemName

      The system or server name of a database.

      If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

      How to configure this property if you have two databases with the same name

      Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

      Copy
      "connection-object1": {
          "dialect": "mssql",
          "schema": "mssql-schema-name",
          "dbname": "Customers",
          "collibraSystemName": "Customers-Europe"
      },
      "connection-object2": {
          "dialect": "oracle",
          "schema": "oracle-schema-name",
          "dbname": "Customers",    
          "collibraSystemName": "Customers-USA"
      }

      Yes

      filters

      Optionally, use this section to specify the Looker folders from which you want to ingest metadata.

      Important 
      • If you don't want to filter on Looker Folders, you must completely remove this filters section.
      • You can filter on Looker folders, but not on Looker data sets. That's because Looker data sets are linked directly to the server, instead of a folder, as shown in the Looker metadata overview. Looker data sets are ingested in the default domain, regardless of any filtering.

      Let’s say, for example, you filter on folder B. A Looker Folder asset is created in the specified domain in Collibra, and all of the metadata in folder B is ingested. If folder B has a parent folder A, then a Looker Folder asset is created (in the domain specified for folder B) to preserve the hierarchy, but no metadata from folder A is ingested.

      You can specify more than one Looker folder for ingestion into a single domain in Collibra.

      There are significant benefits to filtering by folder ID. For information, see the filters > folderIds property description.

      You can use wildcards to capture multiple connection string combinations:

      PatternDescription
      *Matches everything.
      ?Matches any single character.
      [seq]Matches any character in "seq".
      [!seq]Matches any character not in "seq".
      		No
      domainId

      The unique resource ID of the domain (or domains), in Collibra, in which you want to ingest data objects from one or more Looker Folders.

      You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.

      		 
      description
      		Any description, as you see fit. 
      folderNames

      The name (or names) of the Looker Folders from which you want to ingest.

      You must specify either a folder name, a folder ID, or both.

      		 
      folderIds

      The ID (or IDs) of the Looker Folder you want to ingest.

      You must specify either a folder ID, a folder name, or both.

      Tip If you filter by folder ID, filtering is carried out via the API, instead of on the Collibra Data Lineage service instances.

      When you filter by folder ID, the lineage harvester accesses only the folders you specify via this property, and sends only that metadata to the Collibra Data Lineage service instance for processing and ingestion in Data Catalog. Conversely, if you filter by folder name (via the folderNames property), metadata from all Looker folders is sent to the Collibra Data Lineage service instance. Only then is filtering applied.
      The advantages are as follows:
      • Faster integration testing, as you can filter on a single folder.
      • Enhanced data security and privacy by not harvesting folders that contain sensitive information.
      • Improved processing times by not including folders dedicated to, for example, development and testing. This is especially beneficial for organizations with a lot of data in Looker.
      		 
      Example 
      .
      Copy
      {
         "Connections":{
            "connection-object1":{
               "schema":"mssql-schema-name",
               "dbname":"mssql-database-name",
               "collibraSystemName":"mssql-system-name"
            },
            "connection-object2":{
               "schema":"oracle-schema-name",
               "dbname":"oracle-database-name",
               "collibraSystemName":"oracle-system-name"
            }
         },
         "filters":[
            {
               "domainId":"605fa8ae-f8c6-4261-938b-8326e2806f3d",
               "description":"Databricks_folder",
               "folderIds":[
                  "abc-123",
                  "def-456"
               ]
            },
            {
               "domainId":"245bc5a4-4c30-44b5-8356-ddbe708b56d6",
               "description":"personal",
               "folderIds":[
                  "hij-789*",
                  "jkl-101112"
               ]
            }
         ]
      }
      		 No
      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      		 No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      		The namespace that is used. The value of this property must match the namespace in the OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      No

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      		The namespace that is used. The value of this property must match the namespace in the OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      No

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for AWS Glue namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      The namespace that is used by AWS Glue. The value of this property must match the namespace in the AWS Glue OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      Yes

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for AWS Glue namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      The namespace that is used by AWS Glue. The value of this property must match the namespace in the AWS Glue OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      Yes

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for Airflow namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      The namespace that is used by Airflow. The value of this property must match the namespace in the Airflow OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      Yes

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The source configuration to reduce the amount of data objects to be processed and enhance the performance of Collibra Data Lineage.

      Specify the following properties in JSON format and enter the content in this field.

      Property

      		DescriptionRequired?
      datasources

      An array of mappings for Airflow namespaces to Collibra system names and databases. This section includes the properties to translate the system name, database, schema, and dialect.

      No

      namespace

      The namespace that is used by Airflow. The value of this property must match the namespace in the Airflow OpenLineage files.

      Yes

      group

      Specifies a group name used to organize files in the namespace.

      Specify this property only when the type property is set to file. Otherwise, the system uses the namespace value as the file group.

      No

      type

      The type of data source that this namespace contains. Specify one of the following values:

      • database.
      • file.

      If you do not specify this property, Collibra Data Lineage derives the value from the JSON schema. If you specify this property, your provided value takes precedence.

      When this property is set to file, you must specify the group property.

      Yes

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      The following rules apply when you specify the system name:

      • If you do not specify this property, it is set to DEFAULT.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      database

      The name of the default database that the namespace connection refers to.

      The following rules apply when you specify the database name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses database names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      schema

      The name of the default schema, to be used with the namespace connection.

      The following rules apply when you specify the schema name:

      • The value of this property is determined in the following order of precedence:
        1. The value you specify.
        2. If not specified, Collibra Data Lineage derives the value from the namespace and name fields based on the OpenLineage naming convention.
        3. If the value cannot be derived, Collibra Data Lineage uses schema names defined in queries or output dataset names.
      • This property is used only when the type property is set to database.
      • This property could assist in stitching to Catalog assets.

      No

      dialect

      When no columnLineage is present, Collibra Data Lineage tries to parse any SQL present. Set the dialect to parse SQL properly.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No 

      {
    "datasources": [
        {
            "namespace": "s3a://my-dev",
            "group": "file_group_my-dev",
            "type": "file"
        },
        {
            "namespace": "s3a://my-dev2",
            "group": "file_group_my-dev2",
            "type": "file"
        },
        {
            "namespace": "snowflake://myorg.snowflake.com",
            "type": "database",
            "collibraSystemName": "snowflake",
            "database": "snowflake_db",
            "schema": "snowflake_db",
            "dialect": "snowflake"
        }
    ]
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      No

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Warning 
      • You can only specify one source ID per SQL Server Reporting Service (SSRS) or Power BI Report Server (PBRS). Ingesting the same SQL Server Reporting Service (SSRS) or Power BI Report Server (PBRS) under different source IDs will fail.
      • Any single SSRS or PBRS can be ingested only once. If you create more than one connection for the same SSRS or PBRS, integration will fail. If you want to ingest from multiple unique SSRS or PBRS, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

      Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

      We highly recommend that you specify only one source ID per SSRS or PBRS service account.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      SSRS/PBRS connection

      The Microsoft SSRS/PBRS connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to SSRS-PBRS.

      Yes

      Domain ID

      The unique reference ID of the domain in Collibra Platform in which you want to ingest the SSRS assets.

      Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      		 Yes
      Folder Filter

      This field allows you to include only specific folders that contain reports or KPIs in the ingestion process.

      Important This field is mandatory. If you want to ingest all folders, enter *.

      You can filter on multiple folders by:

      • Specifying folder names.
      • Specifying the full path to folders.
      • Using a wildcard.
      • Using a combination of these approaches. For example: folder1, /database/folder2, /folder3/*
      ScenarioConfiguration

      Ingest all folders with the name Folder3, anywhere in the folder hierarchy.

      Folder3

      Note Reports in child folders of Folder3 are not included in the ingestion. As such:
      • Reports in /Folder1/Folder2/Folder3 are included in the ingestion.
      • Reports in /Folder3/ChildFolder are not included in the ingestion.

      Ingest Folder1 and Folder2.

      Folder1, Folder2

      Ingest two folders that are both named Folder1.

      In this case, specify the full paths to the folders, for example:

      /Database1/Folder1, /Database2/Database3/Folder1

      Use a wildcard to ingest all child folders of Folder1.

      /Folder1/*

      Note The reports in all child folders of Folder1 are ingested, but the reports in Folder1 itself are not ingested.

      Ingest all reports from Folder1 and all of the reports in the child folders of Folder1.

      /Folder1/*, /Folder1

      Tip For more information about connecting to a SSRS or PBRS folder, see the Microsoft documentation.

      Yes

      Source configuration

      This field allows you to provide <source ID> configuration file JSON code.

      The <source ID> configuration file allows you to:
      • If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in SSRS and PBRS.
      • Provide additional information about databases in SSRS and PBRS, which is necessary if the databases do not contain all information to process the SQL source code correctly.

      If you previously integrated SSRS-PBRS via the lineage harvester, you can copy and paste in this field the JSON code from your SSRS-PBRS <source ID> configuration file.

      Property

      		Description

      Required?

      DataSources

      This section contains all connections for which you want to create a technical lineage.

      The DataSources section refers to shared data sources in SSRS and PBRS. For more information about shared data sources, see the Microsoft documentation.

      Yes

      <data source type>

      The path of a connection object in SSRS and PBRS.

      Yes

      dbname
      		The name of the database of a supported data source in SSRS and PBRS.

      No

      schema

      The name of the default schema of a supported data source in SSRS and PBRS.

      No

      dialect

      The dialect of the supported data source in SSRS and PBRS.

      No

      collibraSystemName

      The system or server name of the database.

      If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

      How to configure this property if you have two databases with the same name

      Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

      Copy
      "Redshift": {
          "dbname": "Customer",
          "schema": "redshift-schema-name",
          "dialect": "redshift",
          "collibraSystemName": "Customers-Europe"
      },
      "Oracle": {
          "dbname": "Customer",
          "schema": "oracle-schema-name",
          "dialect": "oracle",
          "collibraSystemName": "Customers-USA"
      }

      Yes

      CustomDataSources

      You can use custom data processing extensions that are used to support embedded data sources of which the data source definition is specified locally in a report or embedded data set.

      The CustomDataSources section refers to embedded data sources in SSRS and PBRS. For more information about embedded data sources, see the Microsoft documentation.

      No

      <path to report>/<custom data source name>

      The full path to the report and the custom data source name.

      You can use wildcards to match multiple folders, reports or data sets. The connection information is this section is used to add missing information or to overwrite parsed information.

      No

      dbname
      		The name of the database of a custom data source in SSRS and PBRS.

      No

      schema

      The name of the schema of a custom data source in power. If you don't provide the schema name, the default schema is used.

      No

      dialect

      The dialect of the custom data source in SSRS and PBRS.

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for a SAP Hana data source.
      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      Example 
      .
      Copy
      {
          "DataSources": {
              "Redshift": {
                  "dbname": "redshift-database-name",
                  "schema": "redshift-schema-name",
                  "dialect": "redshift",
                  "collibraSystemName": "redshift-system-name"
              },
              "Oracle": {
                  "dbname": "oracle-database-name",
                  "schema": "oracle-schema-name",
                  "dialect": "oracle",
                  "collibraSystemName": "oracle-system-name"
              }
          },
          "CustomDataSources": {
              "/path to report/custom data souce name": {
                  "dbname": "mssql-database-name",
                  "dialect": "mssql"
              }
          }
      }
      		 No
      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      		 No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Matillion connection

      The Matillion connection that you created.

      Note Collibra Platform 2023.03 or newer is required to use the Matillion connection.

      No

      Group NameThe name of your group in Matillion.

      Yes

      Project Name

      The name of your project in Matillion.

      You can only add the name of one project. If you want to create a technical lineage for other projects, add a technical lineage for Matillion capability for each project.

      Note Each capability requires a separate Matillion connection.

      Yes

      Environment Name

      The name of your environment in Matillion.

      You can only add the name of one environment. If you want to create a technical lineage for other environments, add a technical lineage for Matillion capability for each environment.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Start timestamp

      The timestamp of tasks in Matillion, which indicates the amount of metadata that technical lineage via Edge collects.

      Specify this field with a UNIX timestamp in milliseconds. The default value is 1, which gets as much history as Matillion provides. Matillion provides 7 days of history by default.

      Yes

      Source Configuration

      The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

      Property

      		Description

      Mandatory?

      found_dbname=<database name>;found_hostname=<server name>

      The information of the supported data sources in Matillion to be collected by Collibra Data Lineage.

      <database name>
      The database name in Matillion.
      <server name>
      The name of the server that the database is running on. You can specify found_hostname=* to include all servers.
      Tip 

      You can use wildcards to capture multiple connection string combinations:

      PatternDescription
      *Matches everything.
      ?Matches any single character.
      [seq]Matches any character in "seq".
      [!seq]Matches any character not in "seq".

      Yes

      dbname

      The name of the database asset in Data Catalog.

      If you leave this property blank, the database is stitched to the database of DEFAULT in Data Catalog.

      No

      schema

      The name of the schema asset in Data Catalog. Specify this property with the schema name that you created when you registered the data source.

      If you leave this property blank, the schema is stitched to the schema of DEFAULT in Data Catalog.

      No

      dialect

      Select one of the following dialects for your data source

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for a IBM Db2 data source.
      • generic, for any data source.
      • greenplum, for a Greenplum data source.
      • hana, for a SAP HANA data source.
      • hive, for a Hive data source.
      • impala, for a Impala data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for an PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      • vertica, for a Vertica data source.

      No

      collibraSystemName

      The system or server name of the data source. Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source. Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      If you leave this property blank, the system is stitched to the system of DEFAULT in Data Catalog. If you are missing lineage or your lineage objects aren’t stitching to Catalog assets in Data Catalog as you expect, ensure this property is specified properly.

      For example, if you have two databases named Customers. When you register the data source, the Customers-Europe and Customers-USA system assets are created. You can then configure this property as follows.

      {
    "found_dbname=dbtest;found_hostname=test": {
        "dbname": "Customers",
        "dialect": "mssql",
        "collibraSystemName": "Customers-Europe"
    },
    "found_dbname=testsid;found_hostname=*": {
        "dbname": "Customers",
        "schema": "oracle-schema-name",
        "dialect": "oracle",
        "collibraSystemName": "Customers-USA"
    }	
}

      Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

      No

      Copy
      {
          "found_dbname=dbtest;found_hostname=test": {
              "collibraSystemName": "mssql-system-name"
          },
          "found_dbname=testsid;found_hostname=*": {
              "dbname": "oracle-database-name",
              "schema": "oracle-schema-name",
              "collibraSystemName": "oracle-system-name"
          }    
      }
      Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

      No

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      Note A value is required, but it is not used when technical lineage for Matillion is created.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Warning 
      • You can only specify one source ID per MicroStrategy Intelligence Server. Ingesting the same MicroStrategy Intelligence Server under different source IDs will fail.
      • Any single MicroStrategy Intelligence Server can be ingested only once. If you create more than one connection for the same MicroStrategy Intelligence Server, integration will fail. If you want to ingest from multiple unique MicroStrategy Intelligence Servers, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

      Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

      We highly recommend that you specify only one source ID per MicroStrategy Intelligence Server.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      MicroStrategy connection

      The MicroStrategy connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to MicroStrategy.

      Yes

      Domain ID

      The unique reference ID of the domain in Collibra Platform in which you want to ingest the MicroStrategy assets.

      Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      		 Yes

      URL for reports

      		This optional property ensures that the correct URL to data objects in MicroStrategy is included on the asset pages of corresponding MicroStrategy assets. The required value depends on which platform you run MicroStrategy:
      • For J2EE, use: "MicroStrategy/servlet/mstrWeb"
      • For .NET, use: "MicroStrategy/asp/Main.aspx"

      No

      MicroStrategy Library URL

      If you are using a custom URL to connect to the MicroStrategy Library Server, use this field to specify the custom library URL.

      Important You only need to specify the URL if both of the following are true:
      • You are connecting to a proxy server.
      • You are not using the default, hardcoded URL to the MicroStrategy Library Server.

        Example If the URL to your MicroStrategy Library is https://collibra.microstrategy.com/MicroStrategyLibrary/api, you don't need to use this field, as that is the default, hardcoded URL. However, if the URL is something like https://collibra.microstrategy.com/MicroStrategyLibraryProd/api, then use this field and configure it as follows:
        "microStrategyLibraryUrl": "MicroStrategyLibraryProd"

      		 No

      Source configuration

      This field allows you to provide JSON code, to:

      • Specify the default domain, meaning the domain in Collibra in which the corresponding assets of MicroStrategy metadata will be ingested if domain mapping is not configured.
        If you do configure domain mapping, the default domain is still the destination domain of the MicroStrategy Server asset.
      • Optionally, specify from which MicroStrategy projects you want to ingest metadata, and into which domains you want to ingest the corresponding assets.
      • Optionally, configure data source mapping, to map the name of a data source returned by the lineage harvester to the true name of the data source.
        Note Mapping doesn't work for custom SQL.

      If you previously integrated MicroStrategy via the lineage harvester, you can copy and paste in this field the JSON code from your MicroStrategy<source ID> configuration file.

      Property

      Description

      Mandatory

      default_domain_id

      The domain in which you want the corresponding assets of MicroStrategy metadata to be ingested.

      If you configure filtering, only the MicroStrategy Server asset is ingested into this default domain.

      Yes

      filters

      This section allows you to specify:

      • From which MicroStrategy projects you want to harvest metadata.
      • Into which domains in Collibra you want to ingest the corresponding assets.

      If you don't want to filter on projects, don't include this section in your <source ID> configuration file.

      No

      domainId

      The unique resource ID of the domain (or domains) in Collibra in which you want to ingest the MicroStrategy assets.

      Tip If you use a filters section, you must include the domainId property in the section. If, by chance, you want to filter on certain projects, but you want to ingest all assets into the default domain, then the value of the domainId property must match the value of the default_domain_id property.

      For example:
      "default_domain_id": "1234567890",
    "filters": [
        {
            "domainId": "1234567890",
            "projectNames": ["MicroStrategy Tutorial","Testing_MSTR"]
        },

      To find a domain reference ID, open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      Yes

      projectIds
      		The IDs of the MicroStrategy projects from which you want to ingest metadata.

      No

      projectNames
      		The project names of the MicroStrategy projects from which you want to ingest metadata.

      No

      datasourceMapping

      This optional section allows you to configure data source mapping. Include this section only if you need to differentiate between multiple data sources that have the same name.

      Note Mapping doesn't work for custom SQL.

      No

      found_datasource

      The name of the data source that was returned by the lineage harvester, as shown in the technical lineage.

      Note The data source name is case-sensitive.

      Yes

      found_project

      The name of the project in which the data source information resides. You can specify an asterisk (*) to search for data source information across all projects.

      Yes

      mapping

      Use this section to map the data source name that was returned by the lineage harvester to the true name of the data source.

      Example You have a Redshift data source named "RD_pearl", but the lineage harvester has returned the name "Redshift_connection". You can configure the datasourceMapping section as follows:
      Copy
      {
          "datasourceMapping": [
           {
               "found_datasource": "REDSHIFT",
               "found_project": "*",
               "mapping": {
                "dbname": "RD_pearl",
                "collibraSystemName": "TV_dev"
               }
           }
          ]
      }

      Yes

      dbname

      The name of the database to which you want to map the found data source.

      Yes

      schema_name

      The name of the schema in MicroStrategy.

      No

      dialect

      The dialect of the data source in MicroStrategy.

      No

      collibraSystemName

      The system or server name of a database.

      Warning The values of this property must exactly match the name of your System asset in Collibra.

      If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

      If you set the useCollibraSystemName property to false in your lineage harvester configuration file, leave this property empty as follows: "collibraSystemName": "".

      How to configure this property if you have two databases with the same name

      Let's assume that you have a data source named Customers. You use this data source connection in two different projects, Project_A and Project_B, but they are actually two different databases. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-North and Customers-South. You can then configure this property as follows.

      Copy
      "datasourceMapping": [
              {
                  "found_datasource": "Customers",
                  "found_project": "Project_A",
                  "mapping": {
                      "dbname": "Customers",
                      "collibraSystemName": "Customers_North"
                  }
              },
              {
                  "found_datasource": "Customers",
                  "found_project": "Project_B",
                  "mapping": {
                      "dbname": "Customers",
                      "collibraSystemName": "Customers_South"
                  }
              }
          ]

      Yes

      Example 
      .
      Copy
      {
          "default_domain_id": "66ab7911-d84a-43d6-b9f7-40130jd84ds98",
          "filters": [
              {
                  "domainId": "3ae946ec-9030-46b8-ad86-a217b8f99bab",
                  "projectIDs": ["3FF7CF7D4C5A2AEBF14D8B9802698B27"]
              },
              {
                  "domainId": "21cfb3ae-0616-478e-9060-114130540be5",
                  "projectNames": ["Human Resources Analysis Module"]
              }
          ],
          "datasourceMapping": [
           {
               "found_datasource": "REDSHIFT",
               "found_project": "*",
               "mapping": {
                   "dbname": "RD_pearl",
                    "schema_name": "Default_North",
                    "dialect": "spark",
                    "collibraSystemName": "TV_dev"
               }
           }
          ]
      }
      		 No
      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      		 No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      No

      Maximum parallel requests

      This optional property allows you to specify the internal sizing, meaning the amount of tasks that can be executed at the same time.

      The default value is "1", which means that HTTP requests are run in a synchronous manner, instead of in parallel. As value of "5", for example, means that as many as 5 HTTP requests can take place in parallel.

      A lower value reduces the chances of experiencing HTTP 401 Unauthorized errors.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      You must use either this field or the Database name JSON field.

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.
      SynonymsThis query retrieves the alternative names for the database objects.
      ViewsThis query retrieves the view definitions.
      Mviews (materialized views) This query retrieves materialized view definitions.
      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name Override

      If stitching is missing specifically because you edited the full name of your Database asset, you can use this field to specify the current name of your Database asset in Data Catalog.

      Important We strongly recommend that you not edit the full name of your System, Database and Schema assets in Data Catalog. Doing so can lead to errors during the technical lineage creation process.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Warning 
      • You can only specify one source ID per Power BI service. Ingesting the same Power BI service under different source IDs will fail.
      • Any single Power BI service can be ingested only once. If you create more than one connection for the same Power BI service, integration will fail. If you want to ingest from multiple unique Power BI services, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

      Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Power BI Connection

      The Power BI connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to Power BI.

      Yes

      API URL

      The API URL of your Power BI service.

      The default value is https://api.powerbi.com.

      Important This property is only relevant for US government or national cloud Power BI customers, in which case you must include and specify values for both this property and the scope property. For complete information, consult Microsoft's documentation on Power BI for US government customers.

      		 No

      Scope

      Optional property that is intended only for customers with a different scope, such as Chinese tenants.

      Example https://analysis.chinacloudapi.cn/powerbi/api/.default

      Important If you are a US government or national cloud Power BI customer, you must include and specify values for both this property and the apiUrl property. For complete information, consult Microsoft's documentation on Power BI for US government customers.

      No

      Domain ID

      The unique reference ID of the domain in Collibra Platform in which you want to ingest the Power BI assets.

      Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      		 Yes

      Source Configuration

      This field allows you to provide JSON code for database mapping, workspace filtering and specifying the name of a System asset in Collibra.

      • Map the names of the server, database and schema that were collected by the lineage harvester to their true names.
        Note Mapping doesn't work for custom SQL.
      • Configure filtering. We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.
      • If useCollibraSystemName in the lineage harvester configuration file is set to true, use the collibraSystemName property to specify the system name of databases in Power BI. Collibra Data Lineage uses the system names to match the structure of databases in Power BI to assets in Data Catalog.

      If you previously integrated Power BI via the lineage harvester, you can copy and paste in this field the JSON code from your Power BI <source ID> configuration file.

      Property

      		Description

      Mandatory?

      found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>

      The database information of supported data sources in Power BI that is typically collected by the lineage harvester. Specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema). You then use the child properties to map the names collected by the lineage harvester to the true names.

      Important The keys that you specify must be unique. The following configuration would result in an error, because the key found_dbname=databasename1;found_hostname=*;found_schema=schema1 is specified twice.
      {
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "mssql-database-name",
		"schema": "mssql-schema-name",
		"dialect": "mssql",
		"collibraSystemName": "mssql-system-name"
	},
	"found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
		"dbname": "oracle-database-name",
		"schema": "oracle-schema-name",
		"dialect": "oracle",
		"collibraSystemName": "oracle-system-name"
	}
}
      During metadata analysis, if Collibra Data Lineage cannot match a name that you provide in this mapping – let's say, for example, you mistype the name of the database – an analyze error is produced.
      1. In the technical lineage viewer, go to the Setting tab pane, and then click Show status.
      2. In the Sources tab, select the relevant Power BI data source.
      3. In the Transformations code search field, enter "source configuration".
      4. In the Filter by drop-down list, select ANALYZE_ERROR.
        The analyze error is shown in the filter results.

      You can use wildcards to capture multiple connection string combinations:

      PatternDescription
      *Matches everything.
      ?Matches any single character.
      [seq]Matches any character in "seq".
      [!seq]Matches any character not in "seq".

      No

      dbname
      		The true name (display name) of the database collected by the lineage harvester.

      No

      schema

      The true name (display name) of the schema collected by the lineage harvester.

      If the lineage harvester fails to find a specific schema, it uses the schema you specify in this property.

      Important Schema mapping is available for schemas that come from Power Query connections. It is not available, however, if a Power Query connection is created with SQL (or MDX) statements and the schema is specified in those statements.

      No

      dialect

      The dialect of the supported data source in Power BI.

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for a Db2 data source.
      • hana, for a SAP HANA data source.
      • hive, for a Hive data source.
      • impala, for a Impala data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      collibraSystemName

      The system or server name of a database.

      Warning The value of this property must exactly match (including for case-sensitivity) the name of your System asset in Collibra.

      If you set the useCollibraSystemName property to true in your lineage harvester configuration file, but you either don't create a <source ID> configuration file, or don't specify a value for the collibraSystemName property in your <source ID> configuration file, the system name in the technical lineage is "DEFAULT".

      How to configure this property if you have two databases with the same name

      Let's assume you have two databases named Customers. When you prepare the physical data layer in Data Catalog, you create a System asset for each of these databases. Let's say you named them Customers-Europe and Customers-USA. You can then configure this property as follows.

      "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
	"dbname": "Customers",
	"schema": "mssql-schema-name",
	"dialect": "mssql",
	"collibraSystemName": "Customers-Europe"
},
"found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
	"dbname": "Customers",
	"schema": "oracle-schema-name",
	"dialect": "oracle",
	"collibraSystemName": "Customers-USA"
},

      Yes

      filters

      This section allows you to specify the Power BI workspaces from which you want to ingest metadata.

      If you specify a capacity, all of the workspaces in that capacity are also ingested.

      Workspace filtering takes precedence over capacity filtering, meaning workspaces are filtered first. If there is no explicit exclusion of capacities containing workspaces, all capacities containing workspaces are ingested. Filtering of reports and dashboards is subordinate to workspace filtering, meaning that to include reports and dashboards from a certain workspace, that workspace has be ingested as well. Reports and dashboards from a single workspace cannot be ingested in different domains. Any configured dashboard and report filtering is then taken into consideration.

      Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name Sale and Marketing [automobiles] must be formatted as follows:
      Sale and Marketing [[]automobiles[]]

      Important If you don't want to specify the Power BI workspaces from which to ingest, you must completely remove this filters section.

      You can use wildcards to capture multiple connection string combinations:

      PatternDescription
      *Matches everything.
      ?Matches any single character.
      [seq]Matches any character in "seq".
      [!seq]Matches any character not in "seq".

      No

      domainId

      The unique resource ID of the domain (or domains), in Collibra Platform, in which you want to ingest the Power BI assets.

      You can find the domain ID by clicking the domain type. Then look in the URL of your browser to find the ID. The URL looks like https://<yourcollibrainstance>/domain/<domain ID>?<view>.

      Yes

      description

      Any description, as you see fit.

      No

      workspaceNames

      The names of Power BI workspaces from which you want to ingest metadata.

      Any meta-characters in the name of a workspace must be enclosed in square brackets "[ ]". For example, a workspace with the name "Sale and Marketing [automobiles]" should be formatted as follows:
      Sale and Marketing [[]automobiles[]]

      No

      workspaceIds

      The IDs of Power BI workspaces from which you want to ingest metadata.

      We highly recommend that you read through Filtering Power BI workspaces for important information and guidance before configuring your filters.

      		No
      capacityNames

      The names of capacities on which you want to filter.

      		No
      capacityIds

      The IDs of capacities on which you want to filter.

      Important All letters in a capacity ID must be in upper case.

      		No
      excludeWorkspaceNames

      The names of Power BI workspaces that you want to exclude from the ingestion job.

      This is useful if you want to exclude, for example, dedicated development and testing workspaces.

      The metadata of inactive and personal workspaces is not harvested or uploaded to the Collibra Data Lineage service instance. An inactive workspace is one for which no reports or dashboards have been viewed in the past 60 days. My workspace is the personal workspace for any Power BI customer to work with their own, personal content.

      For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

      		No
      excludeWorkspaceIds

      The IDs of Power BI workspaces that you want to exclude from the ingestion job.

      This is useful if you want to exclude, for example, dedicated development and testing workspaces.

      For complete details on the advantages, limitations and configuration considerations of this property, see Filtering Power BI workspaces.

      		No
      Example 
      Copy
      {
          "found_dbname=databasename1;found_hostname=*;found_schema=schema1": {
              "dbname": "mssql-database-name",
              "schema": "mssql-schema-name",
              "dialect": "mssql",
              "collibraSystemName": "mssql-system-name"
          },
          "found_dbname=databasename2;found_hostname=server-name.onmicrosoft.com;found_schema=schema2": {
              "dbname": "oracle-database-name",
              "schema": "oracle-schema-name",
              "dialect": "oracle",
              "collibraSystemName": "oracle-system-name"
          },
          "filters":[ 
              {
              "domainId": "<domain-ref-id>",
              "description": "FirstFilter",
              "workspaceNames": ["*"],
              "excludeWorkspaceIds": ["workspaceC", "workspaceD"]
              },
              {
              "domainId": "<domain-ref-id>",
              "description": "SecondFilter",
              "workspaceNames": ["workspace3", "workspace4"],
              "capacityIds": ["id1","id2"]
              }
          ]
      }
      		 No
      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Metadata is harvested and uploaded in a ZIP file to a Collibra Data Lineage service instance, for processing.

      Use this optional property to specify whether or not the raw metadata should be deleted after it has been processed.

      If you select this option, the raw metadata is deleted after processing. If you don't select this option, it is stored in an Amazon S3 bucket.

      		 

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      		 No

      Use HTTP/1.1 protocol

      		Option to use HTTP/1.1 streams, in case file-size limitations are resulting in timeout errors when using the default HTTP/2 streams. No

      Enable lineage for DAX queries

      Important This is a cloud-only feature. It is not supported for Collibra Platform for Government or Collibra Platform Self-Hosted (CPSH) environments.

      Option to enable DAX analysis via Collibra AI. This feature:

      • Creates column-level lineage that includes your calculated columns and measures in Power BI.
      • Enables stitching between calculated columns in the technical lineage and the corresponding Power BI Column assets in Data Catalog.

      Select this option to enable DAX analysis.

      Clear the checkbox to disable DAX analysis.

      For complete information on this feature, go to DAX analysis via Collibra AI.

      		No
      Enable Attribute Level Lineage for Reports (Preview)

      Option to enable attribute-level (field-level) lineage for Power BI. Enabling this feature will increase the processing time.

      For more information, go to Attribute-level lineage for Power BI (in preview) .

      		 No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Qlik connection

      The Qlik connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to Qlik.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Save Input Metadata

      Select the checkbox if you want to save the input metadata extracted from the data source in ZIP files. The files can be useful for troubleshooting. Select this option only on request of Collibra Support. If this option is selected, you can download the files from the Synchronization Result dialog box once the synchronization activity is completed.

      No

      Logging configuration

      Memory (MiB)

      JVM arguments

      These fields are configuration options that can help when investigating issues with the capability.

      Important Use these fields only at the request of Collibra Support.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Note If you are migrating an SAP HANA data source from the lineage harvester, ensure that you run the ignore-source command with the source ID from the lineage harvester configuration file. When you synchronize this capability, an error occurs if the source ID from the lineage harvester exists even if you use the same source ID for this field. For more information, go to Migrate the technical lineage of a data source.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription

      Columns

      		This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Calculated Views

      This query retrieves calculated views.

      Dependencies of Calculated Views

      This query retrieves dependencies of calculated views.

      Cross-references of Calculated Views

      		Cross-references of Calculated Views

      Calculated view/views synonyms

      This query retrieves all synonyms for views.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      SQL Active

      An option to determine whether to include or remove the technical lineage of the data source with the SQL based input.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      If you have a capability with this option selected and the synchronization of the capability fails with the Missing required parameter hanaUseCloudScanner error message, go to In Edge Harvester, previously configured/working SAP HANA Classic capabilities fail to submit to Edge in Collibra Support Portal for a solution.

      No

      Calculated Views Active

      An option to determine whether to include or remove the technical lineage from calculated views in an SAP HANA Classic on-premises data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      If you have a capability with this option selected and the synchronization of the capability fails with the Missing required parameter hanaUseCloudScanner error message, go to In Edge Harvester, previously configured/working SAP HANA Classic capabilities fail to submit to Edge in Collibra Support Portal for a solution.

      No

      Use Hana Cloud for Calculated Views

      An option to determine whether to include or remove the technical lineage from calculated views in an SAP HANA Cloud/Advanced data source.

      To include technical lineage from the SAP HANA Cloud/Advanced data source, you must select this option and the Calculated Views Active option.

      Note Do not select this checkbox if:
      • You are not getting technical lineage from Calculated views.
      • You want to exclude the technical lineage of this data source.

      No

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Ingestion Method

      The Snowflake ingestion methods that Collibra Data Lineage uses to ingest metadata from Snowflake data sources. Select one of the following values:

      SQL
      The SQL Snowflake ingestion mode. Collibra Data Lineage creates a column-level technical lineage based on SQL statements.
      SQL-API
      The SQL-API Snowflake ingestion mode. Collibra Data Lineage creates a column-level technical lineage based on Snowflake schemas and the access history.

      For more information, go to Technical lineage for Snowflake ingestion methods.

      Yes

      Days

      The number of days of the user access history that Collibra Data Lineage collects and processes. For example, if you set the value to 20, Collibra Data Lineage collects the last 20 days of user access history.

      You can use this field to limit data retrieval from the ACCESS_HISTORY table. This field only takes effect when you use the SQL-API Snowflake ingestion mode.

      Specify a value in the range of 1 - 366. If you do not enter a value, all user access history is collected by default.

      Note A higher value of this field results in Collibra Data Lineage retrieving more data from Snowflake. This might cause a Usage of EmptyDir volume "output" exceeds the limit "15Gi" error when Collibra Data Lineage analyzes the metadata to create the technical lineage.

      No

      Extra Database Definitions (SQL-API mode only)

      Important This field is only valid if you're using the SQL-API ingestion method.

      The name of the database from which CollibraData Lineage collects metadata.

      This field is particularly useful for managing data lineage across multiple databases. By specifying a cross-referenced database, Collibra can accurately link and stitch lineage across databases, ensuring comprehensive visibility and precision in data relationships and dependencies, even across different database systems.

      When you specify a database in this field, Collibra retrieves the model of the specified database. Any object in the database will still appear in the Browse pane, making it accessible for lineage analysis. If the database contains view definitions, these are sent to the SQL scanner, which processes them to generate lineage.

      Tip You can add extra database definitions by clicking Add property.

      No

      Schema Names

      The schema name of your data source. This field takes effect only when you use the SQL-API Snowflake ingestion mode. You can use this field as a filter to include lineage for objects only in the specified schema.

      Ensure that the schema name you specify matches the Schema asset name that you created when you registered the data source in Data Catalog.

      Tip You can add extra schema names by clicking Add property.

      No

      Source Configuration

      This field is no longer relevant for Snowflake and will be removed from the Edge capability template in a future version of Collibra.

      No

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      If you select the SQL Snowflake ingestion mode, the following queries apply:

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.
      ProceduresThis query retrieves the stored procedures.

      Views

      This query retrieves the view definitions.

      If you select the SQL-API Snowflake ingestion mode, the following queries apply:

      QueryDescription

      Object Dependencies

      This query retrieves view definitions.

      Columns Joined

      This query retrieves table and column definition information.

      If you have missing upstream lineage information, while creating technical lineage for Snowflake with the SQL-API ingestion mode, you can use this query as a workaround to fix the issue.

      Access History

      This query retrieves lineage and transformation details.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      External Database Name

      The database value to be used as the database name in the full path (system -> database -> schema -> table). Use this field to ensure successful stitching for a database-less data source. You can specify one of the following values:

      • CData, which CDATA drivers returned as a placeholder. Use this value if you did not create a custom database name by using the CustomizedDefaultCatalogName property when you registered your data source.
      • The custom database name that you specified for the CustomizedDefaultCatalogName property when you registered your data source.

      When you registered a Spark SQL data source on Edge and did not specify the CustomizedDefaultCatalogName property, the following full path to a column in Data Catalog is created and CData is used as the database name: 

      SparkSQL_123 (system) > CData (database) > SparkSQL_ABC (schema) > Table > Column

      When Collibra Data Lineage collects metadata from the Spark SQL data source, the value of the Database Name field is used as the schema name because Spark SQL is database-less.

      If you do not enter a value for the External Database Name field, the value of the Collibra System Name field is used as the database name. If you set the Collibra system name setting to true, the value of the Collibra System Name field is also used as the system name. In this case, the full path is shown as follows and does not match the full path in Data Catalog:

      SparkSQL_123 (system) > SparkSQL_123 (database) > SparkSQL_ABC (schema) > Table > Column

      In this case, the stitching is missing because of the mismatch of the full paths. To match the full path and preserve stitching, you must specify CData for the External Database Name field.

      No

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      You must use either this field or the Database name JSON field.

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      Object namesThis query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription

      Columns

      This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Database Links

      This query retrieves links to other databases.

      SynonymsThis query retrieves the alternative names for the database objects.
      ViewsThis query retrieves the view definitions.
      Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      Yes

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

      If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <source ID>.conf file in this field.

      Property

      		DescriptionRequired?

      DataSources

      The parent element that contains the connection definitions of your data sources in SQL Server Integration Services.

      If you specify the properties in this section and also the ConnStringRegExTranslation property for a data source, the connection definitions in the ConnStringRegExTranslation property takes precedence.

      No

      DataSourceName

      The name of your data source.

      No

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      No

      ConnStringRegExTranslation

      The parent element that opens the connection definitions.

      If you specify this property and also the properties in the DataSources section for a data source, the connection definitions in this property takes precedence.

      No

      <regular expression>

      A regular expression that must match one or more connection strings.

      Note 

      Important considerations:

      • By default, the regular expression is not case sensitive. As a consequence, a regular expression can match with connection strings containing uppercase characters or lowercase characters.
      • The connection string is part of the SSIS connection manager.
      • SSIS connection managers are included in an SSIS package files (DTSX) or in connection manager files (CONMGR).
      Example 

      Regular expression: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.*
      Explanation: The first section, up to .*, is a literal, but not case-sensitive, match of the characters. The dot (.) can match any single character. The asterisk (*) means zero or more of the previous, in this case any character.
      Match: Any connection string that starts with Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.
      Example: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306;Persist Security Info=True;Auto Translate=False;.

      No

      dbname
      		The name of your database, to which the data source connection refers.

      No

      schema

      The name of your schema, to which the regular expression refers.

      No

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      No

      Copy
      {
        "DataSources": {
          "dhb-sql-prod": {
            "dialect": "mssql",
            "collibraSystemName": "my-system-name"
          }
      },
        "ConnStringRegExTranslation": {

          "Data Source=dhb-sql-prod;Initial Catalog=SFG_repl_staging;Provider=SQLNCLI11;Integrated Security=SSPI.*": {
            "dbname": "DATAHUB",
            "schema": "DBO",
            "dialect": "mssql",
            "collibraSystemName" : "WAREHOUSE"
          },

          "Server=sb-dhub;User ID=SYS_USER;Initial Catalog=STAGEDB;Port=6306.*": {
            "dbname": "STAGEDB",
            "schema": "STAGE_OWNER",
            "dialect": "sybase",
            "collibraSystemName" : ""
          }

        }
      }

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container 

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Source Configuration

      The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

      If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the <source ID>.conf file in this field.

      Property

      		DescriptionRequired?

      DataSources

      The parent element that contains the connection definitions of your data sources in SQL Server Integration Services.

      If you specify the properties in this section and also the ConnStringRegExTranslation property for a data source, the connection definitions in the ConnStringRegExTranslation property takes precedence.

      No

      DataSourceName

      The name of your data source.

      No

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      collibraSystemName

      The system or server name of the data source.

      Use this property with the useCollibraSystemName property in the lineage harvester configuration file to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you create when you prepare the physical data layer in Data Catalog. If you don't prepare the physical data layer, Collibra Data Lineage cannot stitch the data objects in your technical lineage to the assets in Data Catalog.

      No

      ConnStringRegExTranslation

      The parent element that opens the connection definitions.

      If you specify this property and also the properties in the DataSources section for a data source, the connection definitions in this property takes precedence.

      No

      <regular expression>

      A regular expression that must match one or more connection strings.

      Note 

      Important considerations:

      • By default, the regular expression is not case sensitive. As a consequence, a regular expression can match with connection strings containing uppercase characters or lowercase characters.
      • The connection string is part of the SSIS connection manager.
      • SSIS connection managers are included in an SSIS package files (DTSX) or in connection manager files (CONMGR).
      Example 

      Regular expression: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.*
      Explanation: The first section, up to .*, is a literal, but not case-sensitive, match of the characters. The dot (.) can match any single character. The asterisk (*) means zero or more of the previous, in this case any character.
      Match: Any connection string that starts with Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.
      Example: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306;Persist Security Info=True;Auto Translate=False;.

      No

      dbname
      		The name of your database, to which the data source connection refers.

      No

      schema

      The name of your schema, to which the regular expression refers.

      No

      dialect

      The dialect of the referenced database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      No

      collibraSystemName

      The system or server name of the data source.

      Specify this property when you set the value of the Collibra system name setting to True to override the default Collibra System asset name for this data source.

      Specify this property with the same name as the name of the System asset that you created when you registered the data source.

      No

      Copy
      {
        "DataSources": {
          "dhb-sql-prod": {
            "dialect": "mssql",
            "collibraSystemName": "my-system-name"
          }
      },
        "ConnStringRegExTranslation": {

          "Data Source=dhb-sql-prod;Initial Catalog=SFG_repl_staging;Provider=SQLNCLI11;Integrated Security=SSPI.*": {
            "dbname": "DATAHUB",
            "schema": "DBO",
            "dialect": "mssql",
            "collibraSystemName" : "WAREHOUSE"
          },

          "Server=sb-dhub;User ID=SYS_USER;Initial Catalog=STAGEDB;Port=6306.*": {
            "dbname": "STAGEDB",
            "schema": "STAGE_OWNER",
            "dialect": "sybase",
            "collibraSystemName" : ""
          }

        }
      }

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
      ViewsThis query retrieves the view definitions.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      No

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Warning 
      • You can only specify one source ID per Tableau server or Tableau online account. Ingesting the same Tableau server or Tableau online account under different source IDs will fail.
      • Any single Tableau server or Tableau online account can be ingested only once. If you create more than one connection for the same Tableau server or Tableau online account, integration will fail. If you want to ingest from multiple unique Tableau server or Tableau online accounts, you have to create a new Edge connection for each one, configure a new capability template for each one, and each must have a unique source ID.

      Warning If you are switching between the lineage harvester and Edge, the value in this field must exactly match the value of the id property in your lineage harvester configuration file.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Tableau connection

      The Tableau connection that you created for ingestion in Data Catalog.

      Tip Select the name that you provided in the Name field when you created a connection to Tableau.

      Yes

      Domain ID

      The unique reference ID of the domain in Collibra Platform in which you want to ingest the Tableau assets.

      Open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      Yes

      REST only

      Indication whether or not you want to use both the Tableau REST API and Tableau Metadata API to harvest Tableau metadata.

      • Cleared: The lineage harvester will use the REST API and Metadata API to harvest Tableau metadata.
      • Selected (default): The lineage harvester will only use the REST API to harvest Tableau metadata.
      Note This field must be cleared, to:
      • Enable technical lineage and the automatic stitching of Column assets to Tableau Data Attribute assets.
      • Harvest owner information for Tableau projects, workbooks and data models.

      No

      Exclude images

      Indication whether or not you want to excluding the downloading of images.

      • Cleared: Images are downloaded.
      • Selected (default): Images are not downloaded.

      Note The maximum number of images that can be uploaded to Collibra per day is determined by the configuration of the file upload service, in Collibra Console. For complete details, see the Upload configuration settings in DGC service configuration: options.

      No

      Site ID

      The site IDs of the Tableau sites that you want to include in the ingestion process.

      To ingest from multiple Tableau sites, enter each site ID in a separate Site ID field.

      To ingest the default Tableau site, enter "Default" or leave the field empty. This field is not case sensitive.

      Warning If you enter "Default", you must include the double quotation marks. The site IDs of any other Tableau sites must not be enclosed in double quotation marks. If the formatting of the site IDs does not conform to this detail, ingestion will fail.
      Example 

      Let's say that you want to ingest from the default Tableau site and a site named ABC.

      1. In the Site ID field, enter "Default" for the default Tableau site.
      2. Click Add property.
        An empty Site ID field is aded.
      3. In the new Site ID field, enter the ID of the Tableau site ABC.
      Tip Ensure that you specify the correct value. The correct value is the URL of the site to which you want to sign in. When you manually sign in to Tableau Server or Tableau Online, the site ID is the value that appears after /site/ in the browser address bar. In the following example URLs, the site ID is MarketingTeam:
      • Tableau Server: http://MyServer/#/site/MarketingTeam/projects
      • Tableau Online: https://10ay.online.tableau.com/#/site/MarketingTeam/workbooks

      On Tableau Server, however, the URL of the default site does not specify the site. For example, the URL for a view named Profits, on a site named Sales, is http://localhost/#/site/sales/views/profits. The URL for this same view on the default site is http://localhost/#/views/profits. The site name Sales does not figure in the URL.

      Yes

      Site Name

      The site name, or names, of the Tableau sites you specified in the Site ID field.

      If you don't provide a site ID in the Site ID field, or if you enter "Default", leave this field empty.

      You must enter a name for every site ID you enter.

      Concurrency level

      This field is intended to help if you are experiencing HTTP 401 Unauthorized errors due to too many concurrent HTTP calls, using the same token. It allows you to specify the internal sizing, meaning the amount of tasks that can be executed at the same time.

      The default value is 10, meaning as many as 10 HTTP requests can take place in parallel. Consider reducing the value if you are experiencing HTTP 401 Unauthorized errors. Setting the value to 1 effectively disables the concurrency level, so that HTTP requests will be run in a synchronous manner, instead of in parallel.

      No

      Source configuration

      This field allows you to provide JSON code for database mapping, domain mapping and filtering.

      If you previously integrated Tableau via the lineage harvester, you can copy and paste in this field the JSON code from your Tableau <source ID> configuration file.

      Property

      		DescriptionMandatory?
      collibraSystemNames

      This section contains the system information for different Tableau data sources. Depending on the kind of data source or connection, you have to specify how to connect to this data source.

      For more information, see the Tableau documentation. We also recommend to check the list of supported connectors in Tableau.

      		No
      files

      This section contains connection information to one or more files in Tableau.

      If you do not have files in Tableau, you can remove this section.

      		No
      filePath
      		The full path to the file. For example, the path to a JSON file.No
      collibraSystemName
      		The system name of the file.No
      connectors

      This section contains connection information to one or more connectors in Tableau.

      If you do not have connectors in Tableau, you can remove this section. The values that you specify for this property are not case-sensitive.

      		No
      connectorUrl
      		The URL of the connector. For example, the URL to Google Analytics.No
      collibraSystemName
      		The system name of the connector.No
      cloudFiles

      This section contains connection information to one or more cloud files in Tableau's input data.

      If you do not have cloud files in Tableau, you can remove this section.

      		No
      name
      		The name of the file. For example, the name of a Zendesk file.No
      collibraSystemName
      		The system name of the cloud file.No
      hostnameMapping

      This section allows you to map Tableau technical database, server and schema names to the respective real names, to preserve stitching.

      Warning 
      • hostnameMapping replaces the following deprecated properties, which have been removed from this topic:
        • The databaseMapping property.
        • The databases sub-section of the collibraSystemNames section.

      hostnameMapping must not be used in combination with either of these properties.

      If you use the hostnameMapping section, you can still use the collibraSystemName property in conjunction with the files, connectors or cloudfiles sub-sections.

      Example configuration

      Copy
      "hostnameMapping": {
          "found_dbname=databasename1;found_hostname=*;found_schema=test": {
              "dbname": "mssql-database-name",
              "schema": "mssql-schema-name",
              "dialect": "mssql",
              "collibraSystemName": "mssql-system-name"
              }
          }

      For more example configurations, go to Tableau hostname, schema, and system name mapping.

      		No
      found_dbname=<database name>;found_hostname=<server name>;found_schema=<schema name>

      The database information of supported data sources in Tableau that is typically collected by the lineage harvester. It allows you to specify the name of the database (found_dbname), on which server a database is running (found_hostname), and optionally, the name of the schema (found_schema).

      		No
      dbname

      The name of the database of a supported data source in Tableau.

      		No
      schema

      The name of the default schema of a supported data source in Tableau.

      If the lineage harvester fails to find a specific schema, it uses the default schema.

      		No
      dialect

      The dialect of the supported data source in Tableau.

      You don't have to specify a dialect; it will automatically be detected. If, however, you are using a dialect that is not supported, you can use this property to specify a supported dialect that is a close comparison. That way, most of your queries will be detected and processed.

      • redshift, for an Amazon Redshift data source.
      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • greenplum, for a Greenplum data source.
      • hive, for a HiveQL data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • hana, for a SAP HANA data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.
      		No

      filters

      This section defines:

      • From which Tableau sites and projects you want to harvest metadata.
      • Into which domains in Collibra you want to ingest the corresponding assets.

      Filtering is transitive, which means that all resources in a specified project, such as Tableau workbooks and all sub-projects, are ingested.

      Tableau assets that are not mapped to the specified domains, for example the Tableau Server assets and the parent projects (if you specify their sub-projects), are ingested in the default domain.

      Important 
      • Filtering does not affect the amount of raw metadata that is harvested from Tableau and sent to the Collibra Data Lineage service instance. Rather, it determines which metadata is ingested as assets in Data Catalog.
      • The domainMapping and filters sections are mutually exclusive. Do not include both domainMapping and filters sections in your JSON file.
      Tip 
      • If you want to ingest all of the projects in a Tableau site into multiple domains in Collibra, use the domainMapping section.
      • If you want to ingest all of the projects in a Tableau site into the default domain, use only the domainID property in the lineage harvester configuration file. The domainID property represents the default domain.
      • If you want to ingest all of the projects in a Tableau site into a single domain in Collibra, use site filtering.
      • If you want to ingest metadata from only some of the projects in a Tableau site, use project filtering.
      • You can use site filtering and project filtering together:
        • If filtering on the same site, this "filtering" is actually domain mapping, because nothing is filtered out. The contents of the projects are ingested in the specified domains, and the rest of the contents of the site are ingested in a different, specified domain.
        • If you are site filtering on a specific site and project filtering a different site, then site filtering is again a form of domain mapping, and the filtered projects are ingested in their specified domains.
        • If your lineage harvester configuration file includes sites that are not mentioned in the filters section of your <source ID> configuration file, those sites are ingested in the default domain.
      		No
      sites

      The Tableau sites to be ingested and the domain in which you want to ingest metadata from the Tableau sites.

      If you have only one Tableau site, do not include a sites section in your <source ID> file. Instead, use a projects section, to filter on Tableau projects. Include a sites section only if all of the following are true:

      • You have more than one Tableau site.
      • You want to ingest all of the metadata from only one Tableau site into a single domain in Collibra.
      • The domain into which you want to ingest is not the default domain, meaning the domain specified in the domainId property in your lineage harvester configuration file.

      		No
      site_name: domain_id
      site_name
      The name of the site to be ingested. The site name is case-sensitive.
      domain_id
      The unique reference ID of the domain in Collibra in which you want to ingest metadata. The domain ID is case-sensitive.
      To ingest all metadata from a Tableau site in the specified domain, specify the site name and a separate domain ID for each site that you list on the siteIds property in the lineage harvester configuration file for Tableau. If the site_name or domain_id property is not specified for a site, the metadata from the site is ingested in the default domain.

      To find a domain reference ID, open the relevant domain in Collibra. The URL looks like: https://<yourcollibrainstance>/domain/22258f64-40b6-4b16-9c08-c95f8ec0da26?view=00000000-0000-0000-0000-000000040001. In this example, the reference ID is in bold.

      Example configuration

      Copy
      {
        "filters":{
          "sites":{
            "Training":"ca60b822-781b-4b3a-b44d-f65bd107ff92"
          },
          "projects":{
            "Testing > Databricks":"e8f4e4a8-4062-4a33-9b44-3ce3e18e4e22",
            "Product Demo > Customer Insights":"a305e6f7-7a49-49aa-aa85-41b1e689121b"
          }
         }
      }
      		No
      projects

      The Tableau projects to be ingested and the domain in which you want to ingest metadata from the Tableau projects or sub-projects.

      Project filtering is not relevant for those who have an Explorer role in Tableau, because Explorers need to configure permissions for each data object in Tableau that they want to ingest. As the Administrator role has access to all data objects, project filtering allows Administrators to specify which projects to ingest.

      		No
      site_name > project_name : domain_id

      The site_name should be the Tableau site name. The project_name should be the Tableau project name.

      The domain_id should be the unique reference ID of the domain in Collibra in which you want to ingest metadata.

      When you specify the site and project names, the following rules apply:

      • Add spaces before and after >. The spaces are separators between the site and project.
      • Specify the full exact site and project names.
      • The values are case-sensitive.

      When you specify a Tableau project, all assets in the project are ingested in the specified domain. If you want to ingest assets from different Tableau projects in one domain, you can specify the same value for domain id for different projects.

      Example configuration

      Copy
      "Collibra_tab_partner_site > JB_Test_2812": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
      		No
      site_name > project_name > sub-project_name : domain_id

      The site_name should be the Tableau site name. The project_name should be the Tableau project name. Optionally, use sub-project_name to specify the Tableau sub-project name.

      The domain_id property should be the unique reference ID of the domain in Collibra in which you want to ingest metadata.

      When you specify the site, project and sub-project names, the following rules apply:

      • Add spaces before and after >. The spaces are separators between the site and project.
      • Specify the full exact site and project names.
      • The values are case-sensitive.

      Example

      Copy
      "Collibra_tab_partner_site > JB_Test_2812 > ProjectJJ2": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
      		No
      domainMapping

      This section defines in which domains in Collibra you want to ingest assets from your Tableau sites and Tableau projects.

      Domain mapping is transitive, meaning that all resources, such as Tableau workbooks and data attributes in a parent Tableau site, project or sub-project, are ingested in the same domain as the parent.

      Important The domainMapping and filters sections are mutually exclusive. Do not include both domainMapping and filters sections in your JSON file.

      Tip 
      • If you want to ingest all of the projects in a Tableau site into multiple domains in Collibra, use this domainMapping section.
      • If you want to ingest all of the projects in a Tableau site into the default domain, use only the domainID property in the lineage harvester configuration file. The domainID property represents the default domain.
        Note Tableau assets that are not mapped to specific domains via this domainMapping section, for example Tableau Server assets, are ingested in that default domain.
      • If you want to ingest all of the projects in a Tableau site into a single domain in Collibra, use site filtering.
      • If you want to ingest metadata from only some of the projects in a Tableau site, use project filtering.

      Example

      Let's say that you have a Tableau site named "Site-1". You want to ingest all Tableau projects in "Site-1" in a domain named "Domain-1" in Collibra, with the exception of one Tableau project named "Project-Default", which you want to ingest in "Domain-2". You should configure the domainMapping section as follows.

      Copy
      "domainMapping": {
           "<Site-1>": "reference-id-of-Domain-1",
           "<Site-1> > <Project-Default>": "reference-id-of-Domain-2"
          }

      If you want to specify a domain for a sub-project of "Project-Default", use the <site name> > <project name> > <sub-project name> property, as described below.

      For the properties in this domainMapping section, ensure that you maintain the spaces before and after ">", for example "Site-1 > Project-Default". The spaces serve as a separator between the site and the projects.
      		No
      site name

      The Tableau site name, followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau site.

      In the configuration file, use the actual site name, along with the domain reference ID, for example: "Collibra_tab_partner_site": "afc8cfb0-91f1-4075-a3e5-7ce6d1f9bcc9"
      		No
      site name > project name

      The Tableau project name, preceded by the name of the Tableau site to which it belongs, and followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau project.

      In the configuration file, use the actual site and project names, along with the domain reference ID, for example: "Collibra_tab_partner_site > JB_Test_2812": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
      		No
      site name > project name > sub-project name

      The Tableau sub-project name, preceded by the name of the Tableau site and project to which it belongs, and followed by the unique reference ID of the domain in Collibra in which you want to ingest resources from the Tableau sub-project.

      In the configuration file, use the actual site, project and sub-project names, along with the domain reference ID, for example: "Collibra_tab_partner_site > JB_Test_2812 > ProjectJJ2": "d224a1a5-43b4-43b2-8df0-ddf8f2726b82"
      		No
      Example 
      .

      For more hostnameMapping configuration examples and results, go to Tableau hostname, schema, and system name mapping.

      {
  "collibraSystemNames": {										
    "files": [
      {"filePath": "C:\\ProgramData\\Tableau\\Tableau Server\\data\\files\\sample.xls",
        "collibraSystemName": "sample-files"
      }
    ],
    "connectors": [
      {
        "connectorUrl": "tableau-server-connector-url.com",
        "collibraSystemName": "Oracle-connector"
      }
    ],
    "cloudFiles": [
      {
        "name": "file-name",
        "collibraSystemName": "FILE"
      }
    ]
},
  "hostnameMapping": {
	"found_dbname=*;found_hostname=abc.net;found_schema=DEFAULT": {
		"dbname": "CData",
		"schema": "Jan_1_2022",
		"dialect": "spark",
		"collibraSystemName": "TV_testing"
	}
},
  "filters": {
	 "sites":{
	   "site_name":"domain_id"
	 },
        "projects":{
          "site_name2 > project_name2": "domain-reference-id2",
          "site_name3 > project_name3 > subproject_name": "domain-reference-id2"
	 }
   }	
}
      .

      For more hostnameMapping configuration examples and results, go to Tableau hostname, schema, and system name mapping.

      {
  "collibraSystemNames": {										
    "files": [
      {"filePath": "C:\\ProgramData\\Tableau\\Tableau Server\\data\\files\\sample.xls",
        "collibraSystemName": "sample-files"
      }
    ],
    "connectors": [
      {
        "connectorUrl": "tableau-server-connector-url.com",
        "collibraSystemName": "Oracle-connector"
      }
    ],
    "cloudFiles": [
      {
        "name": "file-name",
        "collibraSystemName": "FILE"
      }
    ]
  },
  "hostnameMapping": {
	 "found_dbname=Test;found_hostname=*;found_schema=DEFAULT": {
		"dbname": "CData",
		"schema": "Jan_1_2022",
		"dialect": "spark",
		"collibraSystemName": "TV_testing"
	 }
  },
  "domainMapping": {
	 "<Site-1>": "reference-id-of-Domain-1",
	 "<Site-1> > <Project-Default>": "reference-id-of-Domain-2"
  }	
}

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      		 

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      No

      Paging

      This option allows you to customize the Tableau API pagination settings.

      The default values are sufficient in most cases; however, you can decrease them to help mitigate node limit errors, or increase them to speed up API calls.

      If the integration fails because of timeout errors due to page sizing limits, Collibra Data Lineage automatically adjusts the limits and retries. For example, if failure occurs with worksheetsPageSize set to 100, the value is automatically reduced to 50 and another integration attempt is automatically started. If it fails again, the value is again halved. If integration is still unsuccessful with an adjusted value of 1, an error is thrown and no further attempts are started. If integration is eventually successful, the page size value is restored to its original value, in this example 100, for the next synchronization.

      "paging": {
	"databasesPageSize": 100,
	"tablesPageSize": 100,
	"tablesColumnsPageSize": 100,
	"tableColumnsPageSize": 1000,
	"datasourcesPageSize": 50,
	"datasourcesFieldsPageSize": 50,
	"datasourceFieldsPageSize": 100,
	"worksheetsPageSize": 100,
	"worksheetsFieldsPageSize": 100,
	"worksheetFieldsPageSize": 1000,
	"usersPageSize": 100,
	"dashboardsPageSize": 100,
	"columnsLimit": 20,
	"fieldsLimit": 20
	}

      Settings per metadata type and descriptions

      Metadata typeSetting and description
      Dashboard
      • dashboardsPageSize: The number of dashboards per page.
      Worksheet
      • worksheetsPageSize: The number of worksheets per page.
      • worksheetsFieldsPageSize: The number of worksheet fields per page.
      Database
      • databasesPageSize: The number of databases per page.
      Table
      • tablesPageSize: The number of tables per page.
      • tablesColumnsPageSize: The number of table columns per page.
      Table columns
      • tableColumnsPageSize: The number of table columns per page.
      Users
      • usersPageSize: The number of users per page.
      Data source
      • datasourcesPageSize: The number of data sources per page.
      • datasourcesFieldsPageSize: The number of data source fields per page.
      • columnsLimit: The number of data source field columns per page.
      • fieldsLimit : The number of referenced data source fields per page.
      Data source field
      • datasourceFieldsPageSize: The number of data source fields per page.
      • columnsLimit: The number of data source field columns per page.
      • fieldsLimit : The number of referenced data source fields per page.

      No

      Debug

      This setting is not valid for this integration. It should be set to false.

      No

      Log level

      Only complete this field on the request of or together with Collibra Support.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      JDBC Connection

      The JDBC connection that you created for Catalog JDBC ingestion.

      Important If you used the TechLin Admin Connection field to specify an Edge connection, do not use this field to specify another Edge connection.

      Yes

      Do not use this field, however, if you specify an Edge connection in the TechLin Admin Connection field.

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      External Database Name

      The database value to be used as the database name in the full path (system -> database -> schema -> table). Use this field to ensure successful stitching for a database-less data source. You can specify one of the following values:

      • CData, which CDATA drivers returned as a placeholder. Use this value if you did not create a custom database name by using the CustomizedDefaultCatalogName property when you registered your data source.
      • The custom database name that you specified for the CustomizedDefaultCatalogName property when you registered your data source.

      When you registered a Teradata data source on Edge and did not specify the CustomizedDefaultCatalogName property, the following full path to a column in Data Catalog is created and CData is used as the database name: 

      Teredata_123 (system) > CData (database) > Teredata_ABC (schema) > Table > Column

      When Collibra Data Lineage collects metadata from the Teradata data source, the value of the Database Name field is used as the schema name because Teradata is database-less.

      If you do not enter a value for the External Database Name field, the value of the Collibra System Name field is used as the database name. If you set the Collibra system name setting to true, the value of the Collibra System Name field is also used as the system name. In this case, the full path is shown as follows and does not match the full path in Data Catalog:

      Teredata_123 (system) > Teredata_123 (database) > Teredata_ABC (schema) > Table > Column

      In this case, the stitching is missing because of the mismatch of the full paths. To match the full path and preserve stitching, you must specify CData for the External Database Name field.

      No

      Database Name

      The name or names of databases from which you want to harvest metadata. Click + Add Database Name, to add another database name.

      Yes

      Database name JSON

      This field provides an alternative method for providing multiple database names. You can upload or drag and drop a JSON file with database names.

      Example ["jsonDb_1", "jsonDb_2"]

      		You must use either this field or the Database Name field.

      Queries

      The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use.

      When you add a capability, default queries are shown in the code fields and the Use default value checkbox is selected. On occasion, to improve performance, we update the default queries. When that happens, the next time your data sources are synchronized, the new default queries are checked against the previous default queries. If there is a difference, the new queries are used.

      Note Collibra Data Lineage can only check for changes between the new default queries and the previous set of default queries. If the queries in your Oracle Edge capabilityTechnical Lineage for Snowflake capability are older than the previous set of queries (or if you have customized them) they are recognized as customized queries and cannot be updated. Therefore, you won't benefit from the performance improvements.

      To benefit from the performance improvements, you can create a new capability and copy the set of default queries from that into your existing capabilities. You can, then, modify them to suit your needs.

      If you want to use customized queries, clear the Use default value checkbox, and then enter your queries.

      Note 
      • If you use customized queries, ensure that you use only supported SQL syntax.
      • Collibra Support does not provide support for customized SQL queries. After synchronization, if no lineage was created, we recommend that you edit your queries or reach out to Collibra Coaching Services.

      Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

      QueryDescription
      ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

      Object Names

      		This query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

      Yes

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      Debug

      An option to enable logging of a JDBC job. If you enable logging, you can download the output file of the JDBC job in the Edge Jobs dashboard (in preview). The output file contains the logs of the JDBC driver. For more information about downloading the output file, go to Download job output files.

      Select one of the following values:

      True
      Enables logging of the JDBC job.
      False
      Disables logging of the JDBC job. This is the default value.

      No

      Log level

      An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

      No

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Capability template

      The capability template. The value that you select in this field determines which sections appear on the page.

      Select the following capability:

      Technical Lineage for SqlDirectory

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Shared Storage Connection

      The Shared Storage connection that you created.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      Yes

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      		 
      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

      FieldDescriptionRequired?

      Name

      The name of the capability.

      Yes

      Description

      The description of the capability.

      Yes

      Source ID

      The name of the data source. The name must be unique and cannot contain special characters, for example, /.

      Yes

      TechLin Admin Connection

      If you want to use the OAuth authentication type to connect to the Collibra Data Lineage service instances, you have to create a Technical Lineage Admin Edge or Collibra Cloud site connection and select the OAuth authentication type. Then, in this field, you specify the name of the Technical Lineage Admin Edge or Collibra Cloud site connection.

      No

      Cloud Connection

      The name of the AWS connection, Azure connection, or GCP connection that you created

      Yes

      Cloud Storage Bucket/Container

      The name of the bucket or container in the cloud-based storage system. Do not include the protocol or prefix, for example don't include s3:// or gs://.

      Yes

      Cloud Storage Region

      The AWS S3 cloud storage region.

      Important Use this field only if your files are stored in an AWS S3 bucket.

      No

      Azure Cloud Storage Account

      Your Azure cloud storage account.

      Important Use this field only if your files are stored in an Azure Data Lake Storage container.

      No

      Cloud Storage Path

      The path to the folder (in the container or bucket) that contains the files.

      No

      Mask

      The pattern of the file names in the directory. By default, the value is *.

      No

      Dialect

      The dialect of the database.

      You can enter one of the following values:

      • azure, for an Azure SQL Server data source.
      • bigquery, for a Google BigQuery data source.
      • db2, for an IBM DB2 data source.
      • hana, for an SAP HANA data source.
      • hana-cviews, for getting lineage from calculated views in an SAP HANA Classic on-premises data source.
      • hana-cviews-v2, for getting lineage from calculated views in an SAP HANA Cloud/Advanced data source.
        Important 

        To get technical lineage including calculated views, you must harvest SAP HANA by adding two Technical Lineage for SqlDirectory capabilities with the Shared Storage connections. In one capability, specify the hana dialect, and in the other, specify the hana-cviews or hana-cviews-v2 dialect.

      • hive, for a HiveQL data source.
      • greenplum, for a Greenplum data source.
      • mssql, for a Microsoft SQL Server data source.
      • mysql, for a MySQL data source.
      • netezza, for a Netezza data source.
      • oracle, for an Oracle data source.
      • postgres, for a PostgreSQL data source.
      • redshift, for an Amazon Redshift data source.
      • snowflake, for a Snowflake data source.
      • spark, for a Spark SQL data source.
      • sybase, for a Sybase data source.
      • teradata, for a Teradata data source.

      Yes

      Collibra System Name

      The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

      The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

      Yes

      Database

      The name of your database, which is also the name of your Database asset in Data Catalog.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Schema

      The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

      Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

      Yes

      Database Link Mapping

      If you are using DBLinks, this optional field allows you to configure, per data source, the database and schema to which DBLink points.

      The configuration format is as follows:

      {"<dblink_name>": {"database":"<database>","schema":"<schema>"}, ...}

      The schema provided here is only taken into consideration if a schema is not explicitly specified in the SQL query. As such, the schema specified here can be considered a default or fallback mapping.

      Tip  If you’re using a DBLink to target another source, you need to share the databasae model between the targeted (independent) source and the dependent source. Use the Dependent On Sources option to configure that dependency and share the database model.

      Let's say that you have two Oracle data sources, A and B.

      Source A

      You have the following database configuration:

      • Database_A > Schema_A1 > Table_1 > Column_1, Column_2
      • Database_A > Schema_A2 > Table_2 > Column_x, Column_y

      Source B

      You have DBLinks with names dblink.example.com, dblink2.example.com, and dblink3.example.com, all of which point to Source A.

      Tip Configure the Dependent On Source option, recognizing the fact that Source B is dependent on target (independent) Source A.

      You want your DBLink aliases configured as follows:

      • dblink.example.com → Database_A > Schema_A1
      • dblink2.example.com → Database_A > Schema_A1
      • dblink3.example.com → Database_A > Schema_A2

      You specify this in the Database Link Mapping field as follows:

      {
	"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink2.example.com": {"database":"Database_A","schema":"Schema_A1"},
	"dblink3.example.com": {"database":"Database_A","schema":"Schema_A2"}
	}

      With this done, the following example queries on Source B are successfully analyzed by the Collibra Data Lineage service instance:

      Multiple dblinks under the same database scope

      You can configure multiple dblinks under the same database scope:

      {
	"dbScope1": {
		"dblink.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink2.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		},
		"dblink3.example.com": {
			"database": "Database_A",
			"schema": "Schema_A1"
		}
	}	
}

      Multiple dblinks under the same schema scope

      You can configure multiple dblinks under the same schema scope. In the following example, DB1 is scoped for the entire database. DB2 is scoped based on the schema.

      {
    "DB1": {
        "[DB LINK NAME]": {
            "database": "[REMOTE DATABASE NAME]",
            "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
        }
    },
    "DB2": {
        "SCHEMA1": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        },
        "SCHEMA2": {
            "[DB LINK NAME]": {
                "database": "[REMOTE DATABASE NAME]",
                "schema": "[REMOTE DATABASE DEFAULT SCHEMA]"
            }
        }
    }
}

      Important If the same DBLink, for example dblink.example.com, exists in multiple databases, the formatting shown in the previous example still applies, but you need to enclose it in curly brackets and specify the relevant database, as follows:
      • Basic formatting, as shown in the previous example:
        "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}
      • Formatting if the DBLink exists in multiple databases and you want to apply it only in a database named "dbScope1":
        "dbScope1": {"dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      If a DBLink is referenced in multiple mappings, as shown in the following example, the first mapping is used.

      "dbScope1": {
   "dblink.example.com": {"database":"DevDB_A","schema":"DevSch_A1"}
}, 
   "dblink.example.com": {"database":"Database_A","schema":"Schema_A1"}}

      In this case, occurrences of dblink.example.com in the database named "dbScope1" are mapped to:

      "database":"DevDB_A","schema":"DevSch_A1"

      No

      Property

      This section contains the custom parameters you can specify to create technical lineage. Click Add property to add a property.

      You can use this field to set the HTTP timeout duration by adding the httpTimeout property: 

      TypeValue TypeNameExample value

      Text

      Plaintext

      httpTimeout

      Sets the HTTP timeout duration, in seconds. You can enter a value in the range of 1 to 3599. The default value is 15.

      		15
      Warning If you are a Collibra Platform for Government customer, this field is required to connect to a Collibra Data Lineage service instance:
      TypeValue TypeNameValue

      Text

      Plaintext

      techlinHost

      This is the URL of the Collibra Data Lineage service instance to which you want to upload metadata, for example techlin-europe-west1.collibra.com.

      		Example: techlin-europe-west1.collibra.com

      Text

      Secret

      techlinKey

      This is the unique API key to connect to a Collibra Data Lineage service instance.

      Specify a unique user key for each Collibra environment. If you're not sure what your user key is, contact your Collibra Collibra Account Team.

      		<your-techlin-key>

      Yes for US government customers.

      Dependent On Sources

      This option allows you to provide table-definition details from an independent data source to a data source that is dependent on those details. This is needed to avoid analysis errors and to have a complete lineage that includes lineage from the SQL statements from dependent data sources.

      To use this option, enter the source ID of the independent source.

      Let's consider an example. Let's say that you want to create a technical lineage for two data sources:

      Database1 contains the DDL that specifies that the database has a table named "Table1", which has three columns: Col1, Col2, Col3, and Col4.

      Database2 contains an SQL statement: SELECT * from Database1.Schema1.Table1.

      The SQL statement in Database2 refers to the table in Database1. Therefore, to get lineage from the statement in Database2, the table definition from Database1 must be known. In this case, we say that Database2 is dependent on Database1. Database1 is considered the independent data source.

      To configure this option, specify the Source ID of the independent data source, in this example, Database1, as shown in the following imagee

      An image showing the Dependent on Sources field in the Edge capability

      Important If a dependent data source contains lowercase column names, this feature will only work for the following dialects: Oracle, Snowflake, and Teradata. For all other dialects:
      • An analyze error is raised, prompting you to provide the DDL file.
      • The only workaround is to consolidate your SQL statements and DDL file in a single data source.

      For complete information, go to Sharing database models across data sources.

      No

      Database-System mapping

      This optional field allows you to map databases to their rightful systems, to obtain stitching. This resolves missing stitching, which occurs when Collibra Data Lineage associates multiple databases with the default system name that you provide in the Collibra System Name field.

      Let’s say you use the Collibra System Name field to specify the system name SystemA. This system name will be associated with all of the databases mentioned in your SQL statements in your SQL files.

      Now let’s say that you have a SQL statement that selects data from a database DB2 (which is in SystemB) and inserts the data in database DB3 (which is part of SystemC). Both of these databases will be associated with SystemA, and both will appear under SystemA in the Browse tab pane. Therefore, stitching fails and duplicate databases might appear.

      To resolve this issue and obtain stitching, you can use this field to map DB2 to its rightful SystemB, and DB3 to its rightful SystemC.

      No

      Delete Raw Metadata After Processing

      Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

      Select this option to indicate that the raw source metadata is deleted after processing.

      Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

      No

      Analyze Only (Deprecated)

      Important This option is deprecated and will be removed in a future version of Collibra. We recommend that you no longer use it. The mandatory Processing Level setting, below, replaces this option.
      • The "Analyze" option in the Processing Level setting is the equivalent of selecting the Analyze Only option.
      • The "Sync" option in the Processing Level setting is the equivalent of clearing the Analyze Only option.

      No

      Processing Level

      Important This setting replaces the deprecated Analyze Only option, which will be removed in a future version of Collibra.

      For each of your data sources, you have to specify one of the following values: Load, Analyze, or Sync. Then, when you synchronize your technical lineage, the following process begins:

      1. Metadata for all data sources is loaded, regardless of the value of this setting for a particular data source.
      2. Metadata from data sources for which the value of this setting is either Analyze or Sync, is analyzed.
      3. Metadata from data sources for which the value of this setting is Sync, is synchronized.

      ValueDescription
      Load

      Harvest metadata from the data source and upload it to your Collibra environment. This allows you to inspect and, if necessary, edit the harvested metadata before uploading it to the Collibra Data Lineage service instance for analysis.

      When the job is done, you can download and review the metadata:

      1. Open the Activities list.
      2. In the row containing the job, click Result.
        The Synchronization Results dialog box appears.
      3. Click download and save the ZIP file to your hard drive.

      Tip The download link resembles the following: https://integrations.collibra-abc.com/rest/2.0/files/01944f12-7665-7d9c-8bc5-aa426b6a63cc. Take note of the file ID, in this example: 01944f12-7665-7d9c-8bc5-aa426b6a63cc. After you inspect the metadata, you can send the ZIP file for analysis by using the "Analyze files" option. Alternatively, you can upload the ZIP file using the POST /files API. In either case, you need to specify the file ID.

      Analyze

      Load and analyze the metadata on the Collibra Data Lineage service instance.

      Synchronization does not start after analysis; it starts only after either:

      Important  If you want to synchronize multiple data sources, we strongly recommend that you select this option in the respective Edge or Collibra Cloud site capabilities for each of your data sources. This allows you to synchronize all data sources in a single job, thereby maximizing efficiency and mitigating the risk of failed synchronization jobs.
      Sync

      Load, analyze, and synchronize metadata from all data sources. Synchronization starts – or is queued, if another synchronization job is running – immediately after analysis.

      Important If you want to synchronize multiple data sources and you select this option, each data source is processed as a separate job. This is highly inefficient and will likely lead to failed sync jobs. For complete information and important considerations, go to Tips for successful lineage synchronization.

      Yes

      Active

      The option determines whether to include or remove the technical lineage of the data source.

      Select this option to include the technical lineage of this data source.

      Clear the checkbox to exclude the technical lineage of this data source.

      Yes

    8. Click Add.
      The capability is added to the Edge or Collibra Cloud site.
      The fields become read-only.

  37. You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /technicalLineage/harvester/{assetId} API, where {assetId} is the UUID of the asset for which to generate the technical lineage.

    If you include multiple databases in your technical lineage capability template, you don't have to synchronize each database separately. One synchronization job will harvest the metadata from all relevant databases.

    Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

    Prerequisites

    Steps

    1. Open a Database asset page of a registered database.
    2. In the tab bar, click Configuration.
    3. On the Lineage Extraction tab pane, click Synchronize Lineage.
      A notification indicates synchronization has started.
      The synchronization job is started. The lineage is ingested based on the configuration provided.
    1. Open the Database asset page of a registered database.
    2. In the tab bar, click Configuration.
    3. On the Lineage Extraction tab pane, click Add Schedule.
    4. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  38. You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

    If you include multiple databases in your technical lineage capability template, you don't have to synchronize each database separately. One synchronization job will harvest the metadata from all relevant databases.

    Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

    Prerequisites

    • The permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
      • Schemas
      • Tables
      • Columns

    Steps

    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronize Configuration tab pane, click Synchronize.
      A notification indicates synchronization has started.
      The synchronization job is started. The lineage is ingested based on the configuration provided.
    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronization Schedule tab pane, click Add schedule.
    6. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  39. You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

    If you include multiple databases in your technical lineage capability template, you don't have to synchronize each database separately. One synchronization job will harvest the metadata from all relevant databases.

    Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

    Prerequisites

    • The permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
      • Schemas
      • Tables
      • Columns

    Steps

    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronize Configuration tab pane, click Synchronize.
      A notification indicates synchronization has started.
      The synchronization job is started. The lineage is ingested based on the configuration provided.
    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronization Schedule tab pane, click Add schedule.
    6. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  40. You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

    If you include multiple databases in your technical lineage capability template, you don't have to synchronize each database separately. One synchronization job will harvest the metadata from all relevant databases.

    Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

    Prerequisites

    • The permissions to retrieve the metadata of the following database components through the JDBC Driver Database Metadata methods:
      • Schemas
      • Tables
      • Columns

    Steps

    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronize Configuration tab pane, click Synchronize.
      A notification indicates synchronization has started.
      The synchronization job is started. The lineage is ingested based on the configuration provided.
    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Find the connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the connection, expand them to find your capability.
      The capability configuration page opens.
    5. On the Synchronization Schedule tab pane, click Add schedule.
    6. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  41. beYou can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

    If you include multiple databases in your technical lineage capability template, you don't have to synchronize each database separately. One synchronization job will harvest the metadata from all relevant databases.

    Warning During the ingestion process, relations of the type "Data Element targets / sources Data Element" are automatically created between certain assets. Any relations of this type that you manually create between assets will be deleted during the synchronization process. If you want to manually create such relations and ensure that they are maintained, you can create a custom technical lineage.

    Steps

    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Locate the Databricks connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the Databricks connection, expand them to locate your technical lineage capability.
      The technical lineage capability configuration page opens.
    5. In the Configuration Section section, click Add Configuration.
    6. Complete the fields as needed.
      FieldAction
      SystemSelect the System asset in which the Databricks assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
      Catalog Name If you want to create technical lineage using views in a custom catalog instead of system tables, enter the custom catalog name in this field.
      For more information about the custom catalog, see the Requirements and permissions section.
      Include FilterTo include specific workspaces, catalogs, or schemas in technical lineage, click Add Include pattern under Include Filter. If you do not specify an include filter, technical lineage includes all lineage from Databricks Unity Catalog.

      The following rules apply when you enter the include pattern:

      • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
      • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
      • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
      • Lineage is collected only when both the source and the target are in the schema, catalog, or workspace specified in the include filter.
      • * > HR: Includes all schemas under the HR catalog.
      • * > Orders > fk*: Includes schemas starting with fk under the Orders catalog.
      • * > * > profiling: Includes all schemas named profiling under any catalog.
      Exclude FilterTo exclude certain workspaces, catalogs, or schemas from technical lineage, click Add Exclude pattern under Exclude Filter. If you do not specify an exclude filter, technical lineage includes all lineage from Databricks Unity Catalog.

      The following rules apply when you enter the exclude pattern:

      • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
      • The exclude filter takes precedence over the include filter.
      • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
      • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
      • * > testDB: Excludes all schemas under the testDB catalog.
      SQL Sources Limit

      Specify the limit on the number of SQL statements included for each relation in the technical lineage graph. The default value is 5.

      Include SQL transformations

      Select this option to enable Collibra Data Lineage to extract transformation logic from notebooks, jobs, SQL queries, and dashboards, and include it in the technical lineage viewer. You can view the transformation logic in the Source code pane of the technical lineage viewer.
      Clear the checkbox if you do not want Collibra Data Lineage to ingest transformation logic.

      Include external locations

      Select this option for Collibra Data Lineage to collect metadata from the external locations to create end-to-end lineage.

      Clear the checkbox to exclude metadata from external locations.

      Ingest Volumes (In preview)

      Select this option to collect metadata from volumes to create end-to-end lineage.

      Clear the checkbox to exclude metadata from volumes.

      Ingest Notebooks (In preview)

      Select this option to collect transformations from different entities like notebooks.

      Clear the checkbox to exclude transformations from different entities.

    7. Click Save.
    8. In the Configuration Section section, click Synchronize now.
      A notification indicates synchronization has started.
      The synchronization job is started. Collibra Data Lineage ingests the metadata from Databricks Unity Catalog and processes the metadata to create technical lineage.
    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Locate the Databricks connection that you used when you added the technical lineage capability, and click the link in the Capabilities column. If multiple capabilities exist for the Databricks connection, expand them to locate your technical lineage capability.
      The technical lineage capability configuration page opens.
    5. In the Configuration Section section, click Add Configuration.
    6. Complete the fields as needed.
      FieldAction
      SystemSelect the System asset in which the Databricks assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
      Catalog Name If you want to create technical lineage using views in a custom catalog instead of system tables, enter the custom catalog name in this field.
      For more information about the custom catalog, see the Requirements and permissions section.
      Include FilterTo include specific workspaces, catalogs, or schemas in technical lineage, click Add Include pattern under Include Filter. If you do not specify an include filter, technical lineage includes all lineage from Databricks Unity Catalog.

      The following rules apply when you enter the include pattern:

      • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
      • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
      • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
      • Lineage is collected only when both the source and the target are in the schema, catalog, or workspace specified in the include filter.
      • * > HR: Includes all schemas under the HR catalog.
      • * > Orders > fk*: Includes schemas starting with fk under the Orders catalog.
      • * > * > profiling: Includes all schemas named profiling under any catalog.
      Exclude FilterTo exclude certain workspaces, catalogs, or schemas from technical lineage, click Add Exclude pattern under Exclude Filter. If you do not specify an exclude filter, technical lineage includes all lineage from Databricks Unity Catalog.

      The following rules apply when you enter the exclude pattern:

      • Enter the plain string format, for example, WorkspaceId > CatalogName > SchemaName.
      • The exclude filter takes precedence over the include filter.
      • You can use the ? and * wildcards in the workspace IDs, catalog names, and schema names.
      • If a workspace, catalog, or schema matches multiple lines, the most detailed match is taken into account.
      • * > testDB: Excludes all schemas under the testDB catalog.
      SQL Sources Limit

      Specify the limit on the number of SQL statements included for each relation in the technical lineage graph. The default value is 5.

      Include SQL transformations

      Select this option to enable Collibra Data Lineage to extract transformation logic from notebooks, jobs, SQL queries, and dashboards, and include it in the technical lineage viewer. You can view the transformation logic in the Source code pane of the technical lineage viewer.
      Clear the checkbox if you do not want Collibra Data Lineage to ingest transformation logic.

      Include external locations

      Select this option for Collibra Data Lineage to collect metadata from the external locations to create end-to-end lineage.

      Clear the checkbox to exclude metadata from external locations.

      Ingest Volumes (In preview)

      Select this option to collect metadata from volumes to create end-to-end lineage.

      Clear the checkbox to exclude metadata from volumes.

      Ingest Notebooks (In preview)

      Select this option to collect transformations from different entities like notebooks.

      Clear the checkbox to exclude transformations from different entities.

    7. Click Save.
    8. On the Synchronization Schedule tab pane, click Add Schedule.
    9. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  42. After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra Platform.

    Steps

    1. Open the Activities list.
    2. In the row containing the synchronization job, click Result.
      The synchronization report summary appears.
    3. Optional, if you selected the Save Input Metadata checkbox when the capability was created, click Download to save the extracted metadata file.

    Tip The Job ID is useful if you need to troubleshoot the synchronization job with Collibra Support.

  43. After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra Platform.

    Steps

    1. Open the Activities list.
    2. In the row containing the synchronization job, click Result.
      The Synchronization Result dialog box opens.
    3. Optional, if you selected the Save Input Metadata checkbox when the capability was created, click Download to save the extracted metadata file.

    Tip The Job ID is useful if you need to troubleshoot the synchronization job with Collibra Support.

    If there are errors in the synchronization result, go to Technical lineage via Edge troubleshooting for Databricks in Collibra Support Portal for more information.

  44. You can synchronize your technical lineage manually or automatically by adding a synchronization schedule.

    If you want to synchronize technical lineage by using the Collibra Catalog Cloud Ingestions API, use the /genericIntegration/{ingestibleId}/run API, where {ingestibleId} is the capability ID.

    Steps

    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Locate the GCP connection that you used when you added the technical lineage for Google Dataplex capability, and click the link in the capability column. If multiple capabilities exist for the GCP connection, expand them to locate your technical lineage for Google Dataplex capability.
      The synchronization configuration page opens.
    5. In the Synchronization Configuration section, click Add Configuration.
    6. Complete the fields as needed.
      FieldAction
      SystemSelect the System asset in which the Dataplex assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
      Project IDs

      To add a Project ID where Dataplex is enabled, click Add Project Id. You can add multiple Project IDs. The capability will search in these projects.

      Important If you choose Workload Identity Federation (WIF) using GKE as the connection type when creating the GCP connection, this field is required.
      Dataplex LocationsTo add a Dataplex location, click Add Dataplex Location.
      If a new location is added in Dataplex after you created the technical lineage, you can use this field to add the location. When you synchronize the technical lineage after adding the location, Collibra Data Lineage collects data sources only from the specified location.
      For more information, go to Dataplex locations in Google Cloud documentation.
      Type of lineage

      Select the type of lineage you want to create:

      • Table lineage: Create table-level lineage.
      • Column lineage: Create column-level lineage.
      GCS BucketIf you selected Column lineage in the Type of lineage field, enter the path to the GCS bucket you created in Dataplex to store the exported lineage, for example, gs://lineage-export-bucket.
      Skip ingesting SQL queries from interactive BigQuery jobsUse this option to control whether Collibra Data Lineage ingests SQL queries from interactive BigQuery jobs.

      By default, this option is not selected. Collibra Data Lineage ingests the SQL queries and includes them in the transformation and source code on the Sources tab page.

      If you select this option, CollibraData Lineage does not ingest the SQL queries and excludes them from the transformation and source code.

    7. Click Save.
    8. Click Synchronize.
      A notification indicates the synchronization has started.
    1. On the main toolbar, click Products icon → Catalog.
      The Catalog homepage opens.
    2. In the tab bar, click Integrations.
      The Integrations page opens.
    3. Click the Integration Configuration tab.
    4. Locate the GCP connection that you used when you added the technical lineage for Google Dataplex capability, and click the link in the capability column. If multiple capabilities exist for the GCP connection, expand them to locate your technical lineage for Google Dataplex capability.
      The synchronization configuration page opens.
    5. In the Synchronization Configuration section, click Add Configuration.
    6. Complete the fields as needed.
      FieldAction
      SystemSelect the System asset in which the Dataplex assets were ingested. Collibra Data Lineage stitches the ingested data objects to the selected assets when synchronization begins.
      Project IDs

      To add a Project ID where Dataplex is enabled, click Add Project Id. You can add multiple Project IDs. The capability will search in these projects.

      Important If you choose Workload Identity Federation (WIF) using GKE as the connection type when creating the GCP connection, this field is required.
      Dataplex LocationsTo add a Dataplex location, click Add Dataplex Location.
      If a new location is added in Dataplex after you created the technical lineage, you can use this field to add the location. When you synchronize the technical lineage after adding the location, Collibra Data Lineage collects data sources only from the specified location.
      For more information, go to Dataplex locations in Google Cloud documentation.
      Type of lineage

      Select the type of lineage you want to create:

      • Table lineage: Create table-level lineage.
      • Column lineage: Create column-level lineage.
      GCS BucketIf you selected Column lineage in the Type of lineage field, enter the path to the GCS bucket you created in Dataplex to store the exported lineage, for example, gs://lineage-export-bucket.
      Skip ingesting SQL queries from interactive BigQuery jobsUse this option to control whether Collibra Data Lineage ingests SQL queries from interactive BigQuery jobs.

      By default, this option is not selected. Collibra Data Lineage ingests the SQL queries and includes them in the transformation and source code on the Sources tab page.

      If you select this option, CollibraData Lineage does not ingest the SQL queries and excludes them from the transformation and source code.

    7. Click Save.
    8. On the Synchronization Schedule tab pane, click Add Schedule.
    9. Enter the required information and click Save:
      FieldDescription
      RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
      Cron

      The Quartz Cron expression that determines when the synchronization takes place.

      This field is only visible if you select Cron expression in the Repeat field.

      Every

      The day on which you want to synchronize, for example, Sunday.

      This field is only visible if you select Weekly in the Repeat field.

      Every first

      The day of the month on which you want to synchronize, for example, Tuesday.

      This field is only visible if you select Monthly in the Repeat field.

      At

      The time at which you want to synchronize automatically, for example, 14:00.

      • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
      • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
      Time zoneThe time zone for the schedule.

  45. After you synchronize the technical lineage, you can view the ingestion report. This shows the impact of technical lineage synchronization on the assets in Collibra.

    Steps

    1. Open the Activities list.
    2. In the row containing the synchronization job, click Result.
      The Synchronization Result dialog box opens.
    3. Optional, if you selected the Save Input Metadata checkbox when the capability was created, click Download to save the extracted metadata file.

    Tip The Job ID is useful if you need to troubleshoot the synchronization job with Collibra Support.

What's next

View the technical lineage.

Important 
  • Metadata and asset details are ingested, but technical lineage is not yet available.
  • This integration is in preview. If you’d like to test the new integration, go to the Collibra Preview Portal and get started.
Important 

The following example shows a technical lineage for Google Dataplex.

Important