Add an Edge capability to an Edge site

After you have created and installed an Edge site, you can add an Edge capability to perform specific tasks on a data source. For example, you can register a data source by using a JDBC connection that belongs to an Edge capability.

Prerequisites

Steps

Tip 

The information in this section varies depending on the capability template that you select.

Capability template

Select a data source and the connection type if needed to see the related information.

Currently, the information is shown for:

X
Tip 

For information about ingesting metadata from the following BI tools and creating a technical lineage via Edge, go to the dedicated sections:

  1. Open an Edge site.
    1. On the main menu, click , and then click Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of the Edge sites.
    3. In the table, click the name of the Edge site whose status is Healthy.
      The Edge site page opens.
  2. In the Capabilities section, click Add capability.
    The Add capability page is shown.
  3. Enter the required information.
    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    S3 synchronization

    Yes

    S3 service account

    		This section contains information about how to connect to Amazon S3.
    AWS Connection
    		The AWS connection to be used.

    Yes

    IAM role
    		The IAM role used by the AWS Glue crawlers.

    Yes

    Encryption options

    Select the type of encryption used to store the IAM role.

    Default: To be encrypted by Edge management server.

    Yes

    Delete Glue database left after previous synchronization of the file system

    Select the checkbox if you want the capability to delete the Glue databases created by previous runs of the capability, before the capability starts the synchronization.
    If you deselect this checkbox, the Glue databases created by previous runs are not removed. This can be useful for troubleshooting.

    By default, this checkbox is selected.

    No

    Save input metadata

    Select the checkbox if you want to save the input metadata extracted from the data source in ZIP files. The files can be useful for troubleshooting.
    Select this option only on request of Collibra Support. The Collibra Support team can provide the location of the saved ZIP files after the S3 synchronization.

    By default, this checkbox is not selected.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Databricks Unity Catalog synchronization

    Yes

    Databricks Connection

    		 
    Databricks Connection
    		The Databricks connection to be used.

    Yes

    Configuration

    		This section contains information on how to connect to Databricks Unity Catalog. 
    Save input metadata
    		If you select this option the metadata extracted from the data source will be saved in a file that can be used for troubleshooting. Select this option only on request of Collibra Support.

    No

    Exclude Schemas (will be removed soon)

    Comma-separated list of the schemas that you don't want to integrate.

    Note The listed schemas will be excluded for all databases. We recommend using this field to list the schemas that are automatically generated in a database, such as information_schema and default, and which you don't want to integrate.

    No

    Filters and Domain Mapping (Beta)

    Text in JSON format to include or exclude databases and schemas, and to configure domain mappings.
    This feature is intended for non-production use at this moment, as it is in Beta.

    • The text must be in JSON format and can contain an include and an exclude block.
    • In the include block, you can specify the domain in which specific catalogs or schemas must be ingested. The format is: “Catalog/Database > schema ”: “domain ID”. For example, "HR-db > address-schema": "30000000-0000-0000-0000-000000000000".
    • In the exclude block, you can specify the catalogs or schemas that you don't want to ingest. For example, "* > test".
    • The exclude block has priority over the include block.
    • If the include block is not present, we ingest all assets into the same domain as the System asset.
    • If there is no explicit domain mapping for a schema, we use the domain specified for the database.
    • You can use the keyword default as a domain ID. In that case, the catalog or schema will be ingested in the same domain as the System asset.
    • A match with a database has priority over a match with a schema.
    • The integration fails before the synchronization starts, if one or more domain IDs specified in the include block don't exist.
    • The integration fails before the synchronization starts if a domain ID is left empty in the include block.
    • You can use the ? and * wildcards in the catalog and schema names. If a catalog or schema matches multiple lines, the most detailed match is taken into account.
    Example 
    {
    "include": {
    "HR-db": "20000000-0000-0000-0000-000000000000",
    "HR-db > address-schema": "30000000-0000-0000-0000-000000000000",
    "Orders > fk*": "40000000-0000-0000-0000-000000000000",
    "Orders > *": "50000000-0000-0000-0000-000000000000",
    "* > profiling": "60000000-0000-0000-0000-000000000000",
    "sales": "default"
    },
    "exclude": [
    "testDB",
    " * > information_schema"
    ]
    }

    In this example:

    • Assets from the "HR-db" database will be ingested into the domain with ID "20000000-0000-0000-0000-000000000000". However, all assets from the "HR-db > address-schema" schema will be ingested into the domain with id "30000000-0000-0000-0000-000000000000".
    • All assets from the "Orders” database with schemas starting with fk (fk*) will be ingested into the domain with ID "40000000-0000-0000-0000-000000000000", and all other assets from the "Orders” database will be ingested into the domain with ID "50000000-0000-0000-0000-000000000000".
    • All assets from the "sales" database will be ingested in the same domain as the System asset.
    • Assets from the "profiling" schema will be ingested into the domain with ID "60000000-0000-0000-0000-000000000000". However, the "profiling" schema in the database "Orders" will be ingested in the domain with ID "50000000-0000-0000-0000-000000000000" because a database match has priority over a schema match.
    • All assets from the "testDB” database will be excluded.
    • All assets from the “information_schema” schema in all databases will be excluded.

    No

    Extensible Properties Mapping (Beta)

    Databricks Unity Catalog allows you to add additional properties to Catalog, Schema, and Table objects.

    "properties": {
    "property1": "string",
    "property2": "string"
    },

    Via the Extensible Properties Mapping field, you can ingest the values from the properties field in Table objects into specific attributes in Table assets. You do this by adding the mapping between the Properties fields for Table objects in Databricks Unity Catalog and the attribute IDs to ingest the data in, using a JSON string.

    Important 
    • This feature is intended for non-production use at this moment, as it is in Beta.
    • Currently, this feature is only available for Tables. Support for Catalog and Schema properties will be added soon.
    • If you use this feature, make sure to set up all required characteristic assignments for the asset types.
    • The text must be in JSON format and can contain a Tables block.
    • In each block, you specify the property name and the attribute ID to which you want to map the value in the property. The format is: "[property name]": "[attribute resource ID]". For example, "Description from source system": "00000000-0000-0000-0001-000500000074".
    Example 
    {
    "tables": {
    "delta.lastCommitTimestamp": "00000000-0000-0000-0000-000000003114"
    },
    }

    In this example:

    • In the Table assets that we create, we'll add the delta.lastCommitTimestamp value in attribute 00000000-0000-0000-0000-000000003114.
    		 
    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    ADLS synchronization

    Yes

    ADLS service account

    		This section contains the information on how to connect to Azure Data Lake Storage.
    Azure Connection
    		The ADLS connection to be used.

    Yes

    Microsoft Purview Account Name
    		The name of your Microsoft Purview account.
    If you enter a Purview account name, the integration uses Microsoft Purview for the integration.

    No

    Save Input Metadata
    		If you select this option the metadata extracted from the data source will be saved in a file that can be used for troubleshooting. Select this option only on request of Collibra Support.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following capability template to ingest Collibra Data Quality & Observability user-defined rules, metrics, and dimensions into Collibra Data Catalog:

    DQ Connector

    Yes

    DQ

    		This section contains information about the Collibra Data Quality & Observability connection.
    Base URL
    		Your Collibra Data Quality & Observability URL

    Yes

    Username
    		The Collibra Data Quality & Observability username for this connection.

    Yes

    Password
    		The Collibra Data Quality & Observability password for this connection.

    Yes

    Encryption options

    Select the type of encryption to use.

    Default: To be encrypted by Edge management server.

    Issuer of the JWT
    		If you have selected Encrypted with public key, enter your JWT issuer.

    No

    Collibra metadata modelThis section contains information about where to ingest Collibra Data Quality & Observability assets.
    DQ Rules domain id
    		The UUID of the Rulebook Domain for the ingested Collibra Data Quality & Observability rules.

    Yes

    DQ Metrics domain id
    		The UUID of the Rulebook Domain for the ingested Collibra Data Quality & Observability metrics.

    Yes

    DQ Dimensions domain id
    		The UUID of the Governance Asset Domain for the ingested Collibra Data Quality & Observability dimensions.

    Yes

    Default DQ Dimension name

    The default Data Quality Dimension, for example Accuracy, Completeness, Consistency and so on.

    Default: Completeness.

    Yes

    DQ Metric classified by DQ Dimension relation type id
    		The UUID of the Data Quality Metric classified by / classifies Data Quality Dimension relation. If left unspecified, this relation will not be added.

    No

    Assets are imported in batches of this size

    The batch size of the ingestion.

    Default: 5000.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select your Edge capability template.

    Note When you select a capability template, you may need to add required custom properties. For example, if you select the S3 synchronization capability template, you have to add credentials to configure the S3 connection.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    GCS synchronization

    Yes

    GCP service account

    		This section contains information on how to connect to Google Cloud Storage.
    GCP Connection
    		The GCP connection to be used.

    Yes

    ConfigurationThis section contains information on the configuration of the crawlers. 
    Maximum number of files per crawler
    		The maximum number of files that can be registered per crawler. The default value is 1,000.

    Yes

    Save input metadata

    Select the checkbox if you want to save the input metadata extracted from the data source in ZIP files. The files can be useful for troubleshooting. Select this option only on request of Collibra Support. The Collibra Support team can provide the location of the saved ZIP files after the synchronization.

    This checkbox is not selected by default.

    No

    Integrate Schemas from Dataplex

    Select the checkbox if you want to integrate the schemas from Dataplex based on the crawler path that will be specified in the GCS integration configuration.
    If the checkbox is not selected, no Dataplex data will be ingested.

    This checkbox is selected by default.

    No

    Project IDs
    		Add a comma-separated list of the Project IDs where Dataplex is enabled.
    The capability will search in these projects for schemas based on the crawler path that will be specified in the GCS integration configuration. If the Project IDs field is empty, the integration will search in the project included in the provided GCP Service Account Credentials JSON.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Catalog JDBC ingestion

    Yes

    Connection

    This section contains information to connect to the data source.

    JDBC connection

    The connection to the data source.

    Yes

    JDBC data source type (Deprecated)

    Deprecated field. The field was used to indicate the type of the data source. You no longer need to change this field. The required value is automatically identified.

    Note The automatically identified value is not shown in this page.

    Yes

    Supports schemas

    A text field where you have to enter True to enable database registration of data sources that have no schema. If the data source has schemas, you can ignore this field.

    Tip If the data source does not have a schema, Data Catalog creates a Schema asset with the same name as the full name of the database.

    No

    Others

    		This section can contain additional capability properties.

    Warning Adding additional properties can have a significant impact on your Edge site. Only add or update them together with Collibra Support.

    Click Add property to add a property.
    The possible properties are: ClosedShow properties

    Note No validation is performed on the values you add.

    No

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    JDBC Profiling

    Yes

    Connection

    This section contains information to connect to the data source.

    JDBC connection

    The connection to the data source.

    Yes

    Others

    		This section can contain additional capability properties.

    Warning Adding additional properties can have a significant impact on your Edge site. Only add or update them together with Collibra Support.

    Click Add property to add a property.
    The possible properties are: ClosedShow properties

    Note No validation is performed on the values you add.

    No

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Catalog JDBC Sampling

    Yes

    Connection

    This section contains information to connect to the data source.

    JDBC connection

    The connection to the data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Amazon Redshift

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Azure

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Synonyms

    This query retrieves the alternative names for the database objects.

    Views

    This query retrieves the view definitions.

    Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Azure

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Synonyms

    This query retrieves the alternative names for the database objects.

    Views

    This query retrieves the view definitions.

    Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Azure

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Synonyms

    This query retrieves the alternative names for the database objects.

    Views

    This query retrieves the view definitions.

    Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Custom Technical Lineage

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for DataStage

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    Yes

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    No

    Source Configuration

    The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

    Property

    		Description

    OdbcDataSources

    Open Database Connectivity data sources in IBM InfoSphere DataStage for which you want to create a technical lineage.

    <data-source-name>

    The ODBC data source name that you use in your DataStage projects.

    This section contains the properties to translate the database, schema and dialect.

    dbname
    		The name of your database, to which the ODBC data source connection refers.
    schema

    The name of your schema, to which the ODBC data source connection refers.

    dialect

    The dialect of the referenced database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.
    collibraSystemName

    The name of the data source's system or server.

    This property is only required when you set the value of the Collibra system name flag setting to True.

    Note 
    Specify this property with the same name as the full name of the System asset that you created when you registered the data source.

    NonOdbcConnectors

    Other data source connectors in IBM InfoSphere DataStage for which you want to create a technical lineage. For example, DB2, Oracle or Netezza.

    Note This section is optional.

    <data-source-connector-ID>

    The data source username and database of the connector that you use in your DataStage projects. This usually looks like for example admin@database-name. The combination of the username and database name should be unique.

    The following section contains the properties to translate the database, schema and dialect.

    dbname
    		The name of your database, to which the data source connection refers.
    schema

    The name of your schema, to which the data source connection refers.

    dialect

    The dialect of the referenced database.

    collibraSystemName

    The name of the data source's system or server.

    This property is only required when you set the value of the Collibra system name flag setting to True.

    Note 
    Specify this property with the same name as the full name of the System asset that you created when you registered the data source.

    Jobs

    The jobs that you want technical lineage via Edge to collect and process to create the technical lineage.

    This section is optional. The following rules apply when you specify this section:

    • Specify jobs that are executed so that the technical lineage graph does not include any job parameters with undefined values.
    • Specify only the first and parent jobs in a sequence of executed jobs. Technical lineage via Edge automatically collects all jobs that are called by the parent jobs.
      For example, if you have the a sequence of jobs that include job1, job2, job3, job4, and job5, where job1 calls job2, job2 calls job3, job3 calls job5, and job4 calls job3. Specify only job1 and job4, and technical lineage via Edge collects and processes all five jobs based on the sequence.

    If you do not specify this section, technical lineage via Edge collects all jobs, but without proper sequencing. Therefore, some inherited parameters might not be parsed.

    JobParameters
    		The runtime parameters that are not in the DSX and ENV files. You can specify multiple job parameters.
    name
    		The name of the job parameter.
    value
    		The value of the job parameter.
    {
  "OdbcDataSources": {
    "oracle-data-source": {
      "dbname": "my-oracle-database",
      "schema": "my-oracle-schema",
      "dialect": "oracle",
      "collibraSystemName": "my-system"
    },
    "mssql-data-source": {
      "dbname": "my-mssql-database",
      "schema": "my-mssql-schema",
      "dialect": "mssql",
      "collibraSystemName": "my-system"
    }
  },
  "NonOdbcConnectors": {
    
    "admin@database-name": {
      "dbname": "my-netezza-database",
      "schema": "my-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    },
    "admin@second-database-name": {
      "dbname": "my-second-netezza-database",
      "schema": "my-second-netezza-schema",
      "dialect": "netezza",
      "collibraSystemName": "my-system"
    }
  },   
  "jobs": [
    "my_job_1",
    "my_job_2"
  ],
  "jobParameters": [
    {
      "name": "job_parameter_name_1",
      "value": "job_parameter_value_1"
    },
    {
      "name": "job_parameter_name_2",
      "value": "job_parameter_value_2"
    }
  ]
}

    Tip Click Copy code to copy the example to your clipboard.Example of the connection_definitions.conf file

    Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the connection_definitions.conf file in this field.

    No

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Db2

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for BigQuery

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Project ID

    The ID of the project.

    Tip You can add extra project IDs by clicking Add property.

    Yes

    Region
    		 The location of your BigQuery data. This is the region that you specified when you create a data set.

    No

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription

    Columns

    This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Columns Tail

    This query retrieves all columns tails.

    Views

    This query retrieves the view definitions.

    Dataset names

    This query retrieves all logical units in the project.

    Other Queries

    This query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Greenplum

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Hive

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    External Database Name

    The database value to be used in the asset path (system -> database -> schema -> table).

    No

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription

    Columns

    This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Object Names

    		This query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical lineage for Informatica Intelligent Cloud Services (IICS)

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    IICS connection

    The Informatica Intelligent Cloud Services (IICS) connection that you created.

    Note Collibra Data Intelligence Cloud 2023.03 or newer is required to use the Informatica Intelligent Cloud Services (IICS) connection.

    No

    Objects

    The objects that you want to export.

    Each object requires a path and a type as shown in the following example, where,

    path
    The path to the object, which is relative to the Explore directory in IICS, for example, Sales.
    type
    The type of the object, for example, Taskflow. IICS scanner's starting point is a Taskflow. Therefore, the only meaningful types to export are: Taskflow, Project, and Folder. The types are not case sensitive.
    Note This field is required. An error occurs if this field is not specified. 
    "objects": [
	{
		"path" : "Sales",
		"type" : "Project"
	}, 
	{
		"path" : "Finance/Task_Flows",
		"type" : "Folder"
	},
	{
		"path" : "Common/Task_Flows/tf_CalendarDimension",
		"type" : "Taskflow"
	}
]

    Tip For more information about the objects that you can export and the required information, go to the Informatica documentation.

    Yes

    Parameter Files

    The Informatica Intelligent Cloud Services parameter files.

    No

    Source Configuration

    The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

    Property

    		DescriptionRequired?

    collibraSystemNames

    This section contains the system information for Informatica Intelligent Cloud Services.

    connections

    This section contains the system connection information. This is required to reference to the system or server of the connection.

    connectionName

    The name of the connection.

    		Yes
    collibraSystemName

    The system or server name of the connection.

    		Yes

    connectionDefinitions

    This section contains the database, schema and dialect information for each connection in Informatica Intelligent Cloud Services.

    Note You can add connection information for each connection in the connections section.

    connectionName

    The name of the connection. The name must match with the name in a connection name in the connections section. This property is required.

    		Yes
    databaseName

    The name of your database.

    		Yes
    schemaName

    The name of your schema.

    		Yes
    dialect

    The dialect of the connection. Specify this property to properly extract and parse queries that are related to this connection.

    You can enter one of the following values:

    • bigquery
    • db2
    • hana
    • hive
    • greenplum
    • mssql
    • mysql
    • netezza
    • oracle
    • postgres
    • redshift
    • snowflake
    • spark
    • teradata
    		No

    Copy code

    {
	"collibraSystemNames": {
		"connections": [
			{
				"connectionName": "DG_con_standby_cmdm_clientors",
				"collibraSystemName": "PUBLIC"
			},
			{
				"connectionName": "DG_con_dev_dg_dgiauser_su",
				"collibraSystemName": "PUBLIC"
			}
		]
	},
	"connectionDefinitions": [
		{
			"connectionName": "DG_con_standby_cmdm_clientors",
			"databaseName": "main",
			"schemaName": "dbo",
			"dialect": "oracle"
		},
		{
			"connectionName": "DG_con_dev_dg_dgiauser_su",
			"databaseName": "main",
			"schemaName": "dbo",
			"dialect": "oracle"
		}
	]
}

    Tip Click Copy code to copy the example to your clipboard.

    		 
    Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

    No

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Informatica PowerCenter

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    Yes

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    No

    Source Configuration

    The connection definitions and system names. Specify the following properties in JSON format and enter the content in this field.

    Property

    		Description

    connectionDefinitions

    This section contains the connection properties to a source in Informatica PowerCenter.

    <connectionName>

    The type of your source or target data source.

    This section contains the connection properties to a source or target in Informatica PowerCenter.

    dbname
    		The name of your source or target database.
    schema

    The name of your source or target schema.

    dialect

    The dialect of the referenced database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    collibraSystemNames

    This section contains the system or server name that is specified in your database and referenced in your connection.

    Note This section is only required when you set the Collibra system name flag setting to True.

    databases

    This section contains the database information. This is required to connect directly to the system or server of the database.

    dbname
    		The name of the database. The database name is the same as the name you entered in the <connectionName> section.
    collibraSystemName

    The system or server name of the database.

    connections

    This section contains the connection information. This is required to reference to the system or server of the connection.

    connectionName

    The name of the connection.

    collibraSystemName

    The system or server name of the connection.

    Copy code

    {
	"connectionDefinitions": {
		"oracle_source": {
			"dbname": "oracle-source-database-name1",
			"schema": "my Oracle source schema",
			"dialect": "oracle"
		},
		"oracle_target": {
			"dbname": "oracle-target-database-name2",
			"schema": "my other oracle target schema",
			"dialect": "oracle"
		}
	},
	"collibraSystemNames": {
		"databases": [
			{
				"dbname": "oracle-source-database-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"dbname": "oracle-target-database-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		],
		"connections": [
			{
				"connectionName": "oracle-connection-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"connectionName": "oracle-connection-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		]
	}
}

    Tip Click Copy code to copy the example to your clipboard.

    Important If you are using variables in Informatica PowerCenter, add the value of the variable instead of the name in the connection definitions. For example, if the parameter file contains $DBConnection_dwh=DWH_EXPORT then you add the following connection definitions: 
    {
	"DWH_EXPORT":

		{ "dbname": "DWH", "schema": "DBO" }
}
    {
	"connectionDefinitions": {
		"oracle_source": {
			"dbname": "oracle-source-database-name1",
			"schema": "my Oracle source schema",
			"dialect": "oracle"
		},
		"oracle_target": {
			"dbname": "oracle-target-database-name2",
			"schema": "my other oracle target schema",
			"dialect": "oracle"
		}
	},
	"collibraSystemNames": {
		"databases": [
			{
				"dbname": "oracle-source-database-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"dbname": "oracle-target-database-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		],
		"connections": [
			{
				"connectionName": "oracle-connection-name1",
				"collibraSystemName": "oracle-system-name1"
			},
			{
				"connectionName": "oracle-connection-name2",
				"collibraSystemName": "oracle-system-name2"
			}
		]
	}
}
    Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field.

    No

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Matillion

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Matillion connection

    The Matillion connection that you created.

    Note Collibra Data Intelligence Cloud 2023.03 or newer is required to use the Matillion connection.

    No

    Group Name
    		The name of your group in Matillion.

    Yes

    Project Name

    The name of your project in Matillion.

    You can only add the name of one project. If you want to create a technical lineage for other projects, add a technical lineage for Matillion capability for each project.

    Yes

    Environment Name

    The name of your environment in Matillion.

    You can only add the name of one environment. If you want to create a technical lineage for other environments, add a technical lineage for Matillion capability for each environment.

    Yes

    Dialect

    The dialect of the database.

    Select one of the following values:

    Snowflake
    A Snowflake data source.
    Redshift
    An Amazon Redshift data source.

    Yes

    Start timestamp

    The timestamp of tasks in Matillion, which indicates the amount of metadata that technical lineage via Edge collects.

    Specify this field with a UNIX timestamp in milliseconds. The default value is 1, which gets as much history as Matillion provides. Matillion provides 7 days of history by default.

    Yes

    Source Configuration

    The source configuration for the data source.

    No

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for MySQL

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Netezza

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Oracle

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.
    Database LinksThis query retrieves links to other databases.
    SynonymsThis query retrieves the alternative names for the database objects.
    ViewsThis query retrieves the view definitions.
    Materialized Views This query retrieves materialized view definitions.
    Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for PostgreSQL

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SAP HANA

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Note If you are migrating an SAP HANA data source from the lineage harvester, ensure that you run the ignore-source command with the source ID from the lineage harvester configuration file. When you synchronize this capability, an error occurs if the source ID from the lineage harvester exists even if you use the same source ID for this field. For more information, go to Migrate the technical lineage of a data source.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription

    Columns

    		This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Calculated Views

    This query retrieves calculated views.

    Dependencies of Calculated Views

    This query retrieves dependencies of calculated views.

    Cross-references of Calculated Views

    		Cross-references of Calculated Views

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    SQL Active

    An option to determine whether to include or remove the technical lineage of the data source with the SQL based input.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    No

    Calculated Views Active

    An option to determine whether to include or remove the technical lineage of the data source with the calculated views input.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Important Calculated views are not supported for SAP HANA Cloud. For details, go to Supported data sources for technical lineage.

    No

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Snowflake

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Ingestion Method

    The Snowflake ingestion methods that Collibra Data Lineage uses to ingest metadata from Snowflake data sources. Select one of the following values:

    SQL
    The SQL Snowflake ingestion mode. Collibra Data Lineage creates a column-level technical lineage based on SQL statements.
    SQL-API
    The SQL-API Snowflake ingestion mode. Collibra Data Lineage creates a column-level technical lineage based on Snowflake schemas and the access history.

    For more information, go to Technical lineage for Snowflake ingestion methods.

    Yes

    Days

    The number of days of the user access history that Collibra Data Lineage collects and processes. For example, if you set the value to 20, Collibra Data Lineage collects the last 20 days of user access history.

    You can use this field to limit reading from the ACCESS_HISTORY table. This field only takes effect when you use the SQL-API Snowflake ingestion mode.

    Specify a value in the range of 1 - 366. If you do not enter a value, all user access history is collected by default.

    No

    Extra Database Definitions

    The name of the database from which Collibra Data Lineage collects metadata, but the database is excluded from the technical lineage that is created. This field is useful for stitching across databases. You can specify a cross-referenced database to ensure correct lineage across all databases that Collibra Data Lineage processes to create the technical lineage.

    Tip You can add extra database definitions by clicking Add property.

    No

    Schema Names

    The schema name of your data source. This field takes effect only when you use the SQL-API Snowflake ingestion mode. You can use this field as a filter to include lineage for objects only in the specified schema.

    Ensure that the schema name you specify matches the Schema asset name that you created when you registered the data source in Data Catalog.

    Tip You can add extra schema names by clicking Add property.

    No

    Source Configuration

    The source configuration for the data source. Specify the following property in JSON format and enter the content in this field. This field applies only when you select the SQL-API Snowflake ingestion mode.

    Property

    		Description

    Required?

    displaySampleQueries

    Indicates whether to display transformations with a question mark (?) or with actual values from queries in the Source code pane in the technical lineage graph. For example, you can choose to display WHERE amount < 100 or WHERE amount < ?.

    Specify one of the following values:

    true
    Actual values from queries are displayed.
    false
    A question mark (?) is displayed. This is the default value
    		No

    Copy code

    			{
			"displaySampleQueries": true
		}

    Tip Click Copy code to copy the example to your clipboard.

    No

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.

    If you select the SQL Snowflake ingestion mode, the following queries apply:

    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Views

    This query retrieves the view definitions.

    If you select the SQL-API Snowflake ingestion mode, the following queries apply:

    QueryDescription

    Object Dependencies

    This query retrieves view definitions.

    Columns Joined

    This query retrieves table and column definition information.

    Access History

    This query retrieves lineage and transformation details.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Spark SQL

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Select the Technical lineage capability template for your data source to create a technical lineage for the JDBC data source.

    Important Technical lineage via Edge is only available in private beta. Please create a support ticket to get access.

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    External Database Name

    The database value to be used in the asset path (system -> database -> schema -> table).

    When ingesting Spark SQL via Edge with a CData driver, the database name is set to CData. Use this field to specify the true database name, to achieve stitching.

    No

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    Object namesThis query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SQL Server

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription

    Columns

    This query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Database Links

    This query retrieves links to other databases.

    SynonymsThis query retrieves the alternative names for the database objects.
    ViewsThis query retrieves the view definitions.
    Other QueriesThis query retrieves other data that technical lineage needs, for example stored procedures, functions, and packages.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SQL Server Integration Services (SSIS)

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    Yes

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    No

    Source Configuration

    The connection definitions, where you specify relevant translations for each data source. Specify the following properties in JSON format and enter the content in this field.

    Property

    		Description

    ConnStringRegExTranslation

    The parent element that opens the connection definitions.

    <regular expression>

    A regular expression that must match one or more connection strings.

    Note 

    Important considerations:

    • By default, the regular expression is not case sensitive. As a consequence, a regular expression can match with connection strings containing uppercase characters or lowercase characters.
    • The connection string is part of the SSIS connection manager.
    • SSIS connection managers are included in an SSIS package files (DTSX) or in connection manager files (CONMGR).
    Example 

    Regular expression: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.*
    Explanation: The first section, up to .*, is a literal, but not case-sensitive, match of the characters. The dot (.) can match any single character. The asterisk (*) means zero or more of the previous, in this case any character.
    Match: Any connection string that starts with Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306.
    Example: Server=sb-dhub;User ID=SYB_USER2;Initial Catalog=STAGEDB;Port=6306;Persist Security Info=True;Auto Translate=False;.

    dbname
    		The name of your database, to which the data source connection refers.
    schema

    The name of your schema, to which the regular expression refers.

    dialect

    The dialect of the referenced database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.
    collibraSystemName

    The name of the referenced data source's system or server.

    The system or server names in table references are considered to be represented by different System assets in Data Catalog. The value of this field is used as the default system or server name.

    This property is only required when you set the value of the Collibra system name flag setting to True.

    Note Specify this property with the same name as the name of the System asset that you created when you registered the data source.

    Copy code

    {
  "ConnStringRegExTranslation": {

    "Data Source=dhb-sql-prod;Initial Catalog=SFG_repl_staging;Provider=SQLNCLI11;Integrated Security=SSPI.*": {
      "dbname": "DATAHUB",
      "schema": "DBO",
      "dialect": "mssql",
      "collibraSystemName" : "WAREHOUSE"
    },

    "Server=sb-dhub;User ID=SYS_USER;Initial Catalog=STAGEDB;Port=6306.*": {
      "dbname": "STAGEDB",
      "schema": "STAGE_OWNER",
      "dialect": "sybase",
      "collibraSystemName" : ""
    }

  }
}

    Tip Click Copy code to copy the example to your clipboard.

    Tip If you previously created a technical lineage for this data source with connection definitions by using the lineage harvester, you can enter the content from the source ID configuration file in this field. 
    {
  "ConnStringRegExTranslation": {

    "Data Source=dhb-sql-prod;Initial Catalog=SFG_repl_staging;Provider=SQLNCLI11;Integrated Security=SSPI.*": {
      "dbname": "DATAHUB",
      "schema": "DBO",
      "dialect": "mssql",
      "collibraSystemName" : "WAREHOUSE"
    },

    "Server=sb-dhub;User ID=SYS_USER;Initial Catalog=STAGEDB;Port=6306.*": {
      "dbname": "STAGEDB",
      "schema": "STAGE_OWNER",
      "dialect": "sybase",
      "collibraSystemName" : ""
    }

  }
}

    Copy code

    No

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Sybase

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves all columns, tables, schemas, databases or projects in the form: database or project > schema > table > column.
    ViewsThis query retrieves the view definitions.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for Teradata

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    JDBC Connection

    The JDBC connection that you created for Catalog JDBC ingestion.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    External Database Name

    The database value to be used in the asset path (system -> database -> schema -> table).

    No

    Database Name

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Tip You can add extra database names by clicking Add property.

    Yes

    Queries

    The queries to download all the data that is required to create technical lineage. The queries vary depending on the data source you use. The query code is automatically available. However, you can modify the query code if needed.

    Example Enter the following filter in a Views query: where v.table_schema not in ('pg_catalog', 'information_schema');. This query excludes the pg_catalog and information_schema schemas, which don't contain customer data. If you want to exclude other schemas, adjust the query to, for example where v.table_schema not in ('pg_catalog', 'information_schema', 'another_schema');.

    Note 
    • If you change queries, you can only use supported SQL syntax.
    • Collibra Support does not provide support for customized SQL files.
    QueryDescription
    ColumnsThis query retrieves the columns, tables, schemas, databases or projects fields in the form: database or project > schema > table > column.

    Object Names

    		This query retrieves a list of object names from which technical lineage can be created. The objects can include stored procedures, views, macros, and so on.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Data Intelligence Cloud. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Data Intelligence Cloud when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

    FieldDescriptionRequired?
    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    Yes

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Technical Lineage for SqlDirectory

    Yes

    Main Properties

    This section contains the information for creating a technical lineage.

    Source ID

    The name of the data source. Specify a name that is unique.

    Yes

    Shared Storage Connection

    The Shared Storage connection that you created.

    No

    Mask

    The pattern of the file names in the directory. By default, the value is *.

    Yes

    Dialect

    The dialect of the database.

    You can enter one of the following values:

    • azure, for an Azure SQL Server data source.
    • bigquery, for a Google BigQuery data source.
    • db2, for an IBM DB2 data source.
    • hana, for an SAP HANA data source.
    • hana-cviews, for getting lineage from calculated views in an SAP HANA data source.
      Important 
      • The hana-cviews dialect is supported for SAP HANA (on-premises). It is not supported for SAP HANA Cloud.
      • To get technical lineage including calculated views, you must harvest SAP HANA by specifying two data sources in the lineage harvester configuration file. In one data source, specify the hana dialect, and in the other, specify the hana-cviews dialect.
    • hive, for a HiveQL data source.
    • greenplum, for a Greenplum data source.
    • mssql, for a Microsoft SQL Server data source.
    • mysql, for a MySQL data source.
    • netezza, for a Netezza data source.
    • oracle, for an Oracle data source.
    • postgres, for a PostgreSQL data source.
    • redshift, for an Amazon Redshift data source.
    • snowflake, for a Snowflake data source.
    • spark, for a Spark SQL data source.
    • sybase, for a Sybase data source.
    • teradata, for a Teradata data source.

    Yes

    Collibra System Name

    The system or server name of the data source. This field is also the full name of your System asset in Data Catalog.

    The value of this field must be the same as the full name of the System asset that you created when you registered the data source.

    Yes

    Database

    The name of your database, which is also the name of your Database asset in Data Catalog.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Schema

    The name of the default schema, if not specified in the data source itself. This corresponds to the name of your Schema asset.

    Note The database and schema names in the SQL statements in your SQL files take precedence over the values that you provide for the Database and Schema fields in the technical lineage for SqlDirectory capability. If your SQL statements contain database and schema names, Collibra Data Lineage uses them for stitching. If your SQL statements do not contain database and schema names, Collibra Data Lineage uses the values of the Database and Schema fields in the capability for stitching. Fore more information, go to Prepare the SQL directory and Automatic stitching for technical lineage.

    Yes

    Advanced Properties

    This section contains the advanced properties for creating a technical lineage.

    		 
    Delete Raw Metadata After Processing

    Technical lineage via Edge harvests raw metadata from specified data sources and uploads it in a ZIP file to a Collibra Data Lineage service instance. This option indicates whether the raw metadata should be deleted from the Collibra Data Lineage service instance after the metadata that is targeted for ingestion in Data Catalog is processed.

    Select this option to indicate that the raw source metadata is deleted after processing.

    Clear the checkbox to keep the raw source metadata after processing. In this case, it is stored in the Collibra infrastructure.

    Note Selecting this option can negatively impact performance.

    No

    Analyze Only

    This option determines whether to only load and analyze the source data on the Collibra Data Lineage service instances.

    When you select this option, the technical lineage of the data source is not created during the synchronization of the capability. Selecting this option is equivalent to entering the load-sources and analyze command with a source specified when you use the lineage harvester.

    This option is not enabled by default.

    You can use this option to control when to start full synchronization of data sources. For example if you have three data sources, A, B, and C, you can synchronize the data sources as follows:

    • During weekdays, synchronize data sources A and B with the Analyze Only checkbox selected in the capabilities. Collibra Data Lineage only loads and analyzes data sources A and B without synchronizing the technical lineage.
    • On the weekend, synchronize data source C without selecting the Analyze Only checkbox in the capability. Collibra Data Lineage synchronizes the technical lineage for all data sources including A, B, and C.

    No

    Active

    The option determines whether to include or remove the technical lineage of the data source.

    Select this option to include the technical lineage of this data source.

    Clear the checkbox to exclude the technical lineage of this data source.

    Yes

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Collibra Protect for AWS Lake Formation

    Yes

    AWS Connection

    		This section contains information about the AWS connection to be used to connect to AWS Lake Formation.

    The AWS Lake Formation connection to be used.

    		Select an AWS Lake Formation connection.

    Yes

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Collibra Protect for BigQuery

    Yes

    GCP Connection

    		This section contains information about the GCP connection to be used to connect to Google Cloud Platform.

    The GCP connection to be used.

    		Select a GCP connection.

    Yes

    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the Edge capability.

    Yes

    Description

    The description of the Edge capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following Edge capability:

    Collibra Protect for Snowflake

    Yes

    Snowflake Connection

    		This section contains information about the JDBC connection to be used to connect to Snowflake.

    The JDBC connection to be used.

    		Select a JDBC connection.

    Yes

  4. Click Create.
    The capability is added to the Edge site.
    The fields become read-only.

More information

ADLS integration

Catalog JDBC ingestion

JDBC Profiling

Catalog JDBC Sampling

S3 synchronization

GCS synchronization

Databricks Unity Catalog integration

DQ Connector

Technical lineage for JDBC data sources and ETL tools (public beta)

Protect for AWS Lake Formation

Protect for BigQuery

Protect for Snowflake