Add the Catalog JDBC ingestion capability for Snowflake

If both Collibra and Snowflake are prepared, you can add the Catalog JDBC ingestion capability to the JDBC Snowflake connection.

Image of the full integration process, indicating you're in the Add the capability step

Prerequisites

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the table, click the name of the site whose status is Healthy.
      The site page opens.
  2. In the Capabilities section, click Add capability.
    The Add capability page opens.
  3. Select the Catalog JDBC ingestion capability template.
  4. Enter the required information.
    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the capability.

    Yes

    Description

    The description of the capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following capability:

    Catalog JDBC ingestion

    Yes

    Connection

    This section contains information to connect to the data source.

    JDBC connection

    The Snowflake connection.

    Yes

    JDBC data source type (Deprecated)

    Deprecated field. The field was used to indicate the type of the data source. You no longer need to change this field. The required value is automatically identified.

    Note The automatically identified value is not shown in this page.

    Yes

    Supports schemas

    A text field where you have to enter True to enable database registration of data sources that have no schema. If the data source has schemas, you can ignore this field.

    Tip If the data source does not have a schema, Data Catalog creates a Schema asset with the same name as the full name of the database.

    No

    Other Settings

    Others

    		This section can contain additional capability properties.
    Click Add propertyAdd Other Settings to add a property.
    For all available other properties, go to Available "other properties".

    Note No validation is performed on the values you add.

    No

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Platform. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Platform when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    For more information, go to Edge logging.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

  5. Click Add.
    The capability is added to the Edge or Collibra Cloud site.
    The fields become read-only.

Available "Other Settings"

You can add the following properties in the Other Settings section to define how metadata is retrieved and how synchronization jobs are configured.

Note No validation is performed on the values you add.

Strategy properties

The following properties define the strategy for the retrieval of tables, columns, tags, schemas, primary keys, and foreign keys.

Name Description and possible values Type Encryption Default value
engine-strategy

engine-strategy defines which of the 2 Snowflake system schemas are used as the source for metadata. The selection depends on the required data freshness and the scale of the environment.
The possible values are:

SNOWFLAKE_INFORMATION_SCHEMA (default)
Provides real-time metadata access via standard database views. This strategy is recommended for standard schemas, because it offers faster retrieval for small to medium-sized data sets.
SNOWFLAKE_ACCOUNT_USAGE
This strategy is optimized for large-scale environments, such as environments with over 10,000 tables or millions of columns. The strategy bypasses JDBC object limits and significantly reduces ingestion time for massive schemas, with a metadata latency of up to 90 minutes.
This strategy has an impact on the required permissions.
 Text Not encrypted (plain text) SNOWFLAKE_INFORMATION_SCHEMA
tags-strategy

The tags-strategy defines the source of the source tags.
The possible values are:

SINGLE_CALL

This strategy retrieves the tags metadata for all tables in the schema through a single bulk request. This increases the performance of the Snowflake metadata synchronization for source tags.
In this case, the Snowflake source tags are retrieved from the SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES table. Therefore, you require the SELECT permission on that table.

CALL_PER_TABLE (default)

This strategy retrieves the tags metadata for each table individually using separate requests.
In this case, the Snowflake source tags are retrieved from INFORMATION_SCHEMA.tag_references_all_columns and INFORMATION_SCHEMA.tag_references tables.
Therefore, you need extra permissions:
GRANT SELECT ON VIEW MY_DB.INFORMATION_SCHEMA.TAG_REFERENCES TO ROLE <role_name>;
GRANT SELECT ON VIEW MY_DB.INFORMATION_SCHEMA.TAG_REFERENCES_ALL_COLUMNS TO ROLE <role_name>;

SKIP

During the synchronization, even if you select you want to ingest the source tags, source tags won't get ingested.
 Text Not encrypted (plain text)

CALL_PER_TABLE
schema-list-strategy

schema-list-strategy controls which schemas are retrieved.
The possible values are:

USE_DATABASE_NAME (default):
To retrieve the schemas the following query is used: SHOW SCHEMAS IN DATABASE "name" or SHOW SCHEMAS IN DATABASE "<session_db>" if connection property CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX set to true.
IGNORE_DATABASE_NAME
To retrieve the schemas the following query is used: SHOW SCHEMAS IN ACCOUNT.
 Text Not encrypted (plain text) USE_DATABASE_NAME
columns-strategy

The columns-strategy controls the method used to retrieve column metadata, such as data types, descriptions, and default values.
The possible values are:

SINGLE_CALL (default)

Retrieves column metadata for all tables in the schema using a single bulk request.
Characteristics:
Optimized latency.
Higher memory usage.
Doesn't guarantee correct values in the Column Position attribute.

CALL_PER_TABLE (default if engine-strategy is JAVA_API)

Retrieves column metadata for each table individually using separate requests.
Characteristics:
Optimized memory usage.
Higher network overhead.
Guarantees a correct value in the Column Position attribute.
 Text Not encrypted (plain text) SINGLE_CALL
primary-keys-strategy

primary-keys-strategy controls the method used to retrieve primary key metadata.
The possible values are:

SINGLE_CALL (default)
Retrieves primary key metadata for all tables in the schema using a single bulk request using: SHOW primary keys in schema *schema_name*.
CALL_PER_TABLE (default if engine-strategy is JAVA_API)
Retrieves primary key metadata for each table individually using separate requests using SHOW primary keys in table *table_name*.
SKIP
During the synchronization, primary key information isn't retrieved.
 Text Not encrypted (plain text) SINGLE_CALL
foreign-keys-strategy

The foreign-keys-strategy property controls the method used to retrieve foreign key (cross-reference) metadata.
The possible values are:

SINGLE_CALL (default)
Retrieves foreign key metadata for all tables in the schema using a single bulk request using SHOW imported keys in schema "%s"."%s";
DUAL_CALL_PER_TABLE
Retrieves foreign key metadata for each table individually using separate requests using SHOW imported keys in table "%s"."%s"."%s".
SKIP
During the synchronization, foreign keys aren't ingested.
 Text Not encrypted (plain text) SINGLE_CALL

Properties for the various synchronization jobs

The following jobs run in view of a synchronization:

The available properties for these jobs are:

What's next

Register the Snowflake data source.