Add the Catalog JDBC ingestion capability for Snowflake

If both Collibra and Snowflake are prepared, you can add the Catalog JDBC ingestion capability to the JDBC Snowflake connection.

Image of the full integration process, indicating you're in the Add the capability step

Prerequisites

Steps

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the table, click the name of the site whose status is Healthy.
      The site page opens.
  2. In the Capabilities section, click Add capability.
    The Add capability page opens.
  3. Select the Catalog JDBC ingestion capability template.
  4. Enter the required information.
    FieldDescriptionRequired

    Capability

    This section contains general information about the capability.

    Name

    The name of the capability.

    Yes

    Description

    The description of the capability.

    No

    Capability template

    The capability template. The value that you select in this field determines which sections appear on the page.

    Select the following capability:

    Catalog JDBC ingestion

    Yes

    Connection

    This section contains information to connect to the data source.

    JDBC connection

    The Snowflake connection.

    Yes

    JDBC data source type (Deprecated)

    Deprecated field. The field was used to indicate the type of the data source. You no longer need to change this field. The required value is automatically identified.

    Note The automatically identified value is not shown in this page.

    Yes

    Supports schemas

    A text field where you have to enter True to enable database registration of data sources that have no schema. If the data source has schemas, you can ignore this field.

    Tip If the data source does not have a schema, Data Catalog creates a Schema asset with the same name as the full name of the database.

    No

    Other Settings

    Others

    		This section can contain additional capability properties.
    Click Add propertyAdd Other Settings to add a property.
    For all available other properties, go to Available "other properties".

    Note No validation is performed on the values you add.

    No

    General

    This section contains general information about logging.

    Debug

    An option to automatically send Edge infrastructure log files to Collibra Platform. By default, this option is set to false.

    Note We highly recommend to only send Edge infrastructure log files to Collibra Platform when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

    For more information, go to Edge logging.

    No

    Log level

    An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

    No

  5. Click Add.
    The capability is added to the Edge or Collibra Cloud site.
    The fields become read-only.

Available "Other Settings"

You can add the following properties in the Other Settings section to define how metadata is retrieved and how synchronization jobs are configured.

Note No validation is performed on the values you add.

Strategy properties

The following properties define the strategy for the retrieval of tables, columns, tags, schemas, primary keys, and foreign keys.

Name Description and possible values Type Encryption Default value
engine-strategy

engine-strategy defines which of the 2 Snowflake system schemas are used as the source for metadata. The selection depends on the required data freshness and the scale of the environment.
The possible values are:

SNOWFLAKE_INFORMATION_SCHEMA (default)
Provides real-time metadata access via standard database views. This strategy is recommended for standard schemas, because it offers faster retrieval for small to medium-sized data sets.
SNOWFLAKE_ACCOUNT_USAGE
This strategy is optimized for large-scale environments, such as environments with over 10,000 tables or millions of columns. The strategy bypasses JDBC object limits and significantly reduces ingestion time for massive schemas, with a metadata latency of up to 90 minutes.
This strategy has an impact on the required permissions.
 Text Not encrypted (plain text) SNOWFLAKE_INFORMATION_SCHEMA
tags-strategy

The tags-strategy defines the source of the source tags.
The possible values are:

SINGLE_CALL

This strategy retrieves the tags metadata for all tables in the schema through a single bulk request. This increases the performance of the Snowflake metadata synchronization for source tags.
In this case, the Snowflake source tags are retrieved from the SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES table. Therefore, you require the SELECT permission on that table.

CALL_PER_TABLE (default)

This strategy retrieves the tags metadata for each table individually using separate requests.
In this case, the Snowflake source tags are retrieved from INFORMATION_SCHEMA.tag_references_all_columns and INFORMATION_SCHEMA.tag_references tables.
Therefore, you need extra permissions:
GRANT SELECT ON VIEW MY_DB.INFORMATION_SCHEMA.TAG_REFERENCES TO ROLE <role_name>;
GRANT SELECT ON VIEW MY_DB.INFORMATION_SCHEMA.TAG_REFERENCES_ALL_COLUMNS TO ROLE <role_name>;

SKIP

During the synchronization, even if you select you want to ingest the source tags, source tags won't get ingested.
 Text Not encrypted (plain text)

CALL_PER_TABLE
schema-list-strategy

schema-list-strategy controls which schemas are retrieved.
The possible values are:

USE_DATABASE_NAME (default):
To retrieve the schemas the following query is used: SHOW SCHEMAS IN DATABASE "name" or SHOW SCHEMAS IN DATABASE "<session_db>" if connection property CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX set to true.
IGNORE_DATABASE_NAME
To retrieve the schemas the following query is used: SHOW SCHEMAS IN ACCOUNT.
 Text Not encrypted (plain text) USE_DATABASE_NAME
columns-strategy

The columns-strategy controls the method used to retrieve column metadata, such as data types, descriptions, and default values.
The possible values are:

SINGLE_CALL (default)

Retrieves column metadata for all tables in the schema using a single bulk request.
Characteristics:
Optimized latency.
Higher memory usage.
Doesn't guarantee correct values in the Column Position attribute.

CALL_PER_TABLE (default if engine-strategy is JAVA_API)

Retrieves column metadata for each table individually using separate requests.
Characteristics:
Optimized memory usage.
Higher network overhead.
Guarantees a correct value in the Column Position attribute.
 Text Not encrypted (plain text) SINGLE_CALL
primary-keys-strategy

primary-keys-strategy controls the method used to retrieve primary key metadata.
The possible values are:

SINGLE_CALL (default)
Retrieves primary key metadata for all tables in the schema using a single bulk request using: SHOW primary keys in schema *schema_name*.
CALL_PER_TABLE (default if engine-strategy is JAVA_API)
Retrieves primary key metadata for each table individually using separate requests using SHOW primary keys in table *table_name*.
SKIP
During the synchronization, primary key information isn't retrieved.
 Text Not encrypted (plain text) SINGLE_CALL
foreign-keys-strategy

The foreign-keys-strategy property controls the method used to retrieve foreign key (cross-reference) metadata.
The possible values are:

SINGLE_CALL (default)
Retrieves foreign key metadata for all tables in the schema using a single bulk request using SHOW imported keys in schema "%s"."%s";
DUAL_CALL_PER_TABLE
Retrieves foreign key metadata for each table individually using separate requests using SHOW imported keys in table "%s"."%s"."%s".
SKIP
During the synchronization, foreign keys aren't ingested.
 Text Not encrypted (plain text) SINGLE_CALL

Properties for the various synchronization jobs

The following jobs run in view of a synchronization:

  • database-list-with-metadata: This job runs when you register a data source and the Database field needs to be filled with available databases.

    Collibra queries the account to list the available databases, equivalent to the native SHOW DATABASES IN ACCOUNT command. If the account has access to only one specific database, meaning SHOW DATABASES IN ACCOUNT returns no results, only the database specified in the connection is retrieved. The retrieval relies on the standard JDBC driver behavior. The actual results depend on the installed driver version. The behavior is described for driver 3.x series. The ingestion logic retrieves the list of databases (catalogs) using the JDBC API method: metaData.getCatalogs(). This directly invokes SnowflakeDatabaseMetaData.getCatalogs() inside the Snowflake JDBC driver.

  • schema-list: This job runs in the Configuration tab of a Database asset to show the schemas in the database.

    For a schema to be included, the role must have the USAGE privilege on both the parent Database and the Schema itself. The ingestion process requests all available schemas within a database, equivalent to the native SHOW SCHEMAS IN DATABASE command.
    The retrieval relies on the standard JDBC driver behavior.The actual results depend on the installed driver version. The behavior is described for driver 3.x series. The ingestion logic calls the standard JDBC API method: metaData.getSchemas(database, null). This directly invokes SnowflakeDatabaseMetaData.getSchemas(catalog, null) inside the Snowflake JDBC driver.

  • ingest-schema: This job runs when you click the Synchronize button.

The available properties for these jobs are:

Name Description Type Encryption Default value

Warning The following properties can have a significant impact on your Edge site. Only add or update them together with Collibra Support. Ensure you add the properties in the capability, not in the connection.
database-list-with-metadata-garbage-collector The garbage collector that is used by this job in the capability.
For information about other possibilities, go to the AZUL documentation.		 Text Not encrypted (plain text) -XX:+UseParallelGC
database-list-with-metadata-requests-cpu database-list-with-metadata-requests-cpu The minimum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.
 Text Not encrypted (plain text) 100
database-list-with-metadata-limits-cpu database-list-with-metadata-limits-cpu The maximum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU. Text Not encrypted (plain text) 950
database-list-with-metadata-requests-memory

The minimum amount of memory requested by this job in the capability.

The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: database-list-with-metadata-limits-memory and database-list-with-metadata-jvm-max-memory.

 Text Not encrypted (plain text) 128
database-list-with-metadata-limits-memory

The maximum amount of memory requested by this job in the capability.

The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: database-list-with-metadata-requests-memory and database-list-with-metadata-jvm-max-memory.

 Text Not encrypted (plain text) 256
database-list-with-metadata-jvm-max-memory

The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job. The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: database-list-with-metadata-requests-memory and database-list-with-metadata-limits-memory.

Important Make sure this amount is lower than the database-list-with-metadata-limits-memory amount.

 Text Not encrypted (plain text) 256
schema-list-garbage-collector The garbage collector that is used by this job in the capability.
For information about other possibilities, go to the AZUL documentation.		 Text Not encrypted (plain text) -XX:+UseParallelGC
schema-list-requests-cpu The minimum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.
 Text Not encrypted (plain text) 100
schema-list-limits-cpu The maximum amount of CPU computing power requested by the capability.
The amount is expressed in milliCPU.		 Text Not encrypted (plain text) 950
schema-list-requests-memory

The minimum amount of memory requested by this job in the capability.

The amount is expressed in mebibytes (Mi).

 Text Not encrypted (plain text) 128
schema-list-limits-memory

The maximum amount of memory requested by this job in the capability.

The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: schema-list-requests-memory and schema-list-jvm-max-memory.

 Text Not encrypted (plain text) 256
schema-list-jvm-max-memory

The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job.

The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: schema-list-requests-memory and schema-list-limits-memory.

Important  Make sure this amount is lower than the schema-list-limits-memory amount.

 Text Not encrypted (plain text) 256
ingest-schema-garbage-collector The garbage collector that is used by this job in the capability.
For information about other possibilities, go to the AZUL documentation.		 Text Not encrypted (plain text) -XX:+UseParallelGC
ingest-schema-requests-cpu The minimum amount of CPU computing power requested by this job in the capability.
The amount is expressed in milliCPU.
 Text Not encrypted (plain text) 100
ingest-schema-limits-cpu The maximum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU. Text Not encrypted (plain text) 1500
ingest-schema-requests-memory

The minimum amount of memory requested by this job in the capability.

The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: ingest-schema-limits-memory and ingest-schema-jvm-max-memory.

 Text Not encrypted (plain text) 128
ingest-schema-limits-memory

The maximum amount of memory requested by the capability.
The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: ingest-schema-requests-memory and ingest-schema-jvm-max-memory.

 Text Not encrypted (plain text) 2048
ingest-schema-jvm-max-memory

The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job. The amount is expressed in mebibytes (Mi).

If you add this property, you also need to add the properties: ingest-schema-requests-memory and ingest-schema-limits-memory.

Important Make sure this amount is lower than the ingest-schema-limits-memory amount.

 Text Not encrypted (plain text) 2048
http-connect-timeout-seconds

The maximum amount of time allowed to create a connection with Collibra.

The value must be set to 30 or higher.

 Text Not encrypted (plain text) 30
http-read-timeout-seconds

The maximum amount of time allowed to wait for a response before closing the connection.

The value must be set to 300 or higher.

 Text Not encrypted (plain text) 300

What's next

Register the Snowflake data source.