Technical lineage for Snowflake ingestion methods

Collibra Data Lineage creates a technical lineage for Snowflake by using the SQL or SQL-API Snowflake ingestion mode. The SQL-API mode was released as part of the 2023.02 release. With this ingestion mode, the lineage harvester extracts and displays lineage beyond the traditional SQL script based lineage.

When choosing the mode, note that the SQL mode extracts lineage by parsing SQL statements, either from schemas and views directly with a JDBC connection, or from any SQL scripts that you provide in the SQL directory. Stored procedures are not supported.

The SQL-API mode parses SQL from views and schemas, and additionally gets lineage information from the ACCESS_HISTORY system view, which is a log of all queries that are run on the system. The SQL-API mode supports stored procedures and other orchestration methods, for example, application queries, ad-hoc queries and queries from streams. This lineage includes everything that is executed in the Snowflake platform with the limitation that a query is required for the lineage to be generated. This mode also accesses much more information and may take longer for lineage processing.

Ingestion mode	Details	Release date
`SQL`	Collibra Data Lineage creates a column-level technical lineage based on SQL statements, either from Snowflake schemas and views directly with a JDBC connection, or from any SQL scripts that you provide in the SQL directory. This is the default mode.	2020
`SQL-API`	Collibra Data Lineage creates a column-level technical lineage based on Snowflake schemas and the access history. With this ingestion method, the technical lineage can provide insights on data flow, including the lineage of stored procedures.	2023.02

Ingestion mode

Details

Release date

SQL

Collibra Data Lineage creates a column-level technical lineage based on SQL statements, either from Snowflake schemas and views directly with a JDBC connection, or from any SQL scripts that you provide in the SQL directory.

This is the default mode.

2020

SQL-API

Collibra Data Lineage creates a column-level technical lineage based on Snowflake schemas and the access history. With this ingestion method, the technical lineage can provide insights on data flow, including the lineage of stored procedures.

2023.02

Use the mode property in the lineage harvester configuration file to indicate which ingestion mode you want to use.

For more information about the supported queries and transformation, go to Supported transformation details.

For more information about Snowflake, go to Snowflake Documentation.