Supported data sources for technical lineage

Collibra Data Intelligence Cloud supports many data sources and metadata sources, including JDBC data sources, ETL tools and BI tools, for which you can create a technical lineage. You use these data sources when you prepare the configuration file and Data Catalog's physical data layer.

For a complete list of required permissions per supported data source type, see the Requirements and permissions section in Prepare the lineage harvester configuration file.

Note Using an older version of a data source might not work as expected; however, we don't expect problems if you use a newer version.

JDBC data sources

The following table shows the supported JDBC data sources and driver versions that have been tested. You can connect to them via a JDBC driver or by creating a folder.

JDBC data source type

Supported versions

Connection type

Scope

Amazon Redshift

1.2.34.1058 and newer

JDBC, Folder

SQL based input without stored procedures.

Azure SQL server

Newest version

JDBC, Folder

SQL based input and stored procedures.

Azure SQL Data Warehouse

Newest version

JDBC, Folder

SQL based input and stored procedures.

Azure Synapse Analytics

Newest version

JDBC, Folder

SQL based input and stored procedures.
Google BigQuery

Newest version

JDBC, Folder

SQL based input without stored procedures.

Greenplum

6.10 and newer

JDBC, Folder

SQL based input.

HiveQL (SQL-like statements)

2.3.5 and newer

JDBC, Folder

SQL based input and connection via an AWS host.

IBM DB2

11.5 and newer

JDBC, Folder

SQL based input without stored procedures.

Oracle

11g, 12c and newer

JDBC, Folder

SQL based input and stored procedures.

PostgreSQL

9.4, 9.5 and newer

JDBC, Folder

SQL based input without stored procedures.

Microsoft SQL Server

2014, 2016 and newer

JDBC, Folder

SQL based input and stored procedures.

MySQL

5.7, 8 and newer

JDBC, Folder

SQL based input without stored procedures.

Netezza

7.2.1.0 and newer

JDBC, Folder

SQL based input without stored procedures.

SAP Hana

2.00.40 and newer

JDBC, Folder

SQL based input and SAP HANA Information views, which includes attributes, analytic views and calculation views from database table or view data sources.

Script-based calculation views and stored procedures are out of scope.
Snowflake

Newest version

JDBC, Folder
  • SQL based input without stored procedures.
  • SQL-API based input with stored procedures.

For more information, go to Technical lineage for Snowflake ingestion methods.

Spark SQL

2.4.3 and newer

JDBC, Folder

SQL based input and connection via an AWS host.

For Spark SQL data source, we recommend using the folder connection type to connect to the directory with your SQL queries.

Sybase Adaptive Server Enterprise

16.0 SP02 and newer

JDBC, Folder

SQL based input without stored procedures.

Teradata

15.0, 16.20.07.01 and newer

JDBC, Folder

SQL based input, including BTEQ scripts.

ETL tools

The following table shows the supported ETL tools and driver versions that have been tested. You can connect to them via an API or by creating a folder.

ETL tool

Supported versions

Connection type

Scope
Azure Data Factory (beta) 2 API Commonly supported transformations, activities and datasets in Azure Data Factory. For details, go to Supported Azure Data Factory transformation details.

IBM InfoSphere DataStage

11.5 and newer

Folder

Commonly used DataStage ETL components including SQL overrides and transformation details.

Collibra Data Lineage supports IBM InfoSphere DataStage transformation logic.

You have to prepare a folder with all data objects that you want to process.

Informatica Intelligent Cloud Services, specifically Cloud Data Integration

Tip Data Integration is one of the Informatica Intelligent Cloud services.

Cloud, newest only

API

Commonly used transformations in Informatica Intelligent Cloud Services: Data Integration, including SQL overrides.

Supported data sources are locally stored flat files and databases.

Informatica PowerCenter

9.6 and newer

Folder

Commonly used transformations in Informatica PowerCenter, including SQL overrides.

You have to prepare a folder with all data objects that you want to process.

Matillion

Newest version

API

SQL based input without stored procedures.

The lineage harvester can only access Redshift and Snowflake projects.

SQL Server Integration Services (SSIS)

2012 and newer

Package format version 6 or newer.

Folder

All commonly used transformations in SSIS, data flows and mappings, including SQL overrides.

Important SQL statements from Excel are not supported.

You have to prepare a folder with all data objects that you want to process.

BI tools

The following table shows the supported BI tools.

BI tool

Tested versions

Connection type

Tableau

Newest

API.

You have to prepare:

Power BI (deprecated)

Newest

API and XMLA endpoints.

You have to run the Power BI harvester and the lineage harvester to ingest Power BI metadata.

Note Integration via the Power BI harvester is deprecated. We will continue to fix issues, but the development of new features and improvements is discontinued.
Power BI Newest

API.

The new Power BI integration includes many enhancements, including consolidated harvesters, meaning you no longer need the Power BI harvester. You only need to prepare:

Looker

Newest

API.

Collibra Data Lineage automatically creates a technical lineage, but stitching is not available.

You have to prepare a lineage harvester configuration file for Looker ingestion.

SQL Server Reporting Services (SSRS) or Power BI Report Server (PBRS)

  • SSRS: 2017 and newer
  • PBRS: 2019 and newer

API.

You have to prepare:

MicroStrategy

Newest

Direct connection to the repository.

Stitching is not available and there is no true technical lineage. There is only a diagram view that you can access via a Column or Table asset, but not via MicroStrategy assets.

You have to prepare a lineage harvester configuration file for MicroStrategy ingestion.

You can access:

  • Microsoft SQL Server repository.
  • Any local or remote PostgreSQL database. The MicroStrategy Intelligence Server has an embedded PostgreSQL repository, as its default repository. For complete information on the default, embedded repository, see the MicroStrategy repository documentation.
Tip For information on ingesting metadata from the following BI tools and creating a technical lineage, see the dedicated sections:

Custom technical lineage

You can create a custom technical lineage to include data objects from data sources that are not listed above. For more information, see Working with a custom technical lineage.

Authentication

Technical lineage supports the following means of authentication:
  • For all data sources, except for external directories: username and password.
  • Google BigQuery data sources: username and password or a service account key file. For more information, see the Google BigQuery documentation.
  • Power BI: username and password or service principal.
  • Snowflake: username and password or key pair authentication.
  • Tableau: username and password or token-based authentication.
  • No other authentication methods are supported.