GCP lineage integration preflight checks

To ensure successful metadata ingestion and lineage generation, complete the following preflight checks.

In your GCP environment

Collibra Data Lineage harvests technical lineage for Google Cloud Platform (GCP) by leveraging the Data Lineage API in GCP. Be sure to review the supported transformation details to determine whether to create technical lineage for GCP or Google BigQuery.

Ensure that your GCP service account used to create a GCP connection has the following roles and permissions depending on your chosen lineage granularity. You can choose to create table-level lineage or column-level lineage for GCP when you synchronize the capability. You can choose to use the Legacy (Export API) or Streaming API (Preview) lineage flow by setting the Lineage flow field when you synchronize the capability.

The permissions your GCP service account needs depend on whether you create table-level or column-level lineage, and on the lineage flow you select:

Lineage type Role or permission Purpose
Table-level Enable the Data Lineage API in GCP

Enables Collibra Data Lineage to retrieve table-level lineage from the selected GCP projects.

For more information, go to Data Lineage API in Google Cloud documentation.

The Data Lineage Viewer role

 Grants access to table-level lineage
The BigQuery Admin role Grants access to lineage from stored procedures created by you and other GCP users.
The bigquery.jobs.get permission

Enables retrieval of SQL transformation code from BigQuery jobs.

For more information, go to IAM basic and predefined roles reference in the Google Cloud documentation.
Column-level with the Legacy (Export API) lineage flow Enable the exportMetadata API, which is in Preview.

Enables export of column-level lineage metadata.

To enable the API , contact Google by following the steps in the article How to Request Access to Google Column Level Lineage (preview) export API.

Create a Google Cloud Storage (GCS) bucket with the following permissions:

  • storage.objects.create (Storage Object Creator)

  • storage.objects.list, storage.objects.get(Storage Object Viewer)
 Provides storage for exported column-level lineage metadata, and allows Collibra Data Lineage to write and read the exported files.

One of the following permissions:

  • datacatalog.entries.exportAll (Data Catalog Admin)
  • roles/datacatalog.metadataExporter permission

Grants permission to export column-level lineage metadata.

Note Google currently classifies the datacatalog.entries.exportAll permission as Testing. Permissions at this level may be subject to change or deprecation.
The bigquery.jobs.get permission

Enables retrieval of SQL transformation code from BigQuery jobs.

For more information, go to IAM basic and predefined roles reference in the Google Cloud documentation.
Column-level with the Streaming API (Preview) lineage flow Enable the Data Lineage API in GCP

Enables Collibra Data Lineage to retrieve column-level lineage from the selected GCP projects.

For more information, go to Data Lineage API in Google Cloud documentation.

The Data Lineage Viewer role

 Grants access to column-level lineage

The following permissions:

  • bigquery.datasets.get
  • bigquery.tables.list
Enables Collibra Data Lineage to list the BigQuery tables for which it retrieves column-level lineage.
The bigquery.jobs.get permission

Enables retrieval of SQL transformation code from BigQuery jobs.

For more information, go to IAM basic and predefined roles reference in the Google Cloud documentation.
Synchronization The resourcemanager.projects.get permission (optional) Enables automatic discovery of Project IDs during synchronization.

The following permissions:

  • bigquery.jobs.create
  • bigquery.jobs.listAll

Enables SQL code extraction through INFORMATION_SCHEMA queries.

You can set the SQL code extraction method on the synchronization page.
Note If you select the Streaming API (Preview) lineage flow for column-level lineage, Collibra Data Lineage sends one Data Lineage API request for each BigQuery table. For large catalogs, request a higher quota for Data Lineage read requests in your Google Cloud project.

In your Collibra environment

Lineage enablement

  • Technical lineage via Edge is enabled in your Collibra environment.
  • You are using Collibra Platform 2024.07 or newer.
  • Your Collibra environment is configured with the latest Collibra UI.

Edge

  • You either created and installed an Edge site or were granted a Collibra Cloud site.
    Important If you're using a Collibra Cloud site, go the Collibra Cloud site documentation to check if your data source is supported.
  • The Edge site status must be Healthy.
  • You've registered the data source via Edge.
  • For technical lineage to be automatically stitched to assets in Collibra, you must have one of the following in place:

Network and proxy configuration

  • Edge can connect to all Collibra Data Lineage service instances in your geographic location.
  • Optionally, you've connected to a proxy server.
  • Optionally, use a custom certificate to allow the Edge capability to connect to your data source. In this case, you've saved the certificate as "ca.pem" in the same directory as the Edge site installer. If you've saved the certificate in another directory, use the --ca argument in the Edge site installation command.

Collibra permissions

You can connect to Collibra Data Lineage by using the basic or OAuth authentication method. The following permissions are required only if you use the basic authentication method. 

To connect to Collibra Data Lineage service instances via OAuth authentication:

To add an Edge capability:

To synchronize technical lineage: