Collibra Data Lineage visualizes lineage for GCP down to the column level. To view the technical lineage for GCP, ensure that you select Objects in the toolbar of your technical lineage graph.
Function scope
Collibra Data Lineage captures lineage for the following Google Cloud assets. Currently, only Column, Table, and File assets are processed and included in the technical lineage.
- BigQuery.
- Other Google Cloud services (GCS), only when they contribute lineage for BigQuery assets. Collibra Data Lineage does not collect metadata directly from other GCS. However, if these services generate lineage for BigQuery assets, GCP captures the lineage, and Collibra Data Lineage ingests it. As a result, any indirect lineage created by these services is reflected in the technical lineage for BigQuery assets.Note The column-level lineage generated in Collibra Data Lineage is subject to the limitations of the data lineage feature in GCP. For details, go to Limitations in the About data lineage topic of the Google Cloud documentation.
Lineage extraction mechanism
Collibra Data Lineage retrieves lineage metadata via the Google Data Lineage API to provide visibility into BigQuery and GCS data flows:
- Technical lineage for GCP can start from GCS or BigQuery, but can end only in BigQuery. Lineage flows that end in a GCS resource are not captured.
- You can choose to create table-level lineage or column-level lineage for GCP when you synchronize the Technical Lineage for GCP capability. For column-level lineage, you can also select the lineage flow:
-
Legacy (Export API)exports the lineage to a GCS bucket by using theexportMetadataAPI. -
Streaming APIretrieves it directly from the Data Lineage API.
Important Google retires the Export API on November 13, 2026. The Legacy (Export API) lineage flow uses this API to retrieve column-level lineage. Table-level lineage is not affected. To keep creating column-level lineage, switch to the Streaming API lineage flow.
To switch the lineage flow, go to Switch to the Streaming API lineage flow.
For more information, go to Switch GCP column lineage to the Streaming API lineage flow in Collibra Support. -
- Transformations are ingested by calling the GCP Process and subsequently the GCP Jobs. Therefore, to ingest transformation details, the Service Account user defined in the Edge connection requires,
- At minimum, the
bigquery.jobs.getpermission - Optionally, the
bigquery.adminrole, which lets the capability ingest the details of all the jobs in the project
- At minimum, the
Project ingestion
Collibra Data Lineage uses the BigQuery project root node type setting to determine how it represents BigQuery projects in technical lineage. To select the setting, go to Synchronize GCP lineage.
- Database (recommended): This is the default. Collibra Data Lineage ingests each BigQuery project as a Database asset with the path (System) > Database > Schema > Table > Column, whether or not the project already exists in Data Catalog. This is consistent with other lineage sources, such as SQL, Power BI, and Looker, so their lineage can merge with lineage from this capability.
- GCP Project (legacy): Collibra Data Lineage ingests each BigQuery project as a GCP Project asset with the path (Domain Name) > GCP Project > Schema > Table > Column. Select this option only if existing lineage for these projects already uses GCP Project as the top-level node.
GCS items in the same project use the same root node type as the BigQuery project.
If two Technical Lineage for GCP capabilities analyze the same project but use different settings, one of the synchronizations fails.
Stitching behavior
Collibra Data Lineage determines the stitching domain based on existing assets for the project in Data Catalog, using the following priority:
- The domain of the existing Database asset.
- The domain of the existing GCP Project asset.
- If neither asset exists, the domain of the System.
Data objects in the technical lineage graph are automatically stitched to corresponding assets when the full name matches.
Stitching works for the column-level lineage, regardless of whether you integrated Knowledge Catalog (formerly known as Google Dataplex Catalog) or registered Google BigQuery databases by using the BigQuery JDBC connector.
Stitching is supported for both Database and GCP Project assets when the full name matches the corresponding asset in Data Catalog.
Stitching is not supported for GCS assets. GCS items displayed in the technical lineage graph aren’t linked to corresponding GCS assets in Data Catalog.
Differences between technical lineage for GCP and Google BigQuery
You can create technical lineage for Google BigQuery by using a JDBC connection or for GCP by using a GCP connection. Consider the following differences to determine which data source and connection type to use.
| Feature | Support in technical lineage for GCP (column-level lineage) | Support in technical lineage for GCP (table-level lineage) | Support in technical lineage for Google BigQuery |
|---|---|---|---|
| SQL transformation code | Yes | No | Yes |
| Executed SQL in stored procedures | No (table-level only) | Yes | No |
| Ingest lineage from... |
BigQuery and other Google Cloud services supported by the data lineage feature in GCP |
BigQuery and other Google Cloud services supported by the data lineage feature in GCP | BigQuery |
| BigQuery external tables | Yes | Yes | Yes |
| Stitching | Yes | No | Yes |