The Lineage flow field on the GCP lineage synchronization page determines how Collibra Data Lineage retrieves column-level lineage.
If you create only table-level lineage, you don't need to switch to the Streaming API (Preview) lineage flow, because both options retrieve table-level lineage in the same way.
The Streaming API (Preview) lineage flow retrieves column-level lineage directly from the Data Lineage API. It doesn't use the exportMetadata API or require a GCS bucket. You also don't need to request access from Google.
The Streaming API (Preview) lineage flow doesn't retrieve the following column-level lineage, which the Legacy (Export API) lineage flow retrieves:
- Lineage into tables that were deleted from BigQuery. Lineage from a deleted table into a table that still exists is retrieved.
- Lineage into temporary tables, such as tables created by BigQuery scripts or third-party tools.
- Lineage into tables in projects that aren't added in the Project IDs field.
If you switch and keep the same source ID, the first synchronization removes this column-level lineage from Collibra. To keep your existing column-level lineage while you switch, we recommend that you use a new source ID. Lineage from the Legacy (Export API) lineage flow remains available as a separate data source until you remove that source. For details, go to About source IDs and the data source lifecycle.
Prerequisites
- The service account has the roles and permissions for the Streaming API (Preview) lineage flow. For details, go to GCP lineage integration preflight checks.
- You have a global role with the Product Rights > System administration global permission.
- You have a global role that has the Manage connections and capabilities global permission, for example, Edge integration engineer.
Steps
You can switch to the Streaming API (Preview) lineage flow in either of the following ways.
Option A: Reuse the existing capability and connection
- Edit the Technical Lineage for GCP capability that creates column-level lineage. In the Source ID field, enter a new source ID.
- On the synchronization page, select Streaming API (Preview) in the Lineage flow field, and then synchronize the capability.
- Verify that the synchronization succeeds and that column-level lineage appears in the technical lineage viewer.
- After 30 days, remove the old data source by using the Ignore sources admin option, and then synchronize.
Google keeps lineage for 30 days. After this period, the old data source no longer contains lineage that the Legacy (Export API) lineage flow can retrieve.
Option B: Create a new connection and capability
- Create a GCP connection.
- Add a Technical Lineage for GCP capability with a new source ID.
- On the synchronization page, select Streaming API (Preview) in the Lineage flow field, and then synchronize the capability.
- Verify that the synchronization succeeds and that column-level lineage appears in the technical lineage viewer.
- After 30 days, delete the old capability.
- Remove the old data source by using the Ignore sources admin option, and then synchronize.