GCP lineage integration preflight checks

To ensure successful metadata ingestion and lineage generation, complete the following preflight checks.

In your GCP environment

Collibra Data Lineage harvests technical lineage for Google Cloud Platform (GCP) by leveraging the Data Lineage API in GCP. Be sure to review the supported transformation details to determine whether to create technical lineage for GCP or Google BigQuery.

Ensure that your GCP service account used to create a GCP connection has the following roles and permissions depending on your chosen lineage granularity. You can choose to create table-level lineage or column-level lineage for GCP when yousynchronize the capability.

  • To create table-level lineage: 
    • Enable the Data Lineage API in GCP for the projects that you want to harvest lineage from.
      For more information, go to Data Lineage API in Google Cloud documentation.
    • The Data Lineage Viewer role.
    • The BigQuery Admin role if you want Collibra Data Lineage to collect lineage from stored procedures that you created and those created by other GCP users.
    • The bigquery.jobs.get permission.
      For more information, go to IAM basic and predefined roles reference in the Google Cloud documentation.
  • To create column-level lineage:
    • Enable the exportMetadata API, which is in Preview. To enable the API , contact Google by following the steps in the article How to Request Access to Google Column Level Lineage (preview) export API.
    • Create a GCS bucket to store the exported lineage metadata. Ensure you have the following permissions on the GCS bucket:
      • The storage.objects.create permission in the Storage Object Creator (roles/storage.objectCreator) role.
      • The storage.objects.list and storage.objects.get permissions in the Storage Object Viewer (roles/storage.objectViewer) role.
    • Ensure you have the following permissions for lineage export: 
      • The datacatalog.entries.exportAll permission in the Data Catalog admin role (roles/datacatalog.admin) role.
        Note Google currently classifies this specific permission as Testing. Permissions at this level may be subject to change or deprecation.
      • Alternatively, grant roles/datacatalog.metadataExporter to your Service Account or the User.
      • The bigquery.jobs.get permission to get SQL transformation code.
        For more information, go to IAM basic and predefined roles reference in the Google Cloud documentation.
  • To synchronize technical lineage for GCP, the following permissions are required depending on the options you select:
    • You can add Project IDs that you want to harvest lineage from. To make Project IDs available for selection, ensure that the service account has the resourcemanager.projects.get permission for the target GCP projects.
      If the service account does not have this permission, you can enter the Project IDs manually on the Synchronization configuration page.
    • You can choose how transformation code is retrieved from BigQuery jobs by setting the SQL code extraction method field to use API calls or build batch queries to INFORMATION_SCHEMA. If you choose to build batch queries to INFORMATION_SCHEMA, the following permissions are required:
      • bigquery.jobs.create
      • bigquery.jobs.listAll

In your CPSH environment

Lineage enablement

  • Technical lineage via Edge is enabled in your Collibra environment.
  • You are using Collibra Platform Self-Hosted 2024.07 or newer.
  • Your CPSH environment is configured with the latest Collibra UI.

Edge

  • You created and installed an Edge site.
    Important If you're using a Collibra Cloud site, go the Collibra Cloud site documentation to check if your data source is supported.
  • The Edge site status must be Healthy.
  • You've registered the data source via Edge.
  • You've integrated Knowledge Catalog or registered Google BigQuery databases by using the BigQuery JDBC connector. For details, go to About working with Google Cloud Platform (GCP).

Network and proxy configuration

  • Edge can connect to all Collibra Data Lineage service instances in your geographic location.
  • Optionally, you've connected to a proxy server.
  • Optionally, use a custom certificate to allow the Edge capability to connect to your data source. In this case, you've saved the certificate as "ca.pem" in the same directory as the Edge site installer. If you've saved the certificate in another directory, use the --ca argument in the Edge site installation command.

CPSH permissions

You can connect to Collibra Data Lineage by using the basic or OAuth authentication method. The following permissions are required only if you use the basic authentication method. 

To connect to Collibra Data Lineage service instances via OAuth authentication:

To add an Edge capability:

To synchronize technical lineage: