Synchronize inbound metadata via the Google Knowledge Catalog integration

Synchronizing via the Google Knowledge Catalog (formerly Dataplex Universal Catalog) integration is the process of integrating metadata from the Knowledge Catalog projects and making the data available in Collibra Platform. You can either synchronize inbound metadata manually or automate the process by adding a synchronization schedule.

The Knowledge Catalog integration configuration page has two tabs for inbound synchronization:

Each tab has its own Synchronize button, but both trigger the same inbound synchronization job. Synchronizing or scheduling from either tab synchronizes both Metadata Inbound and Data Products. They currently cannot be run or scheduled independently.

Important  The Knowledge Catalog integration ingests metadata only; it does not create technical lineage. To create technical lineage for your GCP assets, including Knowledge Catalog assets, go to the technical lineage for GCP documentation.

Prerequisites

In your Collibra environment

In your GCP environment

Steps

  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. Click Integrations.
    The Integrations page opens.
  3. Click the Integration configuration tab.
  4. In the Integration column, click the link for your Knowledge Catalog integration.
    The integration settings page opens.
  5. Select a tab to configure the relevant integration settings:
  6. Click Save.
  7. Click Synchronize.
    A notification indicates that the synchronization has started.
  1. On the main toolbar, click Products icon → Catalog.
    The Catalog homepage opens.
  2. Click Integrations.
    The Integrations page opens.
  3. Click the Integration configuration tab.
  4. In the Integration column, click the link for your Knowledge Catalog integration.
    The integration settings page opens.
  5. Select a tab to configure the relevant integration settings:
  6. Click Save.
  7. Click the Add synchronization schedule icon.
  8. Enter the required information and click Save:
    FieldDescription
    RepeatThe interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
    Cron

    The Quartz Cron expression that determines when the synchronization takes place.

    This field is only visible if you select Cron expression in the Repeat field.

    Every

    The day on which you want to synchronize, for example, Sunday.

    This field is only visible if you select Weekly in the Repeat field.

    Every first

    The day of the month on which you want to synchronize, for example, Tuesday.

    This field is only visible if you select Monthly in the Repeat field.

    At

    The time at which you want to synchronize automatically, for example, 14:00.

    • You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45.
    • This field is only visible if you select Daily, Weekly, or Monthly in the Repeat field.
    Time zoneThe time zone for the schedule.

Metadata inbound

Field Action Required
Data source type In Data source type, select a data source to configure the relevant integration settings: BigQuery or CloudSQL. Yes
Ingestion type In Ingestion type, select Knowledge Catalog ingestion. This integrates the Knowledge Catalog Entries and Aspects. Yes
System

In System, select the System asset in which you want to add the Knowledge Catalog assets.

 Yes
Updated: <timestamp>

Click Updated: <timestamp> next to Metadata Inbound, where timestamp indicates the last time when the data was loaded from Google Knowledge Catalog.
The Project IDs are loaded to the dropdown list of the Project Id field. This can take some time.

A scheduled refresh job is also automatically created when you complete the capability configuration. The scheduled refresh job appears on the Jobs dashboard of your Edge site with the same name as your metadata capability. Unlike the metadata synchronization job, it appears only on the Jobs dashboard and not in the Activities list. If two jobs with the same name appear on the Jobs dashboard, one is the scheduled refresh job and the other is the metadata synchronization job. By default, the scheduled refresh job runs daily at 23:00 server time.

 No
Default Community

This field is only visible if you select CloudSQL as the Data source type.

In Default Community, select a Collibra community to ingest the metadata. Subdomains per schema and database will be automatically created in this community.

 Yes
Database Type

This field is only visible if you select CloudSQL as the Data source type.

Select a database type from which you want to ingest Cloud SQL metadata:

  • Postgres
  • SQL Server
  • MySQL
Yes
Project IDs

To add a Project ID where Knowledge Catalog is enabled, click Add Project Id. You can add multiple Project IDs. The capability will search in these projects.

The following rules apply when you add Project IDs:
  • If you do not add Project IDs here but entered a value in the Project IDs (Deprecated) field in the Knowledge Catalog capability, the capability will search in the projects that you entered in the capability.
  • If you do not add Project IDs here and left the Project IDs (Deprecated) field empty in the Knowledge Catalog capability, the capability will search in the projects that you entered in the Service Account / Workload Identity Federation (WIF) field in the GCP connection. This applies only when the connection type is set to Service Account.
  • Do not add Project IDs here and also enter a value in the Project IDs (Deprecated) field in the Knowledge Catalog capability. This will cause the synchronization to end with an error.
No
GCP locations

Select the GCP locations you want to integrate.

  • If you select locations, the integration ingests Knowledge Catalog assets only from the specified locations. This is the recommended method.
  • If the location is added in Knowledge Catalog but is not visible in the list, you can use this field to add the location for integration. Type the name of the location and press Enter.
  • If you do not select any locations, the integration will search all locations where Knowledge Catalog assets are available. This can potentially cause increased synchronization times and permission errors.

The Knowledge Catalog integration allows for both single-region and multi-region locations. For more information, go to Knowledge Catalog locations in Google Cloud documentation.

 No
Domain include mappings

In Domain Include Mappings, specify the entries in Knowledge Catalog that you want to integrate and the Collibra domains where they need to be added. This is how it works:

  • If no include mappings are defined, we ingest all assets into the same domain as the System asset.
  • If there is no explicit domain mapping for a schema, we use the domain specified for the database.
  • A match with a database has priority over a match with a schema.

To limit the scope of metadata ingestion to specific domains in Collibra, add a domain include mapping:

  1. Click Add Domain Include Mappings.
  2. In Path, add the path to the entries in Knowledge Catalog for which you want to integrate the metadata.
    Tip 

    Use the following pattern: project name > location name > entryGroup name > parentEntry name > childEntry name. In the context of BigQuery, the parentEntry would be a BigQuery dataset name and childEntry would be a BigQuery table name.
    You can use the question mark (?) and asterisk (*) wildcards. To include all entries within a defined scope, use the asterisk (*) wildcard to account for a string of characters. To define a more granular scope, use the question mark (?) wildcard to account for single-character variations.
    If an entry matches multiple lines, the most detailed match is taken into account.

    Example 

    If you use the following syntax, for example, projectA > europe-west1 > @bigquery, the integration ingests all parentEntry and childEntry objects within the BigQuery entryGroup of your ProjectA project in Knowledge Catalog, limited to the europe-west1 location.

    If you want to use wildcards:

    • project? > europe-west1 > @bigquery returns entries from projects with a single-character variation in the project name, such as ProjectA, ProjectB, and so on.
    • projectB > * > @bigquery returns entries from all BigQuery datasets within ProjectB and across all locations.
    • projectB > * > @cloudsql
    • Other examples:
      • * > * > * > datasetX > tableY
      • projectA > europe-west1 > * > datasetA
  3. In Domain, select the Collibra domain in which you want to integrate the metadata.
 No
Domain exclude mappings

Optionally, in Domain Exclude Mappings, specify the path to entries in Knowledge Catalog that you don't want to integrate.

Note The exclude mapping has priority over the include mapping.

To exclude specific metadata from being ingested into Collibra, add a domain exclude mapping:

  1. Click Add Domain Exclude Mappings.
  2. In the field, add the path to entries that you want to exclude.

    Tip You can use the question mark (?) and asterisk (*) wildcards. To exclude all entries within a defined scope, use the asterisk (*) wildcard to account for a string of characters. To limit the scope to a more granular filter, use the question mark (?) wildcard to account for single-character variations.

    Example If you want to use wildcards:
    • projectA > * > @bigquery excludes all BigQuery datasets within ProjectA and across all locations.
    • projectA > europe-west4 > dataset_v?, this exclusion applies at a more granular level. It limits the ingestion to datasets with a single-character variation within projectA and to the europe-west4 location only.
    • projectC > * > @cloudsql

 No
Domain mapping version

In Domain mapping version, select the version of the domain mapping logic to apply:

  • V0: Assets that already exist in Collibra keep their current domain, even if the mapping points to a different one.
  • V1: The domain defined in the mapping takes priority over the existing database or schema domain, so matching assets are moved to the mapped domain.
 No
Columns ingestion mode

In Columns ingestion mode, define how the ingestion must handle nested fields. The available options are:

  • Ingest only parent columns:
    If you select this option, only the highest level fields are ingested as assets in Collibra. The hierarchy is shown via the View Array and View Struct links in the Technical Data Type column of these assets.

    Image of ingested parent only

  • Ingest parent and nested columns:
    If you select this option, Columns assets will be created for all fields. The parent assets also show the hierarchy via the View Array and View Struct links in the Technical Data Type column of these assets.

    Image of ingested parent and nested columns

  • Flatten columns structure:
    If you select this option, only the lowest level fields are ingested as assets.

    Image of ingested columns only

 Yes
Aspect mappings

Aspects in Knowledge Catalog that refer to columns are integrated as Column assets in Collibra during a Knowledge Catalog integration. Optionally, in this field, you can specify additional aspects in Knowledge Catalog that you want to integrate. Some aspects contain data derived from the underlying data, such as data-profile and data-rules. To integrate these, the service account needs the dataplex.entries.getData permission. For more information ,go to Create a Google Cloud Platform connection.

Aspect mapping is supported for Schema, Table, Database View, and Column assets, including partition columns. To map an aspect, select the Knowledge Catalog aspect in Aspect field and corresponding Collibra attribute in the Attribute field. You can optionally scope an aspect mapping to a specific asset type in the Asset field.

Important 

If you use this feature, make sure to add all required characteristics to the asset type assignments. Also note that any entries with spaces in the aspect name are skipped.

To add an aspect mapping:

  1. Click Add Aspect mapping.
  2. Optionally, in the Asset field, select the asset type to which you want to apply the mapping: All asset types, Schema, Table, Database View, or Column. If you don't specify an asset type, the mapping applies to all applicable asset types by default.
  3. In Aspect field, start typing to search the list of available aspect paths from Knowledge Catalog, and then select the path you want to integrate. The suggested paths follow this pattern: location.aspect-type-id>fieldPath, where aspect-type-id is a case-sensitive aspect type ID and field-path is a case-sensitive JSON path to the particular field of the aspect.
    For example:
    europe-west4.custom-aspect>name

    Dataplex aspect mapping example

  4. In the Attribute field, select the attribute in which you want to see the value.

If you have a table in Knowledge Catalog that is partitioned by date, you can create a custom attribute named Partition interval and specify bigquery-table > partitioning.interval → Partition interval as shown in the following image. After synchronization, the partition interval values such as Day, Hour, and so on are added to the Partition interval attribute of the corresponding Column asset in Collibra.

Note If an aspect is removed in Knowledge Catalog, the corresponding attribute in Data Catalog is removed after synchronization only when the asset’s max cardinality is set to one. If the maximum cardinality is greater than one, the attribute is not removed.
 No

Data Products (In Preview)

Field Action Required
Include Data Products

Optionally, set Include Data Products to Yes to also ingest GCP Data Product and Data Product Port assets.

This option is set to No by default.

 No
Default Domain In Default Domain, select the Collibra domain where all GCP Data Product and Data Product Port assets will be created. This field is required when Include Data Products is set to Yes. Yes

What's next

The synchronization job synchronizes the Knowledge Catalog data.

After the synchronization: