Synchronize via the Google Dataplex Catalog ingestion

Important This feature is available only in the latest UI.

Synchronizing via the Google Dataplex Catalog ingestion is the process of integrating metadata from the Google Dataplex projects and making the data available in Collibra Data Intelligence Platform.

You can synchronize manually, or you can automate it by adding a synchronization schedule.

Before you begin

You have enabled the Cloud Resource Manager API in GCP.
You have given the Edge user the required permissions.
You have created a GCP connection.
You have added the Google Dataplex Catalog synchronization capability to the GCP connection.
You created a System asset in which you want to add the Google Dataplex assets, for example BigQuery.
Make sure you are on the latest UI, because the Dataplex Catalog ingestion is available only in the latest UI.

Requirements and permissions

You have a resource role with the Configure external system resource permission, for example, Owner.
You have a global role with the Catalog global permission, for example, Catalog Author.
You have a global role with the View Edge connections and capabilities global permission, for example, Edge integration engineer. For example, Edge integration engineer.

Steps

Manually synchronize Google Dataplex Catalog
Add a synchronization schedule

On the main toolbar, click , and then click Catalog.
The Catalog Home opens.
On the main toolbar, click .
The Create dialog box appears.
In the Register with Edge section of the Create dialog box, click Integration Configuration.
The Integration Configuration tab page opens.
In the Connection Name column, locate the GCP connection that you used when you added the Dataplex capability and click the capability link in the Capabilities column.
The Dataplex capability configuration page opens.
In the Synchronization Configuration section, click Add Configuration.
In Ingestion Type, select Dataplex Catalog ingestion.
This will integrate the Dataplex Catalog Entries and Aspects.
If you want to integrate the metadata from the projects, lakes, zones, tables, and columns, go to Dataplex ingestion.

Complete the fields as follows:

Field	Mandatory / Optional	Action
System	Mandatory	In System, select the System asset in which you want to add the Google Dataplex assets.
Updated: <timestamp>	Optional	Click Updated: <timestamp> next to Synchronization Configuration, where `timestamp` indicates the last time when the data was loaded from Google Dataplex. The Project IDs are loaded to the drop-down list of the Project Id fields that you can use in the following step. This can take some time.
Project ID	Optional	To add a Project ID where Dataplex is enabled, click Add Project Id. You can add multiple Project IDs. The capability will search in these projects. The following rules apply when you add Project IDs: If you do not add Project IDs here but entered a value in the Project IDs (Deprecated) field in the Google Dataplex Catalog synchronization capability, the capability will search in the projects that you entered in the capability. If you do not add Project IDs here and left the Project IDs (Deprecated) field empty in the Google Dataplex Catalog synchronization capability, the capability will search in the projects that you entered in the GCP Service Account field in the GCP connection. Do not add Project IDs here and also enter a value in the Project IDs (Deprecated) field in the Google Dataplex Catalog synchronization capability; otherwise, the synchronization will end with an error.
Dataplex location	Optional	To add a Dataplex location, click Add Dataplex Location. If a new location is added in Dataplex and not yet supported by the integration, you can use this field to add the location for integration. When you add a location in this field, the integration ingests Dataplex assets only from the specified location. For more information, go to Dataplex locations in Google Cloud documentation.
Domain Include Mappings	Optional	In Domain Include Mappings, specify which entries in Google Dataplex that you want to integrate and the Collibra domains where they need to be added. Note If no include mappings are defined, we ingest all assets into the same domain as the System asset. If there is no explicit domain mapping for a schema, we use the domain specified for the database. A match with a database has priority over a match with a schema. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the entries in Google Dataplex for which you want to integrate the metadata. Tip Use the following pattern: project > location > entryGroup > parentEntry > childEntry. You can use the ? and * wildcards. If an entry matches multiple lines, the most detailed match is taken into account. Example `* > * > * > dataset > table` `project > europe-west1 > * > dataset` `project > * > @bigquery` In Domain, select the Collibra domain in which you want to integrate the metadata.
Domain Exclude Mappings	Optional	Optionally, in Domain Exclude Mappings, specify the path to entries in Google Dataplex that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to entries that you want to exclude. Tip You can use the ? and * wildcards. For example:`project > * > @bigquery`.
Columns ingestion mode	Mandatory	In Columns ingestion mode, define how the ingestion must handle nested fields. The available options are: Ingest only parent columns: If you select this option, only the highest level fields are ingested as assets in Collibra. The hierarchy is shown via the View Array and View Struct links in the Technical Data Type column of these assets. Show an image Ingest parent and nested columns: If you select this option, Columns assets will be created for all fields. The parent assets also show the hierarchy via the View Array and View Struct links in the Technical Data Type column of these assets. Show an image Flatten columns structure: If you select this option, only the lowest level fields are ingested as assets. Show an image
Aspect Mappings	Optional	Aspects in Google Dataplex that refer to columns are integrated as Column assets in Collibra during a Dataplex Catalog ingestion. Optionally, in the Aspect Mappings field, specify which other aspects in Google Dataplex that you want to integrate. You do this by adding the mapping between the aspect in Google Dataplex and the Collibra attribute. Show details Important If you use this feature, make sure to add all required characteristics to the asset type assignments. Click Add Another Mapping. In Aspect Field, add the reference to the aspect field you want to integrate. Use the following pattern: `location.aspectName>fieldPath`. For example: europe-west4.aieh-custom-aspect>custom_field1 In Attribute, select the attribute in which you want to see the value.

Click Save.
Click Synchronize.
A notification indicates the synchronization has started.

On the main toolbar, click , and then click Catalog.
The Catalog Home opens.
On the main toolbar, click .
The Create dialog box appears.
In the Register with Edge section of the Create dialog box, click Integration Configuration.

The Integration Configuration tab page opens.
In the Connection Name column, locate the GCP connection that you used when you added the Dataplex capability and click the capability link in the Capabilities column.
The Dataplex capability configuration page opens.
In the Synchronization Configuration section, click Add Configuration.

Complete the fields as follows:

Field	Mandatory / Optional	Action
System	Mandatory	In System, select the System asset in which you want to add the Google Dataplex assets.
Updated: <timestamp>	Optional	Click Updated: <timestamp> next to Synchronization Configuration, where `timestamp` indicates the last time when the data was loaded from Google Dataplex. The Project IDs are loaded to the drop-down list of the Project Id fields that you can use in the following step. This can take some time.
Project ID	Optional	To add a Project ID where Dataplex is enabled, click Add Project Id. You can add multiple Project IDs. The capability will search in these projects. The following rules apply when you add Project IDs: If you do not add Project IDs here but entered a value in the Project IDs (Deprecated) field in the Google Dataplex Catalog synchronization capability, the capability will search in the projects that you entered in the capability. If you do not add Project IDs here and left the Project IDs (Deprecated) field empty in the Google Dataplex Catalog synchronization capability, the capability will search in the projects that you entered in the GCP Service Account field in the GCP connection. Do not add Project IDs here and also enter a value in the Project IDs (Deprecated) field in the Google Dataplex Catalog synchronization capability; otherwise, the synchronization will end with an error.
Dataplex location	Optional	To add a Dataplex location, click Add Dataplex Location. If a new location is added in Dataplex and not yet supported by the integration, you can use this field to add the location for integration. When you add a location in this field, the integration ingests Dataplex assets only from the specified location. For more information, go to Dataplex locations in Google Cloud documentation.
Domain Include Mappings	Optional	In Domain Include Mappings, specify which entries in Google Dataplex that you want to integrate and the Collibra domains where they need to be added. Note If no include mappings are defined, we ingest all assets into the same domain as the System asset. If there is no explicit domain mapping for a schema, we use the domain specified for the database. A match with a database has priority over a match with a schema. Show steps to add a domain include mapping Click Add Domain Include Mappings. In Path, add the path to the entries in Google Dataplex for which you want to integrate the metadata. Tip Use the following pattern: project > location > entryGroup > parentEntry > childEntry. You can use the ? and * wildcards. If an entry matches multiple lines, the most detailed match is taken into account. Example `* > * > * > dataset > table` `project > europe-west1 > * > dataset` `project > * > @bigquery` In Domain, select the Collibra domain in which you want to integrate the metadata.
Domain Exclude Mappings	Optional	Optionally, in Domain Exclude Mappings, specify the path to entries in Google Dataplex that you don't want to integrate. Note The exclude mapping has priority over the include mapping. Show steps to add a domain exclude mapping Click Add Domain Exclude Mappings. In the field, add the path to entries that you want to exclude. Tip You can use the ? and * wildcards. For example:`project > * > @bigquery`.
Columns ingestion mode	Mandatory	In Columns ingestion mode, define how the ingestion must handle nested fields. The available options are: Ingest only parent columns: If you select this option, only the highest level fields are ingested as assets in Collibra. The hierarchy is shown via the View Array and View Struct links in the Technical Data Type column of these assets. Show an image Ingest parent and nested columns: If you select this option, Columns assets will be created for all fields. The parent assets also show the hierarchy via the View Array and View Struct links in the Technical Data Type column of these assets. Show an image Flatten columns structure: If you select this option, only the lowest level fields are ingested as assets. Show an image
Aspect Mappings	Optional	Aspects in Google Dataplex that refer to columns are integrated as Column assets in Collibra during a Dataplex Catalog ingestion. Optionally, in the Aspect Mappings field, specify which other aspects in Google Dataplex that you want to integrate. You do this by adding the mapping between the aspect in Google Dataplex and the Collibra attribute. Show details Important If you use this feature, make sure to add all required characteristics to the asset type assignments. Click Add Another Mapping. In Aspect Field, add the reference to the aspect field you want to integrate. Use the following pattern: `location.aspectName>fieldPath`. For example: europe-west4.aieh-custom-aspect>custom_field1 In Attribute, select the attribute in which you want to see the value.

Click Save.
In the Synchronization Schedule section, click Add schedule.

Enter the required information and click Save:

Field	Description
Repeat	The interval when you want to synchronize automatically. The possible values are: Daily, Weekly, Monthly, and Cron expression.
Cron	The Quartz Cron expression that determines when the synchronization takes place. This field is only visible if you select `Cron expression` in the Repeat field.
Every	The day on which you want to synchronize, for example, Sunday. This field is only visible if you select `Weekly` in the Repeat field.
Every first	The day of the month on which you want to synchronize, for example, Tuesday. This field is only visible if you select `Monthly` in the Repeat field.
At	The time at which you want to synchronize automatically, for example, 14:00. You can only schedule on the hour. For example, you can add a synchronization schedule at 8:00, but not at 8:45. This field is only visible if you select `Daily`, `Weekly`, or `Monthly` in the Repeat field.
Time zone	The time zone for the schedule.

What's next?

The synchronization job synchronizes the Google Dataplex data.
After the synchronization:

You can view a summary of the results from the Activities list.
For information on the integrated data, go to Synchronized data via Google Dataplex Catalog ingestion.