Connect to a Data Quality & Observability Classic source

Important This feature isn't available in the latest UI.

Because the DQ Connector is an Edge capability, you must be able to ingest data via Edge. For information about enabling and configuring Edge, see the Edge Configuration guide.

Create a Data Quality & Observability Classic Edge site

Create an Edge site with the following properties:

Field	Description
Name	The name of the Edge site, for example Collibra-DQ-Edge. Do not use spaces or special characters. This field is mandatory and the name must be globally unique.
Description	The description of the Edge site. We recommend to put at least basic location information of the Edge site. This field is mandatory.

Field

Description

Name

The name of the Edge site, for example Collibra-DQ-Edge. Do not use spaces or special characters.

This field is mandatory and the name must be globally unique.

Description

The description of the Edge site. We recommend to put at least basic location information of the Edge site.

This field is mandatory.

Install the Data Quality & Observability Classic Edge site

Follow the instructions for your environment to Install an Edge site.

Note This process automatically creates an Edge user, which you use later in the setup process.

Connect to your Data Quality & Observability Classic source

Create a connection for each CollibraData Quality & Observability Classic data source you want to synchronize.

Section	Property	Value
Connection settings	Name	The same name as the Collibra Data Quality & Observability Classic connection name.
	Description	The description of the JDBC connection. This field is also visible when you register content.
	Connection provider	The connection provider, which determines the available connection parameters. Same as Collibra Data Quality & Observability Classic.
Connection parameters Example for Username / Password JDBC drive	Username	The same username as the Collibra Data Quality & Observability Classic connection username.
	Password	The same password as the Collibra Data Quality & Observability Classic connection password.
	Driver class name	The same driver name as the Collibra Data Quality & Observability Classic connection driver name.
	Driver Jar	The same driver JAR file as from Collibra Data Quality & Observability Classic.
	Connection string	The same URL as the Collibra Data Quality & Observability Classic connection URL.

Add ingestion capabilities to your Data Quality & Observability Classic connection

You must add a Catalog JDBC ingestion Edge capability template for each connection you have created to extract and process data for your data source.

Field Description Required

Capability

This section contains general information about the capability.

Name

The name of the capability.

Yes

Description

The description of the capability.

Capability template

The capability template. The value that you select in this field determines which sections appear on the page.

Select the following capability:

Catalog JDBC ingestion

Yes

Connection

This section contains information to connect to the data source.

JDBC connection

The connection to the data source.

Yes

JDBC data source type (Deprecated)

Deprecated field. The field was used to indicate the type of the data source. You no longer need to change this field. The required value is automatically identified.

Note The automatically identified value is not shown in this page.

Yes

Supports schemas

A text field where you have to enter True to enable database registration of data sources that have no schema. If the data source has schemas, you can ignore this field.

Tip If the data source does not have a schema, Data Catalog creates a Schema asset with the same name as the full name of the database.

Other Settings

Others

This section can contain additional capability properties.

Click Add propertyAdd Other Settings to add a property.

Show possible properties

Name Description Type Encryption Default value

tags-strategy

To increase the performance of the Snowflake metadata synchronization if you register source tags, we can read the Snowflake source tags from the SNOWFLAKE.ACCOUNT_USAGE schema. To do so, change the value of this property to the value SINGLE_CALL.

Note This method requires you to have the SELECT permission on the SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES table.

Text

Not encrypted (plain text)

CALL_PER_TABLE

By default, we read the Snowflake source tags from the <database_name>.INFORMATION_SCHEMA.tag_ references. This method works with the minimum required permissions to perform the metadata scan.

You can also define the properties the following jobs that can run in view of a database synchronization.

database-list-with-metadata: This job runs when you register a data source and the Database field needs to be filled with available databases.
schema-list: This jobs runs in the Configuration tab of a Database asset to show the schemas in the database.
ingest-schema: This job runs when you click the Synchronize button.

Name	Description	Type	Encryption	Default value
Warning The following properties can have a significant impact on your Edge site. Only add or update them together with Collibra Support.
database-list-with-metadata-garbage-collector	The garbage collector that is used by this job in the capability. For information about other possibilities, see AZUL documentation.	Text	Not encrypted (plain text)	-XX:+UseParallelGC
database-list-with-metadata-requests-cpu	database-list-with-metadata-requests-cpu The minimum amount of CPU computing power requested by by this job in the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	100
database-list-with-metadata-limits-cpu	database-list-with-metadata-limits-cpu The maximum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	950
database-list-with-metadata-requests-memory	The minimum amount of memory requested by this job in the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: database-list-with-metadata-limits-memory and database-list-with-metadata-jvm-max-memory.	Text	Not encrypted (plain text)	128
database-list-with-metadata-limits-memory	The maximum amount of memory requested by this job in the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: database-list-with-metadata-requests-memory and database-list-with-metadata-jvm-max-memory.	Text	Not encrypted (plain text)	256
database-list-with-metadata-jvm-max-memory	The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job. The amount is expressed in mebibytes (Mi). Warning Make sure this amount is lower than the database-list-with-metadata-limits-memory amount. Important If you add this property, you also need to add the properties: database-list-with-metadata-requests-memory and database-list-with-metadata-limits-memory.	Text	Not encrypted (plain text)	256

schema-list-garbage-collector	The garbage collector that is used by this job in the capability. For information about other possibilities, see AZUL documentation.	Text	Not encrypted (plain text)	-XX:+UseParallelGC
schema-list-requests-cpu	The minimum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	100
schema-list-limits-cpu	The maximum amount of CPU computing power requested by the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	950
schema-list-requests-memory	The minimum amount of memory requested by this job in the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: schema-list-limits-memory and schema-list-jvm-max-memory.	Text	Not encrypted (plain text)	128
schema-list-limits-memory	The maximum amount of memory requested by this job in the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: schema-list-requests-memory and schema-list-jvm-max-memory.	Text	Not encrypted (plain text)	256
schema-list-jvm-max-memory	The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job. The amount is expressed in mebibytes (Mi). Warning Make sure this amount is lower than the schema-list-limits-memory amount. Important If you add this property, you also need to add the properties: schema-list-requests-memory and schema-list-limits-memory.	Text	Not encrypted (plain text)	256

ingest-schema-garbage-collector	The garbage collector that is used by this job in the capability. For information about other possibilities, see AZUL documentation.	Text	Not encrypted (plain text)	-XX:+UseParallelGC
ingest-schema-requests-cpu	The minimum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	100
ingest-schema-limits-cpu	The maximum amount of CPU computing power requested by this job in the capability. The amount is expressed in milliCPU.	Text	Not encrypted (plain text)	1500
ingest-schema-requests-memory	The minimum amount of memory requested by this job in the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: ingest-schema-limits-memory and ingest-schema-jvm-max-memory.	Text	Not encrypted (plain text)	128
ingest-schema-limits-memory	The maximum amount of memory requested by the capability. The amount is expressed in mebibytes (Mi). Important If you add this property, you also need to add the properties: ingest-schema-requests-memory and ingest-schema-jvm-max-memory.	Text	Not encrypted (plain text)	2048
ingest-schema-jvm-max-memory	The maximum amount of memory that can be used by the Java virtual machine (jvm) for this job. The amount is expressed in mebibytes (Mi). Warning Make sure this amount is lower than the ingest-schema-limits-memory amount. Important If you add this property, you also need to add the properties: ingest-schema-requests-memory and ingest-schema-limits-memory.	Text	Not encrypted (plain text)	2048

http-connect-timeout-seconds	The maximum amount of time allowed to create a connection with Collibra. The value must be set to 30 or higher.	Text	Not encrypted (plain text)	30
http-read-timeout-seconds	The maximum amount of time allowed to wait for a response before closing the connection. The value must be set to 300 or higher.	Text	Not encrypted (plain text)	300

Note No validation is performed on the values you add.

General

This section contains general information about logging.

Debug

An option to automatically send Edge infrastructure log files to Collibra Platform. By default, this option is set to false.

Note We highly recommend to only send Edge infrastructure log files to Collibra Platform when you have issues with Edge. If you set it to true, it will automatically revert to false after 24h.

For more information, go to logging.

Log level

An option to determine the verbosity level of Catalog connector log files. By default, this option is set to No logging.

Configure destinations for Data Quality & Observability Classic assets

Data Quality & Observability Classic rules, metrics and dimensions require their own domains in Data Catalog. If you don't have existing domains for data quality or wish to use new ones for the quality extraction purpose, create a domain for each type of data quality asset:

Rules: Rulebook Domain
Metrics: Business Asset Domain
Dimensions: Business Asset Domain

Assign permissions for Data Quality & Observability Classic domains

Edge must have the correct resource permissions to manage assets inside the dedicated Data Quality & Observability Classic domains. For each dedicated domain, assign the Technical Steward role to the Edge user.

Note The Edge user is automatically created when you install the Edge site.

Add Data Quality & Observability Classic characteristics to assets

To show Data Quality & Observability Classic statistics for your data source, assign the following characteristic types to the Table and Column asset types:

Asset type	Characteristic type
Table	governed by Governance Asset
Column	is governed by Data Quality Rule

Add a DQ Connector capability

The DQ Connector facilitates the communication with Collibra Data Quality & Observability Classic. Add a DQ Connector capability to your Data Quality & Observability Classic Edge site:

Field	Description	Required
Capability	This section contains general information about the capability.
Name	The name of the capability.	Yes
Description	The description of the capability.	No
Capability template	The capability template. The value that you select in this field determines which sections appear on the page. Select the following capability template to ingest Data Quality & Observability Classic user-defined rules, metrics, and dimensions into Collibra Data Catalog: `DQ Connector`	Yes
DQ	This section contains information about the Data Quality & Observability Classic connection.
Base URL	Your Data Quality & Observability Classic URL	Yes
Username	The Data Quality & Observability Classic username for this connection.	Yes
Password	The Data Quality & Observability Classic password for this connection.	Yes
Encryption options	Select the type of encryption to use. Default: To be encrypted by Edge management server.
Issuer of the JWT	If you have selected Encrypted with public key, enter your JWT issuer.	No
Collibra metadata model	This section contains information about where to ingest Data Quality & Observability Classic assets.
DQ Rules domain id	The UUID of the Rulebook Domain for the ingested Data Quality & Observability Classic rules.	Yes
DQ Metrics domain id	The UUID of the Rulebook Domain for the ingested Data Quality & Observability Classic metrics.	Yes
DQ Dimensions domain id	The UUID of the Governance Asset Domain for the ingested Data Quality & Observability Classic dimensions.	Yes
Default DQ Dimension name	The default Data Quality Dimension, for example Accuracy, Completeness, Consistency and so on. Default: Completeness.	Yes
DQ Metric classified by DQ Dimension relation type id	The UUID of the Data Quality Metric classified by / classifies Data Quality Dimension relation. If left unspecified, this relation will not be added.	No
Assets are imported in batches of this size	The batch size of the ingestion. Default: 5000.	Yes

Connect to a Data Quality & Observability Classic source

Create a Data Quality & Observability Classic Edge site

Install the Data Quality & Observability Classic Edge site

Connect to your Data Quality & Observability Classic source

Add ingestion capabilities to your Data Quality & Observability Classic connection

Configure destinations for Data Quality & Observability Classic assets

Assign permissions for Data Quality & Observability Classic domains

Add Data Quality & Observability Classic characteristics to assets

Add a DQ Connector capability

Next steps