Enable data quality on an object store connection

Note This feature is available as a preview. For more information about previews, go to Previews at Collibra.

The archive break records feature is not supported for file-based jobs at this time.

Data Quality & Observability (cloud) can run data quality jobs directly against files and objects in a cloud object store, without staging the data in a database. Unlike database data sources, object stores do not use JDBC connections. Data Quality & Observability (cloud) reads files directly through your existing Edge connection to your object store.

Data Quality & Observability (cloud) supports:

Item Description

Object stores

  • Amazon S3
  • Azure Blob Storage
  • Azure Data Lake Storage
  • Google Cloud Storage

File formats

  • Avro
  • CSV
  • Delta
  • JSON
  • ORC
  • Parquet

Execution

Spark-based Pullup processing only. Pushdown processing is not available. Collibra reads file metadata, detects the file type, and discovers the schema.

Prerequisites

Steps

Add the capability to your Edge site so Collibra can discover and read files from your object store connection.

  1. On the main toolbar, click Products icon > Cogwheel icon Settings.

    The Settings page opens.

  2. In the tab pane, click Edge.

    The Sites tab opens.

  3. In the table, click the name of your Edge site.

    The Edge site page opens.

  4. Click the Capabilities tab, then click Add Capability.

    The Add Capability dialog box appears.

  5. Select Data Quality Pullup Processing.
  6. In the Connection field, select the AWS connection you configured in the prerequisites.
  7. Click Add.

    The capability is added to the Edge site.

Note After you save the capability, Collibra traverses the object store to discover and cache file and folder metadata. Traversal can take 5–60 minutes depending on the number of files and folders. No data is ingested during traversal. Traversal then runs automatically once a day. To refresh manually or change the auto-refresh cadence, go to Data Quality > Connections > [your connection name]. To reduce traversal time, see Limit file discovery.

Limit file discovery

By default, Collibra discovers every file in every bucket or container that the connection credentials can access. To reduce traversal time and show only relevant files in the Monitoring Overview, add one or both of the following properties to your Edge connection.

Property Description Example
excluded_file_extensions

A comma-separated list of file extensions to exclude from discovery. Enter the extensions without the dot. Values are not case-sensitive. If you do not set this property, or leave it empty, Collibra discovers all files.

Collibra uses only the last extension of a file name. For example, the extension of data.backup.parquet is parquet. This property does not apply to hidden files, such as .gitignore, or to files without an extension, such as _SUCCESS or README.

 log,tmp,crc,bak,md5
container_list

A comma-separated list of the buckets or containers to discover. Spaces around names are ignored. If you do not set this property, Collibra discovers all buckets or containers that the credentials can access.

Use this property if the credentials cannot list all buckets or containers, but can access specific ones.

 my-data-bucket,my-analytics-bucket
  1. On the main toolbar, click Products icon > Cogwheel icon Settings.

    The Settings page opens.

  2. In the tab pane, click Edge.

    The Sites tab opens.

  3. From the Connections tab, click the name of your file or object store connection.
  4. Click Edit.
  5. In the Property section, add the property:
    1. Click Add Property.
    2. In the Type field, select Text.
    3. In the Value type field, select Plaintext.
    4. Enter the property name and value.
  6. Click Save.

    The property is added to the connection.

Note If you exclude a file that already has a job, Collibra does not delete the job. You can still search for the job, open it from its direct link, and run it.

What's next