Configuring archive break records

This topic shows you how to configure archive break records to send details of breaking records from jobs to your data source. Enabling this feature allows you to automatically store row-level data for failed quality checks directly in your database.

You can configure the storage location at the connection level to align with your organization's data architecture.

Note Archive break records is currently only available for Pushdown connections.

Prerequisites

To archive break records, you need the following permissions and configurations:

Requirement Details
Collibra permissions
  • To configure your connection, you have a global role with the Data Quality > Manage Data Sources global permission.
  • To view and download break records, you have Preview Rule Break Records resource permission.
Data source privileges for break records

Because exact permission models vary significantly across different data platforms, consult your organization's database administrator to provision the exact permissions.

The service account connecting to your data source requires the following generalized privileges:

On the source data (to monitor records):

  • Read access (SELECT) on the specific tables and views you intend to monitor.

On the destination schema (to archive break records):

  • Create access (CREATE TABLE and CREATE VIEW) in the designated archive schema.
  • Note You only need create access once per break records table, or you can create it manually with explicit update access. All monitors write to the same break records table, so create access can be safely revoked after initial setup. If the break records storage location changes, create access is required again to generate a new table.
  • Modify access (ALTER TABLE and ALTER VIEW) to update schema structures, such as adding new columns.
  • Drop access (DROP VIEW) to clean up or replace views as needed.
  • Read access (SELECT) on the newly created tables and views.
  • Write/edit access (INSERT, UPDATE, and DELETE) on those created tables to actively manage the archived records over time.
Connection configuration

You have an established connection.

Enabling archive break records

Important Ensure that the service account used for the connection has all the privileges listed above. If any privilege is missing, the system cannot archive break records.

  1. Open a site.
    1. On the main toolbar, click Products iconCogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the site overview, click the name of a site.
      The site page appears.
  2. Click the name of your connection in the connections list.
    The Edit connection dialog box appears.
  3. Click the Edit in the lower-right corner of the dialog box.
  4. Click + Add Property in the Property section.
  5. Enter __readOnly__ in the Name field.
  6. Enter false in the Value field.
  7. Click Save.
  8. image of a __readOnly__ property set to false
  9. On the main toolbar, click Products iconData Quality.
    Data Quality & Observability opens.
  10. Click Connections.
    The Connections page opens.
  11. Open the connection with the recently added property.
  12. Click the Archive break records switch to enable archive break records.
  13. Select a break record archival location. Options include:
    • Same location as job: The system creates break records in the job's schema. For example, If your job resides in a public schema, break records will be created in the same schema.
    • Specify a location: Select a database and schema from the drop-down menu.
    • Note The database and schema must already exist in your connection.

  14. Create and run a job on this connection.
  15. Create a rule that will return breaking records.
  16. Re-run your job.
    When breaking records occur, they will be archived to the location you specified in Step 11.
Tip In addition to direct break record archival to your database, you can preview and download break records in the Monitors tab of the Data Quality Job page. You can adjust the maximum number of break records included in the preview via the Break records preview limit setting on the Rules page in Settings.