Configuring archive break records for Pullup jobs

This topic shows you how to configure archive break records to send details of breaking records from Pullup jobs to your data source. Enabling this feature allows you to automatically store row-level data for failed quality checks directly in your database.

You can configure the storage location at the connection level to align with your organization's data architecture.

Prerequisites

To archive break records, you need the following permissions and configurations:

Requirement Details
Collibra permissions
  • To configure your connection, you have a global role with the Data Quality > Manage Data Sources global permission.
  • To view and download break records, you have Preview Rule Break Records resource permission.
Existing PostgreSQL connection and Data Quality Pullup Processing capability
  • You have an established connection to a PostgreSQL data source.
  • The Data Quality Pullup Processing capability enabled on your connection is already configured on the same Edge site.

    Important Each Data Quality Pullup Processing capability requires a separate PostgreSQL connection, even when multiple capabilities share the same physical database. Create one PostgreSQL connection per Data Quality Pullup Processing capability where you want to archive break records.
Data source privileges for break records

Because exact permission models vary significantly across different data platforms, consult your organization's database administrator to provision the exact permissions.

The service account connecting to your data source requires the following generalized privileges:

On the source data (to monitor records):

  • Read access (SELECT) on the specific tables and views you intend to monitor.

On the destination schema (to archive break records):

  • Create access (CREATE TABLE and CREATE VIEW) in the designated archive schema.
    • Note You only need create access once per break records table, or you can create it manually with explicit update access. All monitors write to the same break records table, so create access can be safely revoked after initial setup. If the break records storage location changes, create access is required again to generate a new table.
  • Modify access (ALTER TABLE and ALTER VIEW) to update schema structures, such as adding new columns.
  • Drop access (DROP VIEW) to clean up or replace views as needed.
  • Read access (SELECT) on the newly created tables and views.
  • Write/edit access (INSERT, UPDATE, and DELETE) on those created tables to actively manage the archived records over time.

Enabling archive break records

Important Ensure that the service account used for the connection has all the privileges listed above. If any privilege is missing, the system cannot archive break records.

  1. Open a site.
    1. On the main toolbar, click Products icon → Cogwheel icon Settings.
      The Settings page opens.
    2. In the tab pane, click Edge.
      The Sites tab opens and shows a table with an overview of your sites.
    3. In the site overview, click the name of a site.
      The site page appears.
  2. Click the name of your PostgreSQL connection in the connections list.
    The Edit connection dialog box appears.
  3. Click Edit in the lower-right corner of the dialog box.
  4. Click + Add Property in the Property section.
  5. Enter __readOnly__ in the Name field.
  6. Enter false in the Value field.
  7. Click Save.
    8. image of a __readOnly__ property set to false
  8. On the main toolbar, click Products icon → Cogwheel icon Settings.
    The Settings page opens.
  9. In the tab pane, click Edge.
    The Sites tab opens and shows a table with an overview of your sites.
  10. In the site overview, click the name of the site where the Pullup connection is configured.
    The site page appears.
  11. Click the Capabilities tab.
  12. Click the Data Quality Pullup Processing capability where you want to archive break records.
  13. Set Archive Break Records Connection to your PostgreSQL connection.
  14. Click Save.
  15. On the main toolbar, click Products icon → Data Quality.
    Data Quality & Observability opens.
  16. Click Connections.
    The Connections page opens.
  17. Open the connection with the recently added property.
  18. Click the Archive break records switch to enable archive break records.
  19. Select the destination database and schema from the drop-down menus, where break records will be written.

    Note The database and schema must already exist in your connection.

  20. Create and run a Pullup job on this connection.
  21. Create a rule that will return breaking records.
  22. Re-run your job.
    When breaking records occur, Collibra writes them to the PostgreSQL database you selected in step 18.
Tip In addition to direct break record archival to your database, you can preview and download break records in the Monitors tab of the Data Quality Job page. You can adjust the maximum number of break records included in the preview via the Break records preview limit setting on the Rules page in Settings.

Helpful resources