Dive deeper into the details of a monitor

You can drill down into a monitor for a closer look at its details and adjust the tolerance levels of adaptive rules.

Example In the screenshot, the details of the column "column2-Row_count" are expanded to show its row count evolution over a period of time. With this information, you may decide to leave this adaptive rule in its current state, or click Advanced and change the tolerance level to instruct the behavioral model to enforce it differently.

screenshot of monitor details drawer

Configuring tolerance levels

Tolerance levels allow you to control how rigidly the adaptive levels of monitors are enforced. By default, the adaptive level for all monitors is set to neutral, but you can choose to enforce them more strictly or leniently.

Prerequisites

You need a global role with the Product Rights > Data Quality global permission or the Data Quality Editor or Data Quality Manager resource role with the Data Quality Job > Train Monitors resource permission.

Steps

  1. Open a Data Quality Job page.
  2. Click Monitors.
  3. In the monitor details table, go to the monitor for which you want to view the details.
  4. On the far right side of the monitor row, click More icon, and then click Open.
    5. The Details drawer appears.
  5. Click Console icon Advanced.
    6. The tolerance options appear.
  6. Select a tolerance option.
    7. Tolerance Description
    Strict

    Rigidly enforce and flag any data points that fall outside what Data Quality & Observability's behavioral model learns to be normal. Any values outside the strict boundaries will be flagged as potential data quality issues.
    Neutral Automatically enforce and flag any data points that fall outside what Data Quality & Observability's behavioral model learns to be normal. Any values outside the neutral boundaries will be flagged as potential data quality issues.
    Lenient

    Relax the enforcement and flagging when data points fall outside of the range that Data Quality & Observability's behavioral model learns to be normal. Any values outside the lenient boundaries will be flagged as potential data quality issues.

    Tip This can help avoid excessive alerts when Data Quality & Observability is in the learning phase of a new Data Quality Job.

Tolerance level calculations

The tolerance level calculations vary depending on the specific type of data quality check being performed. The following tables show the check type, the thresholds of the three tolerance levels in relation to the check type, and the areas each check type impacts.

Uniqueness (standard deviation)

Tolerance level Threshold Impacts
Strict

Uniqueness standard deviation threshold (ABS): 2.1

Uniqueness standard deviation min/max threshold (z-score for min/max drift): 3.1
  • Uniqueness
  • Mean (standard deviation)
Neutral

Uniqueness standard deviation threshold (ABS threshold): 2.5

Uniqueness standard deviation min/max threshold (z-score for min/max drift): 3.4
Lenient

Uniqueness standard deviation threshold (ABS threshold): 3.9

Uniqueness standard deviation min/max threshold (z-score for min/max drift): 6.1

Uniqueness (percent change)

Tolerance level Threshold Impacts
Strict Uniqueness percent change threshold: 11%
  • Uniqueness
  • Type (percent mixed types)
Neutral Uniqueness percent change threshold: 14%
Lenient Uniqueness percent change threshold: 45%

Min/max (standard deviation)

Tolerance level Threshold Impacts
Strict Min/max standard deviation threshold (z-score for min/max drift): 3.1
  • Row count topN/bottomN (standard deviation/min/max)
  • Minimum/maximum (min/max)
  • Mean (standard deviation)
Neutral Min/max standard deviation threshold (z-score for min/max drift): 3.4
Lenient Min/max standard deviation threshold (z-score for min/max drift): 6.1

Min/max (percent change)

Tolerance level Threshold Impacts
Strict Min/max percent change threshold: 12%
  • Row count topN/bottomN (standard deviation/min/max)
  • Minimum/maximum (min/max)
  • Type (percent mixed types)
Neutral Min/max percent change threshold: 18%
Lenient Min/max percent change threshold: 65%

Standard deviation threshold (deviation from the baseline)

Tolerance level Threshold Impacts
Strict Standard deviation threshold: 2.3
  • Mean (standard deviation)
  • Percent null/empty (for lenient only)
Neutral Standard deviation threshold: 2.8
Lenient Standard deviation threshold (null, empty): 3.5

Percent change threshold

Tolerance level Threshold Impacts
Strict Percent change threshold: 3%
  • Percent null/empty
  • Type (percent mixed types)
Neutral Percent change threshold: 8%
Lenient Percent change threshold: 40%

What's next?