Working with the Change Detection modal

The Change Detection modal appears when you click the Actions actions button button. Select either Adaptive, Manual, or Suppress from the options below to see details about each alert type.

Adaptive Level Description
Strict

Rigidly enforce and alert of any data points that fall outside what Collibra DQ's behavioral model learns to be normal. Any values outside the strict boundaries will be flagged as potential data quality issues.

Neutral/System Detected Automatically enforce and alert any data points that fall outside what Collibra DQ's behavioral model learns to be normal. Any values outside the neutral boundaries will be flagged as potential data quality issues.
Lenient

Relax the enforcement and alerting when data points fall outside of the range that Collibra DQ's behavioral model learns to be normal. Any values outside the lenient boundaries will be flagged as potential data quality issues.

Tip This can help avoid excessive alerts when Collibra DQ is in the learning phase of a new dataset.

Pass Value on [Date] Select this option to accept the value from a given date into the rolling behavioral baseline.

Adaptive level calculations

The adaptive level calculations vary depending on the specific type of data quality check being performed. The table below shows the check type, the thresholds of the three adaptive levels in relation to the check type, and the areas each check type impacts.

Adaptive level Threshold Impacts
Cardinality (Standard Deviation)
Strict

Cardinality standard deviation threshold (ABS): 2.1

Cardinality standard deviation min/max threshold (z-score for min/max drift): 3.1

  • Uniques (Cardinality)
  • Mean (Standard Deviation)
Neutral/System Detected

Cardinality standard deviation threshold (ABS threshold): 2.5

Cardinality standard deviation min/max threshold (z-score for min/max drift): 3.4

Lenient

Cardinality standard deviation threshold (ABS threshold): 3.9

Cardinality standard deviation min/max threshold (z-score for min/max drift): 6.1

Cardinality (Percent Change)
Strict Cardinality percent change threshold: 11%
  • Uniques (Cardinality)
  • Type (Percent Mixed Types)
Neutral/System Detected Cardinality percent change threshold: 14%
Lenient Cardinality percent change threshold: 45%
Min/Max (Standard Deviation)
Strict Min/max standard deviation threshold (z-score for min/max drift): 3.1
  • Row Count TopN/BottomN (Standard Deviation/MinMax)
  • Minimum/Maximum (Min/Max)
  • Mean (Standard Deviation)
Neutral/System Detected Min/max standard deviation threshold (z-score for min/max drift): 3.4
Lenient Min/max standard deviation threshold (z-score for min/max drift): 6.1
Min/Max (Percent Change)
Strict Min/max percent change threshold: 12%
  • Row Count TopN/BottomN (Standard Deviation/MinMax)
  • Minimum/Maximum (Min/Max)
  • Type (Percent Mixed Types)
Neutral/System Detected Min/max percent change threshold: 18%
Lenient Min/max percent change threshold: 65%
Standard Deviation Threshold (Deviation from the baseline)
Strict Standard deviation threshold: 2.3
  • Mean (Standard Deviation)
  • Percent Null/Empty (for Lenient only)
Neutral/System Detected Standard deviation threshold: 2.8
Lenient Standard deviation threshold (null, empty): 3.5
Percent Change Threshold
Strict Percent change threshold: 3%
  • Percent Null/Empty
  • Type (Percent Mixed Types)
Neutral/System Detected Percent change threshold: 8%
Lenient Percent change threshold: 40%
Option Description
Lower Bound Alerts when Collibra DQ discovers data in the selected column lower than the limit you set for the Adaptive Rule.
Upper Bound Alerts when Collibra DQ discovers data in the selected column higher than the limit you set for the Adaptive Rule.
pt(s)

The number of points to deduct from the data quality score for a given finding. This relates to the percent change and Z-score when the Z-score ranges from 0.0-6.0.

Turns off alerts for unique findings in the selected column.

Change Detection chart

change detection graph example

The Change Detection line chart shows the range of all passing, failing, and inactive values between the current run date and 20 runs in the past. You can hover your cursor over any available day to view important statistics for that run.

Previous and current TopN/BottomN Values charts

topn/bottmomn values example

The previous and current TopN/BottomN Values charts show the row values that appeared the most and least frequently and satisfy the condition of the Adaptive Rule in the column of reference. In the above example, the Adaptive Rule is a Unique Range check on the "high" column where the condition is high.$uniqueCount NOT BETWEEN 739.9 AND 6963.0. Therefore, the values that appear the most and least frequently and are either greater than 6963.0 or less than 739.9 are considered the TopN and BottomN values.

Indeed, in the above screenshot, the TopN row values of the previous run in the chart on the left were 25.00 and 25.35, which both appeared 11 times. The BottomN values were 27.70, 118.79, 12.16, 11.84, and 105.34, which all appeared only once for that run. On the chart to the right, the TopN value from the current run is 25.20 with 10 occurrences, while the BottomN values are 95.49, 330.94, 35.76, 65.50, and 8.93 with one occurrence apiece.