• Enhancements
• Fixes
• DQ Security

Release Information

• Release date of Data Quality & Observability Classic 2024.09: September 30, 2024
• Release notes publication date: September 5, 2024

Enhancements

Platform

• We added support for FIPS-compliant algorithms.

Integration

• Users of the Quality tab in Collibra Platform who do not have a Data Quality & Observability Classic account can now view the 7-day history of data quality scores, allowing you to monitor the health of your data over time.
• When running a DQ Job with back run and an active integration, the results of the back run are now sent to Collibra Platform where they are stored in the DQ services history table.

Pushdown

• You can now scan for shapes in DQ Jobs created on Trino Pushdown connections.

Jobs

• You can now create DQ Jobs on SAP HANA tables that contain the special characters ::$/-;@#%^&*?!{}~+=

Note The special characters .() are not supported.

Findings

• When the dupelimit and dupelimiui limits on the Admin Limits page are both set to 30, the Findings page now limits the number of dupes findings marked on the Dupes tab to 30.

Alerts

• If your organization uses multiple web pods on a Cloud Native deployment of Data Quality & Observability Classic, you now receive only one alert email when an alert condition is met.

APIs

• When dataset security is enabled on a tenant and a user whose roles meet the requirements of the Dataset Def API and has a their role assigned to the dataset, the API returns the expected results.

Fixes

Integration

• When setting up an integration, the Connections step now has a search component and pagination to prevent table load failure when the tables per schema size exceeds browser memory.
• The dimension type of Adaptive Rules (NULL, EMPTY, MIN, and so on) now correctly maps to the type and sub-type from the DQ dimension table.
• The predicate of custom rules is again included in rule asset attributes.

Pushdown

• You can again download CSV and JSON files containing rule break records of DQ Jobs created on Pushdown connections.
• The rule name no longer appears in the data preview and downloaded rule breaks of rule findings for DQ Jobs created on Pushdown connections.

Note For consistency with rule break downloads in Pullup mode, we plan to separate rule breaks by rule in a future release. As of this release, Pullup mode still includes the rule name and runId in rule break downloads

• DQ Jobs created on Pushdown connections where columns with shape values that contain $ or ‘ now run successfully. Previously, such Jobs failed with an unexpected exception message.
• When running DQ Jobs created on Pushdown connections that scan multiple columns for duplicate values, the data of the columns now appears under the correct column name.
• When a DQ Job created on Pushdown connection contains 0 rows of data and one or more rules are enabled, the rules are now included in the Job run and displayed on the Findings page.

Jobs

• The Explorer connection tree now loads successfully when a schema contains tables that contain unsupported column types.
• Dataset Overview on the Explorer page can now process the select * part of the SQL statement if there is an unsupported column type.
• We fixed an issue on the Jobs page where Data Quality & Observability Classic was unable to retrieve the Yarn Job ID.
• When re-running a DQ Job from the metadata bar on DQ Job that previously ran with backrun (-br), the DQ Job that you re-run will no longer incorrectly initiate a backrun.

Note If the -br option is included in the beginning of your command line, your DQ Job will perform a backrun and -br will be removed from the command line when the DQ Job completes.

Findings

• After retraining a behavioral finding to pass a value for a blindspot, the score now correctly reflects the retrained scoring model.

Profile

• When adding a stat rule for distribution from the +Add Rule option on the Profile page, the computed boundaries of categorical variables in the distribution rule now display correctly.

Rules

• When rules on Pullup datasets time out, the rule output record now displays the out-of-memory (OOM) exception message on the Findings page.
• Run Result Preview on the Rule Workbench now works as expected for custom rules that use the simple rule template.
• When creating a rule for an existing DQ Job created on a Pushdown connection, Run Result Preview now runs without errors.
• You can now use rules that contain a $ (not stat rules) with profiling off for DQ Jobs created on Pushdown connections.

Admin Console

• Admin Limits now require values to be -1, 0, or positive numbers. An inline message appears below the Value field when a limit does not meet the allowed values.

Latest UI

• We improved the performance of the Scorecards page when there are many datasets to load.
• We reduced the number of backend calls on the Profile and Findings pages to improve the load time performance.
• You can now create, edit, and delete alerts for datasets with 0 rows. When you run a job on a dataset with 0 rows, the alerts function as expected.
• Dataset Manager no longer crashes due to slow network calls when you click the Filter icon as the page loads.
• We resolved multiple scenarios where the metadata bar did not display any rows or columns.
• When hovering over a row on the Scheduler page, the row has a gray highlight, but the days of week cells remain green or white, depending on whether a DQ Job is scheduled to run on a given day.
• The date picker on the Job tab of the Findings page is now available for DQ Jobs created on Pushdown connections and you can successfully run DQ Jobs with the dates you select.
• The Findings and other pages now load correctly in Safari browsers.
• DQ Jobs created on Pushdown connections no longer generate duplicate user-defined job failure alerts.
• The donut charts in the database and schema reports from Explorer now consistently display the correct stats.

DQ Security