Release 2022.08

New Features

Rules

Enhancements

Connections

  • You can now authenticate Oracle JDBC connections with Kerberos TGT, Keytab, and Password. (tickets #75267, 76030)
  • You can now authenticate SQL Server JDBC connections with Kerberos Keytab in addition to basic authentication.

Rules

  • Rule Summary enhancements:
    • You can now select different time periods for analysis.
    • You can now view charts from three different pages, including Rule Detail Summary, Rule Breaks, and Rule Dimension Summary.

Security

  • Vulnerabilities identified by Jfrog
    • Vulns 0, criticals 0, high severity 7
    • The majority of the current mediums are due to merging the dq-streaming module into core.
    • For a visual readout, see the DQ Security Metrics section below.

Agent

  • You can now optionally configure individual time zones of DQ Job, Web, and Agent. You should only use this configuration when your instance and containers run in different system time zones. (tickets #87024, 87155)

Behavior

  • The Behavior tab now has a new column, Delta Percent Change (Δ % Change).
  • You can now hover over new tooltips in the following columns:
    • Baseline
    • % Change
    • Δ % Change
    • Zscore
    • Score

Outliers

  • Outlier checks are now optimized to skip in certain circumstances. Outlier checks are only skipped when the history load of a specified date column is empty.
  • You can now update and modify record flags from the command line with -rc, -rcKeys, -rcDateCol, and -rcTbin.

API

  • The v2/gethoot API now properly returns rule dimension information for data sets.
  • The v3/jobs/run API now has improvements to the 400 Bad Request error messages in specific circumstances.

Reports

  • The PDF option is now removed from the Dataset Findingspage. To print dynamic column tables, use CSV or Excel options instead. (ticket #89739)

DQ Connector

  • The version of Collibra Integration Library is now updated to 2.4.12.

Fixes

Connections

  • The new GCS jars are required to use GCS spark-history-server. (ticket #90623)

DQ Job

  • Fixed an issue that caused jobs using .TXT files to incorrectly render custom column names. (ticket #81808)
    • Files with .TXT extensions are now treated as delimited files. Files with .TXT extensions that are not delimited files should use their respective file type from the file type dropdown.
  • Fixed an issue with deployments on K8s where jobs failed when the volume name exceeded 63 characters. (ticket #85372)

Agent

  • Fixed an issue that caused the v2/updateagent API to fail when numCores was empty. (tickets #89737, 92404, 92680)
    • The numCores field is no longer a required field.

Validate Source

  • Fixed an issue that caused validate source jobs to fail when the pkey was mapped to different column names. (ticket #88778)

Rules

  • When using Freeform SQL rules with wild-card operators, rules again correctly pass validation. (ticket #89644)
  • Fixed an issue with regex rules that use the characters ), , , and ; in the rlike, which caused DQ to append spaces to those characters and prevented the regex from operating correctly. (tickets #89417, 92958)
  • Fixed an issue that caused rules with column values containing parentheses ( ) to break due to the addition of padding before and after closing parentheses. (ticket #85176)
  • Fixed an issue that caused rules with special characters such as @ to display incorrectly on the Rules page, Conditions tab, and when exported to Excel.
  • Fixed an issue that prevented data sets with attached rules and roles from being renamed. (tickets #85731, 92059, 94315)

Profile

  • Fixed an issue where certain results in TopN Values and Data Preview displayed in scientific notation. Scientific notation is now removed from the display. (tickets #82163, 89738)

Explorer

  • Fixed an issue that allowed CLOB data types to be visible in the Drag Columns to Target map in the Source tab. (ticket #86902)

API

  • The REST API endpoint v2/updateRoleDatasets again correctly saves roles to data sets.

Known Limitations

Rules

  • The Findings page displays results from computational stat rules on mean as a single-quote string. For example, '573523.87' > 6763
  • Column-level sorting for the Rule Summary feature is not currently available.

Admin

  • When adding a Sensitive Label or a Data Category, the Edit and Update functions do not display the selected record. To properly display the record, you must first refresh the page before editing or updating.

Session Activity

  • While the application UI is being redesigned, it is possible that when the application times out on the legacy side of the application, you might not be able to see it on the new React MUI side. This can happen when you have the DQ application open on multiple tabs.
    • We are not currently tracking session timeout from the legacy UI to React.

Features in preview

DQ Job

  • Collibra is proud to launch a brand new feature, Snowflake Pushdown. Snowflake Pushdown allows for even faster processing and removes the need to set up a separate Spark compute platform to run Collibra Data Quality. Snowflake Pushdown is a private preview feature only available by request. Since this is a feature in preview, some limitations are expected as we continue to improve its functionality. Contact your CSM to learn more about this feature.

DQ Security Metrics

Warning  There is a critical CVE CVE-2016-1000027 that shows up in the image scan due to Spring version. This is a false positive and should be added to the exception list of the customer scan tools. We don’t use HttpInvokerServiceExporter anywhere in the application and are not impacted by it.

There is no fix version available for it from Spring. More details are available at Sonatype vulnerability CVE-2016-1000027 in Spring-web project · Issue #24434 · spring-projects/spring-framework

DQ security vulnerabilities over 5 months

Critical security vulnerabilities over 5 months