• Enhancements
• Fixes
• Limitations
• DQ Security

Release Information

• Release date of Data Quality & Observability Classic 2024.04: April 29, 2024
• Publication dates:
  • Release notes: April 4, 2024
  • Documentation Center: April 4, 2024

Enhancements

Capabilities

• You can now authenticate SQL Server connections using Active Directory Password and Active Directory Service Principal. This enhancement, available in both Java 11/Spark 3.4.1 and Java 8/Spark 3.2.2, allows you to use Azure AD-based Synapse SQL and Azure Service Principal authentication in Data Quality & Observability Classic, to further enable your team to follow Azure authentication best practices and InfoSec policies.
• We improved some of the ways you can work with the Scorecards page:
  • The Page dropdown menu is now sorted alphabetically.
  • You can now rename scorecard pages without the need to recreate them from scratch.
  • You can now change the order in which scorecards are displayed on a scorecards page by clicking the up or down arrows to the left of a scorecard.
• Archive Break Records for both Pushdown and Pullup jobs now parses linkId values in the Sample File Preview and Rule Breaks Preview on the Findings page and in the downloadable CSV file.
• Running a job with a SQLF rule that references a secondary dataset or uses @t1 no longer flashes an extra record in the job log or on the Jobs page with an Agent Id of 0. With this enhancement, only a single job record displays on the Jobs page. (idea #DCC-I-2413)
• When using the Dataset Overview, you can now click Download Results to download a CSV file containing the contents of the Results table.

Platform

• We improved some of the ways you can create or edit alerts:
  • To enhance the fluidity of batch alerting from the Status Alert and Condition Alert modals, you can now select the new Batch option to display the Batch Name field, where you can search, select, or create alert batches.
  • With the Batch option selected, the Alert Recipient field locks. To edit this field, you can click the to unlock it. When the Batch option is not selected, the Alert Recipient field remains unlocked and editable.

Fixes

Capabilities

• We improved the Data Shape Granular setting to include more string length values in Shapes findings. (ticket #135351)
• When joining datasets created from two different data source connections with Kerberos keytab as the authentication type, we fixed an issue that prevented the secondary dataset from loading because its Kerberos authentication was incorrectly passed. (ticket #131309)
  • To ensure that the secondary dataset is passed correctly during Kerberos authentication, add -jdbcprinc and -jdbckeytab to the Agent configuration's Free Form (Appended) section. For example, -jdbckeytab /tmp/keytab/dq-user.keytab -jdbcprinc [email protected]
• We added the ability to export all job schedule records from the Jobs Schedule page, using the Export All option. Previously, the ability to export job schedule records was limited to up to 20 records per page. (ticket #136970)
• We added the following enhancements to the Findings page when data quality findings exceed certain thresholds. (ticket #136922)
  • When there are more than 9,999 findings of any data quality layer, the value displayed in the badge on the corresponding findings tab will round to the nearest thousand with a +. For example, 12,345 will display as 12K+.
  • When there are more than 999,000 findings of any data quality layer, the value displayed in the badge on the corresponding findings tab will always display as 1M+. For example, 1,234,567 will display as 1M+.
  • When a value is truncated, you can hover your cursor over the badge to display the exact number of findings.
• When using SAML SSO to sign into multi-tenant Data Quality & Observability Classic environments, the SAML Enabled Tenants dropdown menu no longer shows the Tenant Name. Instead, the dropdown menu now shows the Tenant Display Name. (ticket #137865)
• We removed the runId from the “Findings - Most Recent Run” link in alert emails to correctly take you to the most recent run of your job when you click the link.
• When the “Findings - Run which Produced the Alert” link in alert emails contains a runId and a timestamp in the URL, you will be taken to that specific job runId and timestamp when you click the link.

Platform

• We introduced pagination to the Rule Summary page while limiting each page to displaying a maximum of 25 records. Previously, the Rule Summary page only displayed 25 records on a single page, even when the number of records should have exceeded 25. (ticket #140229)
• When an admin sets a limit for the datashapelimitui setting from Admin Limits, the Findings page no longer displays Shapes findings beyond that limit. (ticket #129091)
• The Role Management page in the latest UI now allows admins to set the access rights of roles and users when associating them with specific datasets (ticket #138489)
• The Dataset Manager page now loads correctly when the alias name of a dataset is null. (ticket #133400)
• When filtering by row count on the Dataset Manager page, the results included in the filter no longer include daily row counts that exceed the range you select. (ticket #133453)
• JSON files from Amazon S3 connections no longer fail to load when Livy is not enabled.
• When large datasets (for example, one with more than 100 million records) timeout with a 504 error response when loading the table in Explorer, an error message appears in the Explorer UI with details about the error. (ticket #133530)

Pushdown

• Databricks Pushdown now supports ANSI-compliant SQL on the server side. (ticket #136562)
• The out-of-the-box Data Category, Currency_CD, no longer counts the number of null and empty values as part of the underlying SQL query. (ticket #133578)

Latest UI

• We added support for multiple pages of results on the Rule Summary page.
• We improved the performance of the Pulse View when loading large amounts of data.
• We updated the display of the Pulse View when scrolling.
• The Command Line input field on the Findings page now supports vertical scrolling.
• The Source tab on the Findings page now displays all labels for Cell results.
• We updated the Job Schedule export function to include all jobs instead of limiting them to 20.
• We added the ability to assign ACL to datasets from the Role Management page in the Admin Console.
• We added Copy and Export buttons to the View AR modal on the Findings page for Behaviors.
• We fixed the ability to change the upper and lower bound values for AdaptiveRules.
• We now display the exact number of results on the tabs for each of the layers on the Findings page instead of "99+" for all values over 99.
• The Rules tab now updates automatically when you add Quick Rules from the Data Preview section.
• Success messages now appear on the Findings page when you use the Validate, Invalidate, and Resolve functions.

Limitations

Platform

• Due to a change to the datashapelimitui admin limit in this release, you might notice significant changes to the number of Shapes marked on the Shapes tab of the Findings page. While this will be fixed in Data Quality & Observability Classic 2024.06, if you observe this issue in your Data Quality & Observability Classic environment, a temporary workaround is to set the datashapelimit admin limit on the Admin Console > Admin Limits page to a significantly higher value, such as 1000. This will allow all Shapes findings to appear on the Shapes tab.

DQ Security

Important A new high vulnerability, CVE-2024-2961, was recently reported and is still under analysis by NVD. A fix is not available as of now. However, after investigating this vulnerability internally and confirming that we are impacted, we have removed the vulnerable character set, ISO-2022-CN-EXT, from our images so that it cannot be exploited using the iconv function. Therefore, we are releasing Data Quality & Observability Classic 2024.04 with this known CVE without an available fix, and we have confirmed that Data Quality & Observability Classic 2024.04 is not vulnerable.