Release 2022.01

Enhancements

  • DQ Job
    • Fixed issue where backrun "-br" flag was inadvertently added on future runs (error contained in 2021.12) if the initial DQ Job setup Explorer selected backrun
    • Improved validation to not allow for slashes in dataset name
  • Validate Source
    • Fixed potential DQ Job failure with Source turned on for some legacy installations when upgrading from older versions to 2021.11 and newer
  • Explorer
    • DB_VIEWS_ON can be added with TRUE or FALSE values by adding new App Config (Add Custom within Admin -> Configuration)
    • -Addlib flag now working across JDBC connections
    • Update Scope now supports rdEnd
  • Rules
    • When creating rules, run-time limit for each rule (in minutes) can be set on the Rule page UI and on the V3 API (by setting runTimeLimit property). The default is 30 minutes if not explicitly set. This 30 minute limit sets the overall timeout limit for all rules in a particular job. For example, if there are 10 rules with 9 rules with 30 min limit and 1 rule as 90 min limit, then the DQ Job will wait up to 90 min for all 10 rules to finish. This is because all rules must finish before the Rule stage in DQ Job to finish and move to the next stage. We do not support async stages where one long running rule is running while the job itself moves on to the next stage.
    • Added ability to specify score of 0 to a rule
    • Improvement to Stat Rules to fail without exception when result is not within range
  • Profile
    • Fixed ability to remove a business unit from a dataset
    • Fixed issue where data concepts were not correctly displaying on a dataset's Profile page
    • Fixed sensitive labels not being assigned from Discovery
    • Treat certain doubles, floats, decimal types as Decimal format that preserves length and prevents Java from truncating to E11 format
    • Removed commas when displaying date columns
  • Security
    • SAML Login fix for IDPs that use POST binding as default
  • S3
    • Enhanced support where "." in column headers were causing large jobs to not complete
      • Underscores now replace periods and large jobs should no longer hang
  • Connections
    • Updated default Snowflake template connection properties
      • Corrected 'db' parameter placeholder on connection string versus former 'databaseName'
    • Added Connectivity to BigQuery troubleshooting information

Known Limitations

  • Local files using NO_AGENT require a valid $SPARK_HOME on the machine where the web server is running.
  • Supported data types
    • CLOB datatypes are unsupported
  • Explorer
    • -Addlib not yet supported for Remote Files e.g. S3

[Informational Only] Changes To Metastore Made In 2022.01

ALTER TABLE owl_rule ADD COLUMN IF NOT EXISTS run_time_limit DOUBLE PRECISION NOT NULL DEFAULT 30.0;
ALTER TABLE owl_rule ADD COLUMN IF NOT EXISTS scoring_scheme INT4 NOT NULL DEFAULT 0;

ALTER TABLE job_log ALTER COLUMN stage TYPE character varying; -- stage set to varchar because RULE logs rule_nm into stage
ALTER TABLE job_log ALTER COLUMN log_desc TYPE character varying;
ALTER TABLE job_log ALTER COLUMN log_hint TYPE character varying;