Release 2024.09

Security

• We improved the security of lineage capabilities via Edge.

Metadata integrations

• The technical lineage capability synchronization no longer fails due to large metadata files sent to DIP.

Protect

• Protect for BigQuery now properly handles project IDs and dataset names that include dashes.

Lineage Harvester (CLI and Edge)

• When integrating SSRS, Collibra Data Lineage now skips reports for which the configured user does not have sufficient access rights and creates an analyse error.
• When harvesting Oracle data sources, the memory footprint is now reduced, allowing the lineage harvester to successfully harvest metadata from much larger data sources.

Protect

• Protect for Google BigQuery handles non-protected tables being renamed during synchronization.

Security

• We improved the security of capabilities via Edge.

Metadata integrations

• OAuth autentication now works as expected with Edge Proxy for Databricks Unity Catalog.
• When creating technical lineage for Dataplex, Collibra Data Lineage now ingests interactive queries by default. You can disable the ingestion of interactive queries when synchronizing the capability.

Lineage Harvester (CLI and Edge)

• When harvesting PBRS data sources, the Collibra Data Lineage now honors the no_proxy configuration.

Protect

Lineage Harvester (CLI and Edge)

• When harvesting Oracle data sources, Collibra Data Lineage now supports TRIGGERS.
• When creating a custom technical lineage via the CLI lineage harvester or Edge, using the batch definition method, the lineage harvester now validates the syntax in your JSON files.
• The lineage harvester (Edge and CLI) and the technical lineageEdge capabilities are using the DNS names documented in Collibra Data Lineage service instances.
• When harvesting Oracle technical lineage sources, logging can now be enabled in the Edge harvester, to aide troubleshooting. Data object harvesting details are captured in a log file in the metadata batch.
• The Dataplex lineage can now handle GCS file names which contain wildcards.

Lineage Harvester (CLI and Edge)

• The Analyze Only option in the technical lineage Edge capabilities is now deprecated. The functionality is replaced by the new Processing Level drop-down list. This new mandatory setting gives you more control in configuring when metadata is harvested, analysed, and synchronised.
• When ingesting Snowflake data sources using SQL-API mode, the SQL queries to harvest the data are updated to be more lightweight.
• When creating a custom technical lineage via the CLI lineage harvester or Edge, using the batch definition method, the lineage harvester now validates the syntax in your JSON files.

Security

• We improved the security of integrations via Edge.

Metadata integrations

• You can now download the input metadata from the Databricks Unity Catalog Lineage "Synchronization Results" dialog box if you selected the "Save Input Metadata" checkbox when you created the Databricks Unity Catalog Lineage capability.
• When you synchronize a Vertex AI integration, you can now download the input metadata file from the Sync results dialog box on the Activities page. In order to achieve this, you must select the "Save Input Metadata" checkbox when you create a Vertex AI capability.
• When registering a Databricks data source, a new domain is now created for each database and schema if they are new and no explicit domain mapping is provided.
  • Databases follow the naming pattern "systemDomainName > databaseName," while schemas follow "databaseDomainName > schemaName."
  • Domain mapping takes precedence over the creation of new domains.
  • Existing Database and Schema assets remain in their current domains.
• The Dataplex lineage can now handle GCS file names which contain wildcards.

Collibra Protect

• To prevent excessively log entries, Protect for BigQuery no longer logs debug messages.
• BadSqlGrammarException errors reported in Protect for Databricks and Snowflake now include the underlying root cause instead of vague descriptions.

Security

• We improved the security of integrations via Edge.

Data Catalog

• Dataplex technical lineage now also supports Google Cloud Storage (GCS) assets that have a full name which starts with gcs: in addition to gs:.
• The AI model integration for Databricks Unity Catalog now correctly handles cases where a model has an associated run that no longer exists.
• The Databricks Unity Catalog integration now handles errors during synchronization better, ensuring that the synchronization job does not get stuck.
• The Amazon S3 integration now successfully completes when encountering an AWS Glue table with duplicated column names. The synchronization detects duplicated columns and logs a warning about the duplication, advising you to fix the issue on the data source side. The column position will not be included for these columns.
• We updated the error message for AWS SageMaker AI integrations in unsupported regions.
• You can now indicate that you don't want asset statuses to change during a Databricks resynchronization. The Databricks synchronization configuration includes a "Default Asset Status" field. The default value is "Implemented" and is available for all Databricks ingestion types.
  • If "Implemented" is selected, all assets are ingested with "Implemented" status.
  • If "No Status" is selected, all assets are ingested with the first status listed in the Operating Model statuses, and changes will not be overwritten. For example, if you change an asset status from "Candidate" to "Implemented" before the resynchronization, the status remains "Implemented."

Infrastructure

• You can now download a new version of the Edge CLI outside the legacy Edge tool. For more information, go to the Edge CLI documentation.
• Edge sites with a proxy no longer fail or run into issues when updating the proxy settings.

Lineage Harvester (CLI and Edge)

• The capability for technical lineage for OpenLineage is now visible in the “capability template” drop-down list, preparing for public availability. Note that only participants in the private preview should use this feature.
• When ingesting Oracle data sources, the Collibra Data Lineage service instances now support scoped database link names.
• If Tableau integration fails because of timeout errors due to page sizing limits, Collibra Data Lineage now automatically adjusts the settings and retries.

Security

• We have improved the security of Collibra Protect.

Collibra Protect

• Protect for Snowflake supports column names containing spaces and special characters.
• You can now choose how Snowflake checks roles (that is, Protect groups) for applying standards and rules, accommodating Snowflake users who have multiple roles. When adding a capability for Protect for Snowflake, the "Edit Capability" dialog box contains a new "Snowflake role testing" field with the following options:
  • - "CURRENT_ROLE" (default): Checks only the primary role assigned to the Snowflake user.
  • - "IS_ROLE_IN_SESSION": Checks all the roles assigned to the Snowflake user, including secondary roles, within the active session.

Metadata integrations

• The Azure ML integration now supports the integration of custom AI models in Azure ML.
• You can now download the input metadata of an Azure ML synchronization from the Synchronization Results dialog box.

Lineage Harvester (CLI and Edge)

• When synchronizing a technical lineage via Edge, Collibra Data Lineage now returns the correct result message.
• When ingesting Matillion data sources via Edge, the Collibra Data Lineage service instances can now process the "startTimestamp" parameter.

Security

• We improved the security of integrations via Edge.

Metadata integrations

• For Dataplex Lineage , you can now download the metadata input file from the "Synchronization Results" dialog box.
• For Dataplex Lineage, interactive queries that are run in GCP are no longer shown in the source code pane of the Technical Lineage Viewer.
• The Databricks Unity Catalog Lineage integration now supports OAuth. When you create a connection to Databricks Unity Catalog via Edge, you can now select OAuth as the Authentication Type.

Collibra Data Lineage Service

• When ingesting SAP Analytics Cloud data sources, the Collibra Data Lineage service instances now successfully process the metadata, regardless of the order in which the SAP Analytics Cloud assets are returned by the SAP Datasphere API.
• When ingesting Oracle data sources, Collibra Data Lineage now correctly sets the database and schema for packages, and correctly shows the package name.
• When ingesting SAP Hana data sources, the Collibra Data Lineage service instances now support the use of the (+) operator in WHERE clauses.
• When ingesting Snowflake data sources, the Collibra Data Lineage service instances now support:
  • SQL statements that contain a lot of Common Table Expressions.
  • SQL statements with a wildcard in the select list and indirect lineage that refers to the wildcard.

Security

• We have improved the security of Edge, Catalog Data Classification, Technical Lineage for Databricks, Azure ML, Databricks Unity Catalog synchronization, ADLS Synchronization and SAP Datasphere.

Protect

• If a column is replaced by another column with the same masking and Protect groups (GCP principals), Protect for BigQuery now applies masking to the new column.

Metadata integrations

• JDBC Metadata Synchronization via Edge is now more stable. A retry mechanism was added to the data transfer to Data Intelligence Platform.
• The Databricks Unity Catalog metadata integration now support OAuth. When you create a connection to Databricks Unity Catalog via Edge, you can now select OAuth as the Authentication Type. This authentication method is not supported for Databricks Unity Catalog Lineage from Edge 2024.07-91.
• You can now use the AWS SageMaker AI capability if your Edge site is installed using one of the following types of forward proxy:
  • Path through (No authentication)
  • Path through (Basic authentication)
  • No proxy for noProxy hosts defined by Edge
Note The AWS SageMaker AI capability can't be used on an Edge site installed using a Man-in-the-middle (MITM) proxies.
• You can now use the Azure ML capability if your Edge site is installed using one of the following types of forward proxy:
  • Path through (No authentication)
  • Path through (Basic authentication)
  • No proxy for noProxy hosts defined by Edge
  Note The Azure ML capability can't be used on an Edge site installed using a Man-in-the-middle (MITM) proxies.
• : You can now participate in a preview testing for the new Dataplex Catalog integration. In view of this beta, the "Ingestion Type" field is shown when you synchronize a Dataplex data source. If you are interested in using this new feature, do sign up for the beta to get access to the documentation, provide feedback, and get support.

Lineage Harvester (CLI and Edge)

• We fixed an issue with the CLI Harvester URI syntax on Windows.
• When ingesting Netezza data sources via Edge, Collibra Data Lineage now correctly harvests views when multiple databases are specified in the capability.
• When integrating Tableau Server data sources via Edge, Collibra Data Lineage now successfully harvests the default site, even if it was renamed in Tableau.

Collibra Data Lineage Service

• We made various small improvements to the overall performance and user experience.

Collibra Data Lineage Service

• We made several improvements for Oracle data source ingestion, which result in improved lineage extraction and faster processing. The improvements also establish a foundation for future enhancements, such as support for stored procedures and functions.
• When ingesting BigQuery data sources, the Collibra Data Lineage service instances now support the use of the function ANY_VALUE in the PIVOT clause.
• When integrating any of the supported BI or ETL data sources via Edge, theCollibra Data Lineage service instances now correctly analyze wildcards in embedded SQL statements.
• When you ingest Snowflake data sources via the SQL-API mode, Collibra Data Lineage now delivers improved lineage results and fewer parsing errors due to metadata processing improvements.

Infrastructure

• We fixed an issue where technical lineage capabilities that used the Shared Storage connection failed, due to a restrictive bucket quota.

Metadata integrations

• The SAP Datasphere integration now includes Column assets.

Collibra Data Lineage Service

• When ingesting MySQL data sources, the Collibra Data Lineage service instances now correctly construct schema names from the SQL statements.

Infrastructure

• We fixed an issue where the Edge capability list was broken for customers who were using the Classic UI and had a deprecated DQ connector. With this fix, the capability list will display successfully in both the Classic and Latest UI, as expected.

Security

• We have improved the security of Edge and Classification.

Collibra Data Lineage Service

• Improved performance when integrating Power BI with DAX analysis enabled.

Collibra Data Lineage Service

• When ingesting BigQuery data sources, the Collibra Data Lineage service instances now support the use of "values" as an unpivot column name.
• When ingesting Microsoft SQL Server data sources, the Collibra Data Lineage service instances now support:
  • EXECUTE statements with variables.
  • The OPTIMZE_FOR_SEQUENTIAL_KEY option in CREATE INDEX statements.
• When ingesting JDBC data sources, the “Last sync time” column in the Technical lineage Sources tab page now shows the correct time, regardless of any parsing errors. Previously, if there were only parsing errors, the last sync time shown was incorrect.
• When ingesting DataStage data sources, the Collibra Data Lineage service instances now update column names within file_item objects to capital letters.
• When ingesting Snowflake lineage via SQL-API mode, the Collibra Data Lineage service instances no longer return a UNIQUE constraint failed error due to schema versioning.

Security

• We have improved the security of data classification.

Data Quality

• In order to maintain security best practices, we have removed the DQ connector and DQ App Service from Edge. We recommend using the API-based DIP<>DQ integration instead. If you are still migrating off of the DQ connector, we recommend you enable manual upgrades to remain on an older, supported version of Edge until you have fully migrated.

Lineage Harvester (CLI and Edge)

• When integrating SAP Analytics Cloud, you can now configure container or folder filtering via the Data Catalog UI. For more information, go to Create a technical lineage via Edge.
• When creating a technical lineage via the lineage harvester and providing passwords via the command line, you can now use “—passwords-stdin” with the “list-sources” and “ignore-source” commands. For more information on how to provide passwords, go to Technical lineage password manager integration design.

Security

• We have improved the security of Edge.

Infrastructure

• We have made improvements to the Edge installer to reduce future security vulnerabilities.

Security

• We have improved the security of Databricks Unity Catalog synchronization, Technical Lineage for Databricks Unity Catalog and Azure ML.

Metadata integrations

• The Databricks Unity Catalog integration no longer fails if tables are deleted in Databricks during the synchronization process.
• We improved the security of integrations via Edge.

Infrastructure

• We improved the installation commands for Edge sites installed on:
  • Shared clusters via the Edge CLI method.
  • Dedicated or shared clusters via the Helm chart method.
• Edge sites installed on bundled k3s from the 2024.07.42 release are on k8s 1.29. No action is required.

Security

• We have improved the security of Edge.

Security

• We have improved the security of Data classification via Edge.

Metadata integrations

• When integrating SAP Datasphere or SAP Analytics Cloud, SAP assets of types that are not supported by Collibra no longer cause the integration to fail. The presence of such assets now results in a warning.
• Edge integration capabilities can now use a custom truststore when there is TLS termination at the firewall.
• When you integrate Databricks Unity Catalog AI models, you can now choose to exclude all AI models in the 'system' Databricks catalog.

Infrastructure

• Connections tests can now be completed successfully if the direct queries flag is enabled, as expected.

Security

• We have improved the security of Edge and Data classification via Edge.

Data classification

• You can now follow up on the results of a classification job by checking the Results dialog box. (idea #DCC-I-5694)
• For numeric and date columns in Oracle, the sample data no longer shows the value "Invalid value" for all samples.

Security

• We have improved the security of Edge.

Metadata integrations

• The technical lineage for Google Dataplex is now generally available. Via the Technical Lineage for Dataplex capability, you can ingest lineage from Google Dataplex Catalog into Collibra Catalog.

Data classification

• The Unified Data Classification process is now more stable when working with large datasets.

Metadata integrations

• The AWS SageMaker AI integration via Edge is now available in public preview. When you integrate Amazon SageMaker, you integrate the metadata of ML models from Amazon SageMaker AI to Collibra Data Intelligence Platform. The resulting assets represent the AWS SageMaker AI model.
• The Azure ML integration via Edge is now available in public preview. When you integrate Azure AI, you integrate the metadata of ML models from Microsoft Azure ML to Collibra Data Intelligence Platform. The resulting assets represent the Azure ML model.

Security

• We have improved the security of Data classification.