Only show release notes of Collibra Platform for Government-certified featuresRelease 2023.11
Release information
- Publication dates:
- Release notes: October 19, 2023
- Documentation Center: November 2, 2023
- Release date of Collibra 2023.11.0: November 5, 2023
- Upgrade non-production environments: November 5, 2023
- Upgrade production environments: November 19, 2023
- Release date of Collibra 2023.11.1: November 19, 2023
- Release date of Collibra 2023.11.2: November 19, 2023
- Release date of Collibra 2023.11.3: December 10, 2023
- Release date of Collibra 2023.11.4: December 10, 2023
- Release date of Collibra 2023.11.5: December 20, 2023 (Collibra Platform for Government only)
- Release date of Collibra 2023.11.6: On demand only
- Release date of Edge 2023.11.1: November 5, 2023
- Release date of Edge 2023.11.2: November 12, 2023
- Release date of Edge 2023.11.3: November 19, 2023
- Release date of Edge 2023.11.4: November 26, 2023
- Release date of Edge 2023.11.5: December 3, 2023
- Release date of Edge 2023.11.6: December 10, 2023
- Release date of Edge 2023.11.7: December 17, 2023
- Release date of Edge 2023.11.8: January 7, 2024
- Release date of Edge 2023.11.9: January 14, 2024
- Release date of Edge 2023.11.10: January 21, 2024
- Release date of Edge 2023.11.11: January 28, 2024
- Relevant Jobserver version: 2023.11.0-49
Highlights
- Data Catalog
-
- The feature to register source tags from Snowflake is now generally available for Cloud customers. With this general availability, we improved the way these tags are displayed.
- Data Lineage
-
- CollibraData Lineage support for dbt Cloud and dbt Core is now generally available. You can create technical lineage for dbt Cloud and dbt Core by using the lineage harvester and technical lineage via Edge.
- When you create technical lineage for Looker, you can now define the number of concurrent request and connection timeout in the configuration file. Previously, you might experience HTTP 401 Unauthorized errors due to too many concurrent requests.
-
- When creating a custom technical lineage, you can now opt for the new batch definition option, which includes the following benefits:
Support for indirect and table-level lineage as well as column-level lineage.
Support for any asset type with stitching.
You can now submit multiple JSON files.
You no longer have to define every asset used in lineage relation in a separate section.
The new batch definition option is in preview. After a successful preview period, we will announce the deprecation of the single-file definition option.
(idea #DCC-I-783, DCC-I-913, DCC-I-1190, DCC-I-1637)
- When creating a custom technical lineage, you can now opt for the new batch definition option, which includes the following benefits:
- Data Marketplace
-
- The feature to manage personal and group-specific filters in Data Marketplace is now generally available. With this general availability, we now allow one user group to have multiple assigned group-specific filters. (idea #DCC-I-1080)
- You can improve the search experience in Data Marketplace by tailoring the available filters in the Filters pane. (idea #DCC-I-1580)
You can now:- Change the display name of filters including the out-of-the-box filters.
- Add descriptions to filters, which will appear as tooltips in the Data Marketplace Search Results page.
- Define whether a filter should be expanded or collapsed by default.
- The shopping basket feature has been enhanced. (idea #DCC-I-219, DCC-I-2329, CAT-I-282)
- You can now show or hide the shopping basket in your entire environment.
The existing option to show or hide the shopping basket in Data Marketplace now also impacts the visibility of the shopping basket in other areas of your environment, such as Data Catalog.- To make sure that the shopping basket options can be used by customers who are not using Data Marketplace, we moved these options to their own section in the Data Marketplace Settings page.
- The Data Marketplace Settings now include an overview page of all the settings and useful links to documentation, university courses, and demos.
- You can now also define which asset types you want to allow in the shopping basket. By default, the shopping basket allows reports and data sets.
Important When you allow extra asset types, the Request Assets Access workflow must be updated accordingly.
- You can now show or hide the shopping basket in your entire environment.
- Data Privacy
-
- A new version of Data Privacy is now released with the following changes:
- The Business User Interface is now deprecated. As a result, the interface is no longer accessible and the related workflow is removed from the CMA files for 2023.11. The Business User Interface was an external web page for Business Users to view and propose Business Process assets in a domain.
- The button in the email invitation requesting your input on a Process Register domain now opens the Process Register domain from which the email was triggered.
- A new version of Data Privacy is now released with the following changes:
- Edge
-
- Edge is now compatible with the latest versions of k3s and EKS.
K3s: Edge supports k3s versions 1.22 and higher.
- Reinstall your Edge site to upgrade to the latest version of k3s.Important In the next quarterly release, 2024.02, Edge sites using k3s versions older than 1.27 will be deprecated. While we do not anticipate this change to break existing Edge sites, support will be limited for troubleshooting Kubernetes versions that are no longer officially supported by the Kubernetes foundation. You will need to reinstall your Edge site to upgrade to the latest version of k3s before the 2024.02 release. For more information, go to the Edge Compatibility table.
- Reinstall your Edge site to upgrade to the latest version of k3s.
- EKS: Edge supports EKS versions 1.23 and higher for existing Edge sites. New Edge site installations must be installed using EKS 1.27.
You must follow these instructions on how to disable Datadog's pod policy in order to successfully upgrade your EKS version from EKS 1.24 or lower to the latest supported version, EKS 1.27.
Important In the next quarterly release, 2024.02, Edge sites using EKS versions older than 1.27 will be deprecated. While we do not anticipate this change to break existing Edge sites, support will be limited for troubleshooting Kubernetes versions that are no longer officially supported by the Kubernetes foundation. Go to our Support site to follow the instructions on how to upgrade your Edge sites using EKS from EKS 1.24 and lower to EKS 1.27 before the 2024.02 release. For more information, go to the Edge Compatibility table.
- Edge is now compatible with the latest versions of k3s and EKS.
- Platform configuration
-
- You can use your own encryption key to encrypt the virtual disks of your Cloud environment. With this, you have full control over your data and can revoke access to that data at any time.
- SCIM
-
- The System for Cross-domain Identity Management (SCIM) feature is now generally available for Okta and Azure. SCIM enables you to seamlessly automate user provisioning and deprovisioning between your Identity Providers and Collibra and adds standardization, real-time synchronization, and enhanced security to identity management.
Note Currently, due to a token refresh limitation, you can only run SCIM provisioning for Azure AD manually.
Learn more: SCIM documentation
- The System for Cross-domain Identity Management (SCIM) feature is now generally available for Okta and Azure. SCIM enables you to seamlessly automate user provisioning and deprovisioning between your Identity Providers and Collibra and adds standardization, real-time synchronization, and enhanced security to identity management.
New features
Data Marketplace
- The feature to manage personal and group-specific filters in Data Marketplace is now generally available. With this general availability, we now allow one user group to have multiple assigned group-specific filters. (idea #DCC-I-1080)
- If Data Quality is integrated in Collibra, we now show the DQ scores in the Table asset previews in Data Marketplace. This allows users to get immediate visibility on the data quality score from Data Marketplace. (idea #DCC-I-891)
Assessments
- API access to Collibra Assessments is now available. You can use the new Assessments REST API to retrieve an assessment, start an assessment, and get lists of both assessments and templates. Additionally, you can call the Assessments API within a workflow to retrieve or start an assessment. This allows you to extract data from Collibra Assessments for further data processing, reporting, trend analysis, making informed data governance decisions, or reintegrating the data into Collibra assets. (idea #DCC-I-438)
Protect
- Protect now supports the enforcement of data protection standards and data access rules on Databricks.
Note Protect for Databricks uses Databricks functions that are in the public preview stage. Protect for Databricks will be generally available (GA) whenever those Databricks functions reach the GA stage.
- You can now extend the data protection capabilities of Protect by adding your own data protection methods using custom masking functions. Custom masking functions can be defined in a data source and applied to the columns in the data source via data protection standards and data access rules. You can use these functions for format-preserving masking or for other advanced masking.
Security
- You can use your own encryption key to encrypt the virtual disks of your Cloud environment. With this, you have full control over your data and can revoke access to that data at any time.
Enhancements
Data Catalog
- If the name of the out-of-the-box resource role Owner has been changed, the new name is now used in the "Register Data Source" dialog. (idea #DCC-I-2322)
- The “Show inherited responsibilities” checkbox is now available in the views on Catalog index pages, such as "Reports" and "Data Sources". For detailed information and the required setup, go to Show responsibilities in an asset table. (ticket #124292, 124682) (idea #DGCPLAT-I-1435)
- When you define S3 crawlers on Edge via API, you can now specify the level from which tables have to be created during the S3 integration. By default, tables are created from the top level, level 1. You can specify a number if you want to create tables starting from another level, such as 2 or 3.
- Catalog views no longer take an unusual amount of time to load when expanding the hierarchies. (ticket #116828, 117222).
Data Lineage and BI integrations
Note Data Lineage is a cloud-only feature.
- When integrating Power BI, tables and columns are correctly identified for the Power Query M function AzureAnalysis.Database. (ticket #121494)
- You can now configure a proxy connection when integrating Power BI via the lineage harvester.Important TLS proxy connection is not supported.
- When you create technical lineage for SQL Server Integration Services (SSIS), CollibraData Lineage creates a summary of all connection definition errors in a analyze error in the Sources tab page.
- Technical lineage for DataStage now supports parameter resolution and inheritance for sequence jobs. With this enhancement, end-to-end lineage on one canvas will be increased.
- Technical lineage for DataStage now supports runtime parameters in parameter set files. To use this enhancement, you must export DataStage project files by using the with executable option.
Data Governance
- Asset pages now support printing in A5 paper format. (ticket #119398)
Data Marketplace
- The shopping basket feature has been enhanced.
(idea #DCC-I-219, DCC-I-2329, CAT-I-282)- You can now show or hide the shopping basket in your entire environment.
The existing option to show or hide the shopping basket in Data Marketplace now also impacts the visibility of the shopping basket in other areas of your environment, such as Data Catalog.- To make sure that the shopping basket options can be used by customers who are not using Data Marketplace, we moved these options to their own section in the Data Marketplace Settings page.
- The Data Marketplace Settings now include an overview page of all the settings and useful links to documentation, university courses, and demos.
- You can now also define which asset types you want to allow in the shopping basket. By default, the shopping basket allows for reports and data sets.
Important When you allow for extra asset types, the Request Assets Access workflow must be updated accordingly.
- You can now show or hide the shopping basket in your entire environment.
- You can improve the search experience in Data Marketplace by tailoring the available filters in the Filters pane. (idea #DCC-I-1580)
You can now:- Change the display name of filters including the out-of-the-box filters.
- Add descriptions to filters, which will appear as tooltips in the Data Marketplace Search Results page.
- Define whether a filter should be expanded or collapsed by default.
To make this possible, we made some changes to the configuration of relation indexes:
- Relation index names are now case insensitive and should not include an
_. - We removed the feature to reset the relation indexes to their default values.
- The actions that appear in the "Need help?" section when no search results are available, are now also displayed if search results are available. You can find the actions at the bottom of the search results page.
- We improved the way rich text attributes such as descriptions and definitions are displayed.
Edge
- Edge is now compatible with the latest versions of k3s and EKS. For more information on any upgrade actions, go to the Edge Compatibility table.
- K3s: Edge now supports k3s version 1.22, 1.24, 1.25, 1.26 and 1.27.
- Reinstall your Edge site to upgrade to the latest version of k3s.
- EKS: Edge supports EKS versions 1.23, 1.24, 1.25, 1.26, and 1.27 for existing Edge sites. New Edge site installations must be installed using EKS 1.27.
- You must follow these instructions on how to disable Datadog's pod policy in order to successfully upgrade your EKS version from EKS 1.24 or lower to the latest supported version, EKS 1.27.
- K3s: Edge now supports k3s version 1.22, 1.24, 1.25, 1.26 and 1.27.
Browser Extension
- You can now have the Collibra platform URL preconfigured during mass deployment. This feature enables all users to begin using the Browser Extension immediately after opening it for the first time, ensuring a faster onboarding experience. (idea #DCC-I-836)
- When you now click the “Open on Current Page” button in the configuration dialog box, the dialog box closes and the Browser Extension is launched on the current page with the Search box focused, enabling you to use the extension immediately.
Usage Analytics
- You can now view data for a specific period on the Usage Analytics dashboard by using the new Custom option in the Current Period field. (idea #DCC-I-1364)
Security
- The Enhanced CSRF console configuration option is enabled by default for new environments.
- The password complexity console configuration options are preconfigured for new environments to improve security and meet regulatory requirements.
API
- The /users/email/{emailAddress} endpoint of the Collibra Core REST API is deprecated. Use the GET method of the the /users endpoint and provide the email address for the name parameter and EMAIL for the nameSearchFields parameter. (ticket #115146, 115235, 116748)
Miscellaneous
- The gender of the out-of-the-box users, such as the Admin, System, or Workflow users, is now set to Unknown.
- We have updated the license expiration message to account for customers who have already renewed the license or have a multi-year agreement.
Fixes
Data Catalog
- If you defined a Cron expression to schedule the synchronization of a data source via Edge and the defined Cron expression expires, the Configuration page opens showing a notification instead of not loading at all. (ticket #107362)
- Jobserver upgrades will no longer reset the Jobserver Security Configuration to "None". Instead, the configuration is not impacted by the upgrade. (ticket #120119)
- If a user requests sample data via Edge, we no longer validate whether the user has access to the Database asset. We verify only the View permission of the asset for which the user wants to collect the sample data. (ticket #116059)
Data Lineage and BI integrations
Note Data Lineage is a cloud-only feature.
- When integrating Power BI with a Snowflake data source, the “Kind” parameter is now optional in your Power Query Snowflake.Databases function. Previously, if the “Kind” parameter did not follow the database name, an analysis error was produced. (ticket #120494)
- When integrating Power BI, you can now have more than one upstream dataset as the source of other datasets. (ticket #124002, 127341, 127825, 127825, 127877)
- When integrating MicroStrategy, Facts, and Fact Expressions are ingested in Data Catalog as MicroStrategy Data Entity, and MicroStrategy Data Attribute assets, respectively. Relations between Facts and Fact Expressions are now shown on the respective asset pages.
- When integrating MicroStrategy and specifying the Collibra system name, stitching is again achieved as long as the system name you specify in your <source ID> configuration file matches the name of the System asset in Data Catalog. The names are not case-sensitive.
- MicroStrategy Intelligent Cubes are now correctly represented in Collibra as MicroStrategy Data Model assets, and they have the correct relation to their respective BI Report assets. (ticket #121418, 127046)
- When integrating Tableau, hidden dashboards in Tableau now have the "Visible on server" attribute correctly set to “false” on their asset pages in Data Catalog. (ticket #127336)
- When you create technical lineage for Informatica Intelligent Cloud Services, CollibraData Lineage can now parse parameter files that are organized in a flattened structure in the root directory in the ZIP file, rather than in their corresponding sub-directories.
- When you create technical lineage on Edge by using the Shared Storage connection, a file can now contain whitespace. (ticket #126131)
- With this enhancement, you can get more end-to-end lineage when you create technical lineage for Snowflake. Previously, assets containing lowercase characters were not stitched correctly, and duplicated tables were created. (ticket #120837, 124052, 125740)
- When you create technical lineage for DataStage, there is less meaningless ORCHESTRATE errors now.
- Technical lineage for DataStage can now parse Teradata connector correctly.
Data Governance
- The asset workflow form components no longer suggest assets based on full name (only on display name).
- We have clarified in File API documentation that file IDs can be temporary and therefore trying to get a file that used to exist by it's former ID and will return no result.
- We have fixed an issue when deleting a community with sub-communities that had views on at least one sub-community. (ticket #122147)
- When you create an new scoped assignment on the assettype page, the list of characteristics of that new scoped assignment is now automatically opened. This fix applies to releases in preview only.
Data Marketplace
- The Data Marketplace search can now correctly retrieve assets with special characters in the attribute values. (ticket #123470, 123952)
Note We don't support the
^character in a search, an uneven number of/in a search, starting a search with a colon:, or not closing the quotes or brackets",',],},). - We now can display definition attributes in the Data Marketplace search results and the asset preview. If an asset has both a Description and a Definition attribute, we will show the definition.
- When you create a new custom relation index, the maximum number of characters is now 50. Before this change, no character limit was in place.
- If a user doesn't define a language in the Collibra user profile, Data Marketplace now shows the English language, instead of showing an error message. (ticket #120827)
- We fixed an issue that caused an error in Data Marketplace when the number of users in the Created By filter exceeded 65k. (ticket #124373)
Data Privacy
- The Log Potential Security Breach workflow no longer fails if you assign a group to the Community Manager responsibility. (ticket #114307, 111944)
Workflows
- Defining SetActivityStreamListener at the process level now correctly shows the "View relevant changes" section on the workflow task.
- A user task using the same form reference as a previous task no longer displays the default values if they were cleared on the previous user task.
- You can again delete workflows if you have a role with the Workflow Administration global permission.
Workflow Designer
- Signing out of the Workflow Designer no longer causes an internal server error.
- A workflow containing a call activity no longer prevents you from deploying the workflow from the Workflow Designer. (ticket #117953, 119303, 121795, 123706, 124697, 124780, 124856, 127862)
Edge
- We fixed an issue where Edge site backups failed to restore during reinstallation. With this fix, Edge site backups are successfully restored.
- We fixed an issue where large Edge secrets could not be restored when reinstalling an Edge site. With this fix, all secrets are restored when you reinstall an Edge site. (ticket #117349, 121723)
- The Logging section, which includes the Debug and Log level fields, is now available for all types of connections. Previously, this section was only available for JDBC connections.
Search
- When you filter by a tag that contains a comma, the comma is now treated as a part of the tag, instead of as a separator. For example, when you filter by the tag "Diversity, Equity, and Inclusion," the entire tag is treated as a single tag, instead of as separate tags ("Diversity," "Equity," and "Inclusion").
Browser Extension
- The "Search in Collibra" option works again.
- Auto-matching now works on a dashboard whose name contains a special character.
Security
- An HTTP header with the Accept-Language set to * no longer causes a warning in the Collibra logs and the default language is used.
- Environments where the Limited CSRF console configuration option is enabled but the Enhanced CSRF option is not no longer prevent new users for signing in for the first time. (ticket #120568)
- A check preventing the creation of case insensitive user name duplicates no longer prevents you from changing the case of an existing user name. (ticket #121158)
- Fixed a path traversal security vulnerability.
- Fixed an SQL injection security vulnerability.
API
- The domainTypeId query parameter will be deprecated in the REST API documentation, although a similar parameter will still remain as a path parameter.
- Requests to create a new relation type using the POST /relationTypes API, that pass a UUID of an already existing relation type, will now be rejected.
Miscellaneous
- When you apply multiple filters using the "does not contain" operator, the filters now operate under a logical OR operation instead of an AND operation. As a result, an entry will be excluded from the results if it matches at least one of the specified filters, instead of requiring the entry to match all of the filters. (ticket #117258)
- Filtering on inherited responsibilities no longer slows the performance. (ticket #108064)
-
If you go to the Homepage that was last edited by a user who is now deleted, the Homepage no longer crashes. (ticket #123828, 127095)
-
The Business Qualifier drop-down list in a business qualifier diagram no longer shows assets to which you do not have the permission to view. (ticket #115379)
Features
in preview
A public preview is an upcoming feature or product that is made available to all customers before it is fully ready for general availability so it can be tested and evaluated early. Learn more
Data Catalog
Unified Data Classification is a new method to classify your data via Edge. (idea #DCC-I-2411, DCC-I-1965)
- This new method allows for the creation of custom data classes based on classification rules that can be used and detected by an automatic data classification process on Edge.
- Its classification rules can be regular expressions or lists of values.
- The method includes a list of out-of-the-box data classes.
- The data classification capability is independent from the profiling capability. This means you can start the classification of a table or column separately and that the profiling activities take less time.
Important Currently, no data classification migration is available. If you use data classification already and you activate the new data classification method, your current data classes and classifications are no longer available.
Data Marketplace
-
We are launching Data Similarity in Data Marketplace. Data Similarity allows you to see similar Table assets inside of a Table asset preview. This will help your data analysts to navigate the data landscape and find alternative data, and will therefore reduce their time to value.
When you enable the Data Similarity feature (in preview), extra AI algorithms, which use privacy enhancing technologies, will run during data profiling. These algorithms will detect similar data across all data sources that you have profiled after you enabled this feature.Note Data Similarity is currently available for Table assets.
Tip For now, similar data is available in Data Marketplace only. If you are interested in expanding on this feature to help you discover and reduce data redundancies, and reduce storage and risk assessment costs, add your vote in the Ideation platform.
- We have improved on the Collections feature based on your feedback. You can now:
- Search through your collections.
- Sort collections by creation date, last modified data, and name.
- Sort assets in a collection by time added and name.
- Filter assets based on their asset type.
Search
- You can now use the search function without interruption even when the search index is rebuilding, by enabling the new Uninterrupted Search configuration setting.
Note
- This setting is disabled by default.
- When the setting is enabled, the search results may not be current until the rebuild is complete.
- If you change the setting, you need to rebuild the search index for the change to take effect.
Knowledge Graph API
- Using the OR operator for multiple collection filters no longer returns no results.
Collibra maintenance updates
Collibra 2023.11.1
- We fixed an issue where connections did not display on the Connections tab of an Edge site. (ticket #129320, 129324, 129325)
- It takes less time to open the “Register content” page to register a data source via Edge. (ticket #120020)
- We fixed an issue with the relevancy score of certain search results. (ticket #125074)
- We fixed an issue that caused an error in Data Marketplace when the number of organizations exceeded 65k. (ticket #127227)
- Workflows with scripts that use an array of strings where an array of UUIDs is expected continue to exceptionally work. (ticket #129282, 129299, 129389, 129391, 129392, 129413, 129443, 129464, 129499, 129715, 129722)
Important Important: If you have such workflows, update your scripts so that they follow the API specification. To convert a String to UUID, use string2Uuid(<your_string>).
Collibra 2023.11.2
- When creating a hierarchy view, the co-role of a relation is now shown in the Configure Hierarchy dialog box. (ticket #129396, 129542, 129551, 129582, 129631, 129648, 129671, 129862, 130084)
Collibra 2023.11.3
- The /csv endpoints of the Output Module resource of the REST Core API v2 display again attributes with new line characters on the same line in the output file. (ticket #129300, 130386, 130457, 130559, 130790)
- A cache issue no longer causes out of memory errors that can potentially slow down your environment. (ticket #129688, 130732, 130761, 130923)
Collibra 2023.11.4
- We have fixed a regression that made it impossible to migrate views between environments. (ticket #129440, 129624, 129822, 129903, 129937, 129962, 130065, 130162, 130166, 130209, 130303, 130541, 130561, 131312)
Collibra 2023.11.5
- Groovy scrips containing variables named "writer" no longer prevent you from deploying the workflow. (ticket #132192, 132282)
- Collecting application usage data is now disabled by default for Collibra Platform Self-Hosted environments.
Collibra 2023.11.6
- Redeploying a workflow with a sub-process when a user task of the sub-process is still active no longer prevents you from completing the user task. (ticket #134284, 137412)
Data Lineage 2023.11.1
- When integrating MicroStrategy:
- Intelligent Cubes are now ingested even if some properties are missing in the API call. (ticket # 127046)
- MicroStrategy Data Model assets have a relation to MicroStrategy Folder assets.
Data Lineage 2023.11.3
- When integrating Power BI, the Owner in Source attribute on Power BI Dataflow asset pages now correctly reflects the real owner of the dataflow in Power BI.
- When integrating Looker and filtering on a folder, metadata processing no longer fails if there are Looks that don’t pass the filter, but are related to a Tile that is ingested. In this case, processing continues and the following warning message is included in the source code in the Sources tab page: “Look <Look ID> was not found in folder <folder ID>.” (ticket # 125911)
Data Lineage 2023.11.4
- When integrating Power BI:
- If a data set in Power BI has a description, the description is now ingested and shown on the asset page of the corresponding Power BI Data Model asset.
- If reports are ingested from multiple tenants, the report URLs on the asset pages in Collibra now include the tenant ID. Previously, without this differentiation in the URL, the links accessed the page of the same, default tenant, even if a given report was not located in that tenant. (ticket #129557)
Data Lineage 2023.12.2
- When creating technical lineage via Edge, you can now use a man-in-the-middle (MITM) proxy between and the Collibra Data Lineage service instances. For details on which data sources support the use of proxies, see Create a technical lineage via Edge and select your data source. (ticket #120946, 121788, 124013, 124390, 126170, 128684, 130493, 132075)
Data Lineage 2023.12.3
- When integrating Tableau, the relation “Report uses / used in Data Attribute” is now created between Tableau Dashboard assets and Tableau Data Attribute assets.
- When when using the batch definition method (in preview) to create a custom technical lineage, table-level lineage is now correctly shown, even if you don't include the source code files. (ticket #129835)
- When ingesting Snowflake data sources, the Collibra Data Lineage service instances now support SELECT… EXCLUDE statements. (ticket #131342)
Data Lineage 2023.12.3.1
- When attempting to create a custom technical lineage via the new batch definition method (in preview), if your JSON files don't conform to the required JSON schema, an error message is now included in the log file.
Edge and Data Lineage updates
These updates contain security and bug fixes for Data Lineage, Edge sites and their capabilities. These releases may be planned outside the regular monthly or quarterly release. You'll see the fix versions if you are manually upgrading an Edge site or reviewing logs.
Edge 2023.11.1
- We have improved security for Data Classification via Edge.
- To improve profiling performance of the profiling capability on Edge, we no longer perform a dedicated query for a row count when you profile all rows in a table. (ticket #121338)
- The Databricks Unity Catalog Synchronization capability has been updated to use a generated external ID for assets based on the full name. Previously, the external ID was set to the asset Databricks ID.
This change was made to accommodate the use case where an asset, for example a table, was deleted and recreated with the same name in Databricks Unity Catalog. This resulted in multiple Databricks IDs for the same asset which caused the integration to fail.
(ticket #126352, 126606, 126608, 127194, 128382)Important If you re-synchronize integrated data sources with the new version of the Databricks Unity Catalog Synchronization capability, you will get the following error message:
A mapping for the external system id '{your-system-asset-id}' and resource '{resource-id}' already exists.To resolve this, follow the instructions in the knowledge article.
Edge 2023.11.2
- We have improved the security of Edge.
Edge 2023.11.3
- If you run the migrate option for the Databricks Unity Catalog Synchronization capability to use a generated external ID, the complex relations mappings are now removed as well. For information on the migration option, go to https://support.collibra.com/hc/en-us/articles/18702127233815
- We have improved the security of Data Quality & Observability Classic.
Edge 2023.11.4
- We have improved the security of Data Quality & Observability Classic
Edge 2023.11.5
- We have improved the security of Data Lineage.
- We have improved the security of Data Classification.
- We have improved the security of S3 synchronization.
Edge 2023.11.6
- We have improved the security of Data Classification.
- We have improved the security of Catalog JDBC ingestion and JDBC Profiling.
- When creating technical lineage via Edge, you can now use a man-in-the-middle (MITM) proxy between Edge and the Collibra Data Lineage service instances. For details on which data sources support the use of proxies, see Create a technical lineage via Edge and select your data source. (ticket #120946, 121788, 124013, 124390, 126170, 128684, 130493)
Edge 2023.11.7
- We have improved the security of Data Classification, Data Lineage, ADLS synchronization, and Databricks Unity Catalog synchronization.
Edge 2023.11.8
- We have improved the security of Collibra Protect, Catalog JDBC ingestion, and JDBC Profiling.
- The GCS integration now also stores the bucket location in the Location attribute of the GCS bucket asset.
- The ADLS-Purview integration now supports the integration of more than 120 columns from Purview. (ticket #130324)
Edge 2023.11.9
- We have improved the security of ADLS synchronization, Collibra Protect, Databricks Unity Catalog synchronization, GCS synchronization and S3 synchronization.
Edge 2023.11.10
- We have improved the security of Data Classification.
Edge 2023.11.11
- When integrating Tableau, if a password is not provided for the Tableau connection, the integration fails, as expected. The error message now correctly identifies the issue. (ticket #130856)
- Fixed an issue that was resulting in a "result of string concatenation is too long" error, when ingesting Oracle data sources. (ticket #134226, 134613)
- Optimized memory profile when harvesting database models during the integrating of JDBC data sources. This helps reduce the chances of Out Of Memory (OOM) errors. (ticket #130766)
- When you create technical lineage for Matillion, technical lineage via Edge now supports passing a certificate to the data source for authentication.
- Collibra Data Lineage now supports lineage from SAP HANA calculated views in SAP HANA Cloud and Advanced data sources.
- Previously, Collibra used dbt Labs' Discovery API job schema, which gets information from the applied state based on the most successful job runs. This method had 2 issues:
- It depended on access to the Admin API, which not all dbt Cloud customers had.
- It does not include all jobs compile projects or produce models, for example, a macro command.