Release 2023.05

• Release information
• Highlights
• Metamodel changes
• New features
• Enhancements
• Beta features
• Fixes
• Collibra maintenance updates
• Edge updates

Release information

• Release date of Collibra 2023.05.0: April 30, 2023
  • Upgrade non-production environments: April 30, 2023
  • Upgrade production environments: June 4, 2023
• Release date of Collibra 2023.05.1: June 4, 2023
• Release date of Collibra 2023.05.2: On-demand only
• Release date of Collibra 2023.05.3: July 28, 2023 (GovCloud environments only)
• Release date of Edge 2023.05.1: May 21, 2023
• Release date of Edge 2023.05.2: June 4, 2023
• Release date of Edge 2023.05.3: July 2, 2023
• Release date of Edge 2023.05.4: July 9, 2023
• Release date of Edge 2023.05.5: July 16, 2023
• Release date of Edge 2023.05.6: July 30, 2023
• Relevant Jobserver version: 2023.05.0-55

Highlights

• Edge is now FedRAMP authorized. FedRAMP, or the Federal Risk and Authorization Management Program, is a security assessment and authorization standard for U.S. federal agencies. This means that Edge meets the security standards for FedRAMP authorization. If FedRAMP is a requirement for your organization, reach out to your Collibra representative and review the FedRAMP network requirements for Edge installations.

  Edge sites now require access to Datadog's latest release. To do so, add https://7-42-0-app.agent.datadoghq.com to the allowlist of your network. Datadog logs and monitors Edge site performance and infrastructure health, which is required for troubleshooting and support.

• Technical lineage via Edge is now generally available for the following supported data sources:

  • All supported ETL data sources, except Azure Data Factory. Azure Data Factory is not supported yet.
  • All JDBC data sources.
  • Power BI and Tableau.

• Collibra Data Lineage support for creating technical lineage for Azure Data Factory by using the lineage harvester is now generally available.

• When you integrate Power BI, reports and dashboards that are part of an app in Power BI are ingested as Power BI Dashboard and Power BI Report assets, respectively. The URLs on these asset pages now correctly link to the corresponding dashboards and reports in the Power BI app.

• Databricks Unity Catalog file system integration via Edge is now generally available. This integration allows for the registration of Databricks Unity Catalog as a data source in Collibra and the synchronization of the metadata. After the synchronization, the files and directories of the Databricks Unity Catalog are represented in Collibra by specific asset types, retaining the original names.

• The new import wizard now defaults to the current domain if a domain is not mapped.
  

• You can now deploy workflows from the Workflow Designer to the corresponding Collibra environment with the push of a button. This functionality replaces the need to export, download and upload a ZIP file and is available with the upgrade of production environments.

• To simplify the navigation for data consumers, we have added a specific global permission and global role for Data Marketplace. This allows you give data consumers access to only Data Marketplace.
  Users with the Data Marketplace global role or global permission can access Data Marketplace from the Applications menu. Users with the Catalog role can access Data Marketplace and Catalog from the Applications menu.

• Data Marketplace administrators can now define a primary action per asset type. By customizing the primary action based on asset type in the Main Asset Preview Actions tab, you can ensure the most appropriate workflow is available to a data consumer from the asset preview.

Metamodel Changes

• In preparation of the Dataplex integration, following asset types have been added to the metamodel: GCP Project, Dataplex Lake, and Dataplex Zone. This metamodel can change based on feedback from the upcoming Dataplex integration Beta testing.
  These types are added for internal purposes only and should not yet be used outside the Beta testing.
• As part of the migration process from the Java Core API v1 to v2, a number of v1 methods are now deprecated or removed.See what is changing

  Interface v1	Method v1	Change	Interface v2	Method v2
  RightsComponent	isPermittedOnRepresentation	Drop
  RightsComponent	isPermittedOnVocabulary	Drop
  RightsComponent	isPermittedOnCommunity	Drop
  RightsComponent	isComplete	Drop
  RightsComponent	getCurrentUserRights	Drop
  AttributeComponent	addNote	Removed
  AttributeComponent	getExamplesForRepresentation	Removed
  AttributeComponent	getNotesForRepresentation	Removed
  AttributeComponent	getSingleValueListAttribute	Removed
  HyperlinkComponent	findMatchedExpressions	Removed
  HyperlinkComponent	findAutomaticallyLinkedTerms	Removed
  HyperlinkComponent	findManuallyLinkedTerms	Removed
  HyperlinkComponent	doFullRecomputation	Removed
  HyperlinkComponent	cleanHyperlinks	Removed
  HyperlinkComponent	isHyperlinkingEnabled	Removed
  HyperlinkComponent	getHyperlinkedLongExpression	Removed
  RepresentationComponent	remove	Removed
  IssueComponent	createIssueFilter	Removed
  RightsComponent	removeRole	Removed
  RightsComponent	getMemberCount	Removed
  AttributeComponent	changeDateAttributeValue	Removed
  RightsComponent	addRole	Removed
  RelationComponent	findRelationsByType	Removed
  IssueComponent	addIssue	Removed
  ConceptTypeComponent	findAllParentAssetTypes	Removed
  UserComponent	resetPasswordByEmail	Removed
  UserComponent	getCurrentLocale	Convert	User	getLanguage
  UserComponent	getUsersCount	Convert	UserApi	findUsers​(FindUsersRequest)
  UserComponent	isPublicRegistrationAllowed	Drop
  UserComponent	isGuestAccessAllowed	Drop
  UserComponent	setAPIUser	Drop
  UserComponent	changePassword	Drop
  UserComponent	resetPassword	Drop
  UserComponent	requestResetByEmail	Drop
  UserComponent	requestReset	Drop
  UserComponent	activateAccount	Drop
  UserComponent	resendActivationEmail	Drop
  UserComponent	checkActivationCode	Drop
  UserComponent	getAvatarAttachment	Drop
  UserComponent	validatePassword	Drop
  UserComponent	sendActivationEmail	Drop
  UserComponent	isPasswordExpired	Drop
  UserComponent	sendEmailNotifications	Drop
  UserComponent	isAuthenticatedSAMLUser	Drop
  ComplexRelationTypeComponent	getAllComplexRelationTypes	Removed

  For a complete list of changes that impact Java API v1 methods, see the Java API v1 to v2 mapping on the Developer Portal.

New features

Data Catalog

• Data Attributes connected to the column via relation "Data Attribute represents Column" are now included in the Business Context section on the Summary tab of the Column asset page. Data Attributes can give more context to a column from a business perspective. The connected Data Attributes are shown before the other connected Business assets, such as Business Terms and Data Concepts.
• Data Categories connected to a column are no longer included in the Business Context section. They have been moved to a dedicated section, Privacy Context in the Summary tab in the Column asset page. The Privacy Context section makes it easier to understand the privacy and security controls linked to the data. The section displays the Data Categories connected directly and indirectly to the column via the following possible relation: 
  • Column → Data Category via relation "Column represented by Business Asset (Data Category)".
• The Privacy Context section has been added to the Summary tab in the Table and Database View asset pages. This Privacy Context section makes it easier to understand the privacy and security controls linked to the data. The section displays the Data Categories connected to this table via the following possible relations:
  • Table → Data Category via relation "Table represented by Business Asset (Data Category)".
  • Table → Column → Data Category via relations "Table contains Column" and "Column represented by Business Asset (Data Category)".
• Databricks Unity Catalog file system integration via Edge is now generally available. This integration allows for the registration of Databricks Unity Catalog as a data source in Collibra and the synchronization of the metadata. After the synchronization, the files and directories of the Databricks Unity Catalog are represented in Collibra by specific asset types, retaining the original names.

Data Lineage and BI integrations

Note Data Lineage is a cloud-only feature.

• When you create technical lineage for Snowflake by using the SQL-API ingestion method,
  • QUERY_TAG values are now shown in the transformation window for lineage queries.
  • The lineage harvester optimized the results of the columns_joined query. Previously, the view definition would be saved for each column of a view. Now, a view definition is only saved once. This enhancement results in faster processing of lineage for your Snowflake database that has views with many columns.
• When you create technical lineage for Informatica PowerCenter, an error message is logged if any of the following issues occur:
  • A parameter file cannot be parsed.
  • A workflow XML file cannot be parsed or is invalid.
• The new MicroStrategy integration method, via the lineage harvester, is now in beta. The new integration method allows for technical lineage, supports the latest MicroStrategy APIs, and is no longer dependent on a direct connection to the repository.
• On the Sources tab page in the technical lineage graph, you can sort by each column in ascending or descending order. These columns include Scanner type, Success rate, Done, Parsing Error, Analyze Error, and Last sync time.

Data Marketplace

• Data Marketplace administrators can now define a primary action per asset type. By customizing the primary action based on asset type in the Main Asset Preview Actions tab, you can ensure the most appropriate workflow is available to a data consumer from the asset preview.
• On a daily basis, we check the Data Marketplace configuration with the configuration in DIC for inconsistencies, and automatically update the configuration if needed. If, for example, a user group has been deleted in DIC, the Data Marketplace configuration is automatically updated to reflect that change.
  Administrators can now define a user group to receive an email notification about the changes. The email provides an overview of the removed configuration and links to possible Data Marketplace configuration areas to verify.
• To simplify the navigation for data consumers, we have added a specific global permission and global role for Data Marketplace. This allows you give data consumers access to only Data Marketplace.
  Users with the Data Marketplace global role or global permission can access Data Marketplace from the Applications menu. Users with the Catalog role can access Data Marketplace and Catalog from the Applications menu.
  Important 

  • What is the impact?
    • Existing users: To make the transition to the new role and permission easier, existing Data Marketplace users will automatically receive the global Data Marketplace role during the upgrade to 2023.05. You can then refine the permissions if needed.
    • New users: New users need to be assigned to the Data Marketplace global role or global permission manually.
  • Classification still requires the Catalog permission. Users with only Data Marketplace permissions will not see the Classification tags in the Auto-generated abstract or the Column browser.

Data Governance

• We have added more details in the relation header of the exported file, so when you import using this new format, auto-mapping is more accurate.
• A new human-readable unique identifier, called the Public ID, is available via API to Asset Types, Attribute Types, Relation Types, Complex Relation Types, Domain Types, and Scopes. This is foundational work that is currently not available through the UI.
• It is now possible to find out which other part of the Meta model the Asset Types, Domain Types, Statuses, DQ Rules, Complex relation Types, Relation Types, and Attribute Types are being used.

Data Privacy

• Collibra Protect now supports the enforcement of data protection standards and data access rules on AWS Lake Formation.

Enhancements

Data Catalog

• If you are using a CDATA driver older 21.0.7970 (release in October 2021) in Jobserver, you must update the driver to a higher version to be compatible with Java 17.
• You can now define up to five synchronization rules per schema allowing you to specify where data needs to be added to in Collibra. This is especially useful when registering large data sources or data sources covering multiple business areas via Edge.
• The Profiling capability has been enhanced to avoid out-of-memory issues during Data Classification on Edge. (ticket #87189)
• When you integrate a GCS file system, you can now specify the project IDs for Dataplex services that you want to search through to ingest the schema information. This way you can search through a wide range of Dataplex locations.
• The GCS integration is now available via the Register a Data Source link in the Register with Edge section of the Catalog Create dialog box.
  See what this looks like

  
  

Data Lineage and BI integrations

Note Data Lineage is a cloud-only feature.

• When you integrate Tableau, the lineage harvester now automatically connects to the REST API version that matches your Tableau Server or Tableau Online environment.
• Improved Tableau metadata synchronization process. (ticket #102814)
• If a data set or report in Power BI is certified, the corresponding Power BI Data Model and Power BI Report assets in Collibra are now automatically certified.
• When integrating Power BI, the full names of Power BI capacities now include their unique identifiers. This helps to distinguish two capacities with the same name. Upon the first synchronization after this fix, if you use only one Power BI tenant, the Shared Capacity asset is deleted and recreated with the new naming format. If you have multiple Power BI tenants, a Shared Capacity asset with the new naming format is created for each tenant. (ticket #110239)
• Manually synchronizing Power BI and Tableau technical lineage via Edge is now done via the Data Catalog Register Content page.
• When ingesting Spark SQL data sources, the Collibra Data Lineage service instances now benefit from the following parsing enhancements:
  • CREATE VIEW to support TBLPROPERTIES
  • SELECT allowed as column name
  • TABLE allowed as column name
  • CREATE TABLE to support the USING clause
  • CREATE TBALE to support the OPTIONS clause

• The "list-sources" command is enhanced to:

  • Indicate how each data source was ingested, by using the lineage harvester or technical lineage via Edge.
  • List the useSystemName value to each data source.
  • List up to 500 data sources. With this enhancement, you can determine which page to to be displayed and also the number of data sources to be listed on certain pages.
• When you run the full-sync command to refresh your technical lineage, if you change the useSystemName value for the same data source, the new value takes effect. Previously, if the useSystemName values for the same data source differed between the current and previous synchronization processes, the synchronization process would fail with an error indicating the differences. To set the useSystemName values:
  • If you use the lineage harvester, use the useCollibraSystemName property in the lineage harvester configuration file.
  • If you use technical lineage via Edge, use the Collibra system name setting when you enable technical lineage via Edge.

Data Marketplace

• Preconfigured filters are now also visible below the Search field in the Data Marketplace landing page, and not only as tabs in the Search results page. This makes it much easier to access data that you frequently need.
• The quick links in the landing page have been moved to another location in the page. This change ensures the Search field gets the focus.
• The user experience in the Discovery page has been enhanced based on customer feedback.
• You can now search for communities and domains in the Advanced Organization filter.
• The number of search results is now displayed below the search results header. This allows for more space to display preconfigured filters.
• The Clear all filters option has been removed from the Filters pane. To remove all applied filters click the All tab.
• The way to create preconfigured filters (filter tabs) has been optimized. You now create a new filter tab via the Save as New Filter button.
  See what it looks like

  
  Important 

  • With this version, the interface to save a preconfigured or personal filter looks very similar. This might be confusing if you had the Data Marketplace beta feature enabled before.
  • You can no longer define the order of the filter tabs.
    
• As an admin, you can now update the out-of-the-box preconfigured filters, except for the All filter.

Data Governance

• The new import wizard now defaults to the current domain if a domain is not mapped. If the asset ID is mapped, the domain does not default.
• To improve security, if you have a role with the "User Administration" permission, you can no longer modify users or groups that have a role with the "System administration" permission. (ticket #102034)

Assessments

• If you are the owner of an assessment or an administrator, you can now edit the permissions of the assessment regardless of its status. This is applicable even if the assessment was already submitted.
  Note The owner can edit the view permissions and the assignees of the assessment, whereas an administrator can additionally edit the ownership of the assessment.

Insights Data Access and Usage Analytics

Note Insights Data Access and Usage Analytics are cloud-only features.

• You can now provide a Google Analytics 4 tracking ID for capturing web analytics for your Collibra environment for Usage Analytics. This tracking ID is used to have the code snippet embedded on the page to capture default Google Analytics 4 events that occur, such as page visits and form submission.
  Important On July 1, 2023, Google will no longer process new hits from Google Analytics 3 (Universal Analytics) properties. Please be sure to create a Google Analytics 4 tracking ID and work with your representative to have this updated before the deprecation date. For more information on Google’s plans for Universal Analytics deprecation, go to Analytics Help.

Collibra Console

• A list of restricted content types has been added to the Upload configuration in Collibra Console.

Security

• We now use scram-sha-256 to store PostgreSQL users' passwords in a cryptographically hashed form. The passwords are now better protected against brute-force attacks.

Miscellaneous

• The "Rebuild hyperlinks after import" field is no longer needed due to the asynchronous indexing feature. This field is now removed from the Collibra settings.

Fixes

Data Catalog

• For columns without data, the Profiling basic statistics are no longer filled with the maximum Integer value. (ticket #105348)
• The synchronization status in the Database asset Configuration tab now shows the proper "Failed" status when the process fails on the Edge side. (ticket #81921, 87353, 100366, 102089, 108109)
• We have reduced the attempts of submitting Profiling results to Catalog via Edge to two. (ticket #105661)
• We no longer fail to profile tables or create sample data for tables in Databricks via Jobserver, if the ratio of the 'rows to profile' to the 'total row count' is smaller than 0.0005, for example (500 / 1,000,000). (ticket #100873, 101777)
• Modifications to the "Business Context" view on Data Set and Table asset pages no longer causes this view to disappear from the UI. (ticket #101412, 103984, 104862)
• When you register a data source via Edge, empty column and table descriptions are now registered correctly as Null values. This means that you can filter for those assets using the attribute does not exists clause. (ticket #101412, 103984, 104862)
• To ensure Edge is not blocked by IP restrictions when collecting sample data, we changed the route of the requests. (ticket #108298)
• To view sample data via Edge, you no longer require Edge permissions. (ticket #100873, 105423, 107134)
• We have increased the number of times we retry to connect to the Tableau Metadata API to prevent synchronization failures due to network request rate limits. (ticket #102984)
• You can now create a technical lineage via Edge for SAP HANA data sources. Previous attempts failed due to SQLException during metadata harvesting.

Data Marketplace

• Data Marketplace no longer displays duplicate recommendations in the Discovery page.
• Data Marketplace no longer removes double quotes from the Search field. (ticket #107411, 109326)
• Data Marketplace no longer shows rejected data classifications for columns. (ticket #105898)
• For an asset, the number of views is now the same in the Data Marketplace landing page and the Discovery page.
• For relations with a full name that is too large to display, we now show the full name in a tooltip.
• The auto-generated abstract now displays all columns with primary key information instead of only one.
• To show columns in the asset preview, we now use pagination instead of a long list.
• Updating a relation index in Data Marketplace now triggers a relation reindex instead of a full reindex. (ticket #106395)
• You can again check the reindexing of relations activity via the Activities page.
• We show avatars again on the Responsibilities tab in asset previews because we now prevent loading too many avatars.

Data Lineage and BI integrations

Note Data Lineage is a cloud-only feature.

• When ingesting Oracle data sources, SQL queries to extract views no longer include views for which the owner has a user name that start with “APEX”.
• When integrating one of the supported BI tools, if the synchronization processing fails because of a duplicate data source, you now receive a more helpful error message, which includes advice for resolving the issue.
• When integrating Tableau:
  • Filtering on sub-projects no longer results in FOREIGN KEY constraint errors. (ticket #112049)
  • Custom SQL is now successfully processed when Tableau object names contain quotes. (ticket #111164)
• When integrating Power BI, if you use the Databricks.Query query without specifying the database name, the database name in the technical lineage is “Default”. (ticket #112624, 113375)
• When you enter the list-sources command, the sources are listed successfully now without timeout. (ticket #114431)
• In the analysis results attached to the source code or transformation details, line numbers and source code are not overlapping anymore. (ticket #114229, #114243, Ticket #114470)

Data Governance

• When filtering by date, the export now includes all relevant assets including those with localized dates. For example, if you filter by 3 PM (EST) then results for 7 PM (GMT) will also be included.
• You can once again upload larger files, provided they are within the upload limit. Ticket #107743, 109687, 110146, 111179.
• Details of data quality scores in the data quality tab of the asset page now show correctly when there are multiple data quality rules. (#107938)
• Applying a filter on boolean attributes on an asset table no longer results in the table loading slowly. (ticket #105604)
• When adding attributes from a table, the articulation score gets updated correctly again. (ticket #104955)
• Enable statistics once again works as expected for attribute types that support statistics (such as boolean or numeric).
• The hierarchy tree is now updated when a scope that is assigned to at least one community or domain has its name updated.
• You can once again hide or show columns on the Complex Relations Type table.
• We have fixed a bug that was causing the asset name to display twice in edit mode after setting the default asset type in the Asset Type Filter on an asset type assignment page.
• When views are filtered and selected, the table header checkbox now counts all existing views in the table, not only the currently visible ones. (ticket #102398)
• Asset type and domain type descriptions passed through the API are now checked to make sure they don't contain any potentially harmful HTML code.
• Importing tabular formatted data with unused relations no longer causes duplicate entries. (ticket #109456, 110770)
• The new import wizard now defaults to the current domain if a domain is not mapped. If the asset ID is mapped, the domain does not default.

Assessments

• The value in the "Asset type" field of a published assessment template no longer disappears after you save a draft of the template or re-publish the template. (ticket #109068)

Search

• Search results are now shown if you enter a word with a special character, such as sales % or search %, in the global Search field.
• When performing a global search, if you click Show all or Show more in the left pane, more than 16 items are now shown in the dialog box. (ticket #108629)
• The facets that you select when adding a search filter are now saved with the search filter.
• If you change the sort order on a search page, the first page of the search results is now shown.
• If an asset name contains a space and the "allowedCharacters" parameter in the tokenizer settings does not contain double quotation marks, the References tab of the asset page now shows results. (ticket #67907, 82071, 104799)
• If an asset name contains a special character and the "allowedCharacters" parameter in the tokenizer settings contains that special character, the References tab of the asset page now shows results. (ticket #67907, 82071, 104799)

Collibra Console

• You can now correctly use the L-character as special character in the cron expression to create a backup schedule.

Security

• We have fixed a security issue that previously allowed concurrent user sessions from the same user.

Miscellaneous

• Previously, when using the advanced filter on the Assets tab, if you clicked the drop-down list box and then quickly typed the partial name of an attribute, all the applicable attributes were not shown. This issue is now fixed. (ticket #102395)
• The input in the banner message is now sanitized by removing JavaScripts.
• If you delete an asset from the asset page, a message stating that the asset was not found is no longer shown. (ticket #109067, 108628)
• Dashboards containing custom widgets now load properly and do not block switching between dashboards. (ticket #114317, 114577, 114578)

Collibra maintenance updates

Collibra 2023.05.1

• The issue that caused a blank Search page after the upgrade to 2023.05 has been fixed. (ticket #114103)
• Collibra will now redirect successfully (instead of a 404 error message) after you log in when the login is initiated from your IDP (e.g. Azure AD, Okta, etc.). (ticket #113686, 114144, 114161, 114221, 114223, 114253, 114260, 114295, 114319, 114366, 114648, 114817, 114982)
• When you create technical lineage for a data source, if a table in the data source has a schema and file as its parents, the schema is used for stitching to the Schema asset in the Data Catalog. Previously, the stitching process failed and thetechnical lineage was not created.
• The Edge installer package now contains the correct credentials for sites created before 2023.02. (ticket #114123, 114651, 114854, 114870)
• New SSO System Administrators can once again log in to Collibra Data Intelligence Cloud. Ticket #113699,114456, 114543, 114779, 114894, 114979, 115403, 115405, 115910, 116208.
• We increased the network timeout value, to avoid problems for API integrations using the JWT token authentication method. (ticket #116460)

Collibra 2023.05.2

• Reindexing no longer fails when you manually rebuild the search index on a new instance.

Collibra 2023.05.3

• Fixed a cross-site scripting vulnerability.

Edge 2023.05.6

• We have improved the security of Data Classification via Edge.
• The S3 synchronization capability has been updated to prevent any null pointer exception.