About creating and managing a data contract

The data contract is a key component of a data product. It defines the agreement between the data product owner and the data consumers. The data contract specifies the structure, format, service level, quality, and terms of use. It includes a high-level overview of the agreement and a data contract manifest. A data contract manifest is a YAML file that contains the definitions and contents of a data contract. Multiple versions of the manifest can exist. For more information, go to About data contracts.

Available tools to manage data contracts

To manage data contracts and their details, Collibra offers a set of tools. These tools are optimized for the Open Data Contract Standard (ODCS), an open-source framework that describes what is expected in a data contract manifest file.

Collibra provides:

• In-product features, available from the user interface (UI), to create, manage, and view the data contracts and manifest files.

• CLIs and API calls to create, manage, and integrate data contracts and manifest files.

Available actions to manage data contracts

Collibra offers multiple options to manage data contracts. You can:

• Create and update a Data Contract.
• Generate a manifest file based on the Collibra data.
  If you generate a manifest file, the process creates an Open Data Contract Standard manifest file based on Collibra knowledge graph information. For information on the mapping, go to Data Contract mappings.
• Upload a new version of the manifest file.
• Download a manifest file.
• Compare different manifest file versions.
• Set a manifest file as the active one.
• Delete a manifest file.
• Apply a manifest file to Collibra data.
  This action is available from the UI and API.
  If you apply a data contract manifest version to Collibra, the process:
  • Creates and updates relations between Data Product Port and Table assets based on the information in the active manifest version. For information on the mapping, go to Relations mappings.
  • Updates out-of-the-box SLA (Service Level Agreement) attributes on the Data Contract asset in Collibra directly based on the information in the active manifest version. For information on the mapping, go to SLA mappings.

Data Contract mappings

SLA information mapping

The following out-of-the-box SLA (Service Level Agreement) asset attributes are mapped to the manifest file. These attributes are used to generate a manifest file and are updated with manifest data when you apply a manifest file.

When mapping manifest file data to Collibra only manifest file data with titlecase (camelCase) property names are mapped to Data Contract asset attributes. If the manifest property includes a unit, both the value and the unit are combined in the Data Contract attribute.

slaProperties:
	- property: retentionPeriod
	value: 5
	unit: months

Servers information mapping from Collibra to the Data Contract manifest

When you generate a data contract manifest file, Collibra includes a Servers section for the following data sources: Athena, BigQuery, Databricks, MySQL, Oracle, PostgreSQL, Redshift, Snowflake, SQL Server.

The Server information includes fields such as server, type, database, and schema. By default, this information is retrieved from the Edge Connection string if the JDBC data source is registered through Edge. If this isn't available, it is collected from the Database and Schema assets in Collibra.

Generic fields

The manifest properties use the values of the following Collibra asset attributes.

Specific data source fields

The manifest properties use the values of the following Collibra asset attributes.

Relations mapping from the Data Contract manifest to Collibra

When you apply a manifest file to Collibra, Collibra can create and update relations between Data Product Port and Table assets. To do this, Collibra collects data from the manifest file and finds matching assets in Collibra. To find a matching asset, the process determines the schema and database name using the following priority order:

• physicalName
• Custom properties
• Servers section

The process uses the first source in the manifest file in which the required information is found. If the information isn't available in one source, the process moves to the next.

physicalName

If the physicalName contains 3 parts: database, schema, and table, the process uses this information to create the relation. If the physicalName contains only the table name, the process moves to the custom properties to resolve the schema and database name.

Custom properties

If the physicalName doesn't contain the schema or database name, the process uses the custom properties schemaName and databaseName. Both must be defined for the relation to be created. If only one is defined, the relation isn't created.

Servers section

If neither the physicalName nor the custom properties contain the schema or database name, Collibra uses the Servers section, if available.

The following fields are supported:

• Schema: Defined as schema or dataset. Schema takes priority over Dataset.
• Database: Defined as database, catalog, or project. Database takes priority over catalog, which takes priority over project.

Related topics

• Steps overview: Creating and managing data contracts from Collibra
• Creating and maintaining data contracts and their manifest files through CLI and API
• Viewing all data contracts via the Data Contract Registry
• About data products
• Data product asset types and operating model

Helpful resources

• Video: What is a data contract?
• Collibra University: Scaling trust with Data Contracts
• Blog post: Accelerating trusted data product delivery with data contracts