Operational trust for AI agents

Updated: August 4, 2026

Operational trust is the term used throughout this documentation to describe a set of related capabilities that give you a live, signal-driven view of how well your deployed AI agents are performing in production. It is not a named feature in the Collibra interface, but a concept that brings together AI monitoring, the agent Quality tab, and the Agents dashboard.

The foundation of operational trust is AI monitoring. Collibra reads LLM judge (scorer) evaluations and token consumption data directly from Databricks and surfaces those signals across the product. Governance signals, such as assessments, lifecycle status, and documentation completeness, reflect what you know about an agent. Operational signals reflect what the agent is actually doing: how often its responses pass quality evaluations, and how much it consumes in token resources.

Important These capabilities are currently available only for agents integrated via the Databricks AI integration. Support for additional AI platforms is planned for future releases.

How operational trust works

Collibra uses the Edge integration to query aggregated trace data directly from Databricks Delta tables. The integration runs on a monitoring schedule that reads agent interaction traces, computes hourly pass rate and token consumption buckets, and pushes the results to Collibra. Because traces carry only the MLflow experiment ID - not a model version ID - operational trust data is anchored at the AI Agent level, aggregated across all versions of that agent.

Note Both LLM judge pass rates and token consumption require agent interaction traces to be synced to a Delta table in Databricks. If traces are not available for an agent, no operational trust data appears in Collibra for that agent.

Signals collected

Collibra collects two types of operational trust signals from Databricks:

LLM judge pass rates. For each LLM judge (scorer) configured on an agent, Collibra records the number of interactions that passed, failed, or returned no assessment. Pass rates are computed and surfaced as daily and 7-day aggregate values, with current-day metrics updated on an hourly basis. For details on how pass rates are calculated, go to AI monitoring for AI agents.
Token consumption. Prompt and completion token counts aggregated from agent interaction traces. Current-day metrics are updated on an hourly basis.

Note Latency and error rate are not included in 2026.09.

Supported Databricks agent types

Operational trust monitoring applies to Databricks agent types that use an MLflow experiment to log interaction traces. Genie spaces are excluded because they do not use MLflow.

Databricks agent type	Operational trust supported
Knowledge Assistant	Yes
Multi-Agent Supervisor	Yes
AI Information Extraction	Yes
Genie space	No

AI Monitor assets and LLM judges

In Collibra, each LLM judge configured on a Databricks agent is represented as an AI Monitor asset. AI Monitor assets are created automatically when the Databricks integration syncs; you do not create them manually. The name of each AI Monitor asset corresponds to the name of the LLM judge in Databricks.

A single AI Agent can have multiple AI Monitor assets linked to it, one for each judge that evaluates the agent's responses. For complete information about AI Monitor assets, go to AI monitoring for AI agents.

Where operational trust data surfaces

Operational trust data appears in three places in AI Command Center.

Surface	What it shows
AI Trust Score, the Operational Health theme	The 7-day average LLM judge pass rate across all monitors linked to the agent version. If no monitoring data is available for this agent for the past 7 days, this theme is excluded from the Trust Score entirely. For complete information, go to AI Trust Score: contributing factors.
Agent Quality tab	A tab on AI Agent and AI Agent Version asset pages showing per-judge daily pass rate trend lines. Use this view to track an individual agent's quality over time and compare performance across judges.
Agents dashboard	A fleet-wide operational dashboard that shows LLM judge pass rate trends and token consumption across all monitored agents in your environment. Use this view to identify which agents or judges need attention. In the AI monitors table, click a monitor name to see which AI Agent Versions are evaluated by that monitor and how each is performing. From there, click an agent version to navigate to its asset page and Quality tab for detailed trend lines.

Surface

What it shows

AI Trust Score, the Operational Health theme

The 7-day average LLM judge pass rate across all monitors linked to the agent version. If no monitoring data is available for this agent for the past 7 days, this theme is excluded from the Trust Score entirely. For complete information, go to AI Trust Score: contributing factors.

Agent Quality tab

A tab on AI Agent and AI Agent Version asset pages showing per-judge daily pass rate trend lines. Use this view to track an individual agent's quality over time and compare performance across judges.

Agents dashboard

A fleet-wide operational dashboard that shows LLM judge pass rate trends and token consumption across all monitored agents in your environment. Use this view to identify which agents or judges need attention.

In the AI monitors table, click a monitor name to see which AI Agent Versions are evaluated by that monitor and how each is performing. From there, click an agent version to navigate to its asset page and Quality tab for detailed trend lines.

Note Operational trust metrics are stored in a time-series optimized format to power the Quality tab, Agents dashboard, and trust score features. They are not accessible through the standard Collibra platform APIs.