AI monitoring gives you visibility into how well a deployed AI agent's responses meet quality standards. Collibra monitors agents by reading LLM judge (scorer) evaluations from Databricks. For each LLM judge linked to an agent, Collibra collects pass, fail, and no-assessment counts, along with prompt and completion token counts, per hour, from Databricks agent interaction traces. Monitoring results feed into the Operational Health theme of the AI Trust Score for AI Agent Version assets, and are surfaced in the agent Quality tab and the Agents dashboard.
For a conceptual overview of how AI monitoring fits into the broader operational trust framework, go to Operational trust for AI agents.
| Data collected | Description |
|---|---|
| Judge pass count | The number of agent interactions that received a passing verdict from the LLM judge in the hour. |
| Judge fail count | The number of agent interactions that received a failing verdict from the LLM judge in the hour. |
| Judge no-assessment count | The number of agent interactions for which the LLM judge returned no verdict in the hour, for example, because the judge had not yet completed evaluation at the time of sync. |
| Prompt token count | The total number of prompt (input) tokens consumed by agent interactions in the hour. |
| Completion token count | The total number of completion (output) tokens generated by agent interactions in the hour. |
Enabling the feature
AI monitoring must be enabled. In 
Settings > AI Governance, switch on Operational trust for Agent Bricks. For complete steps, go to Set up AI Command Center
How AI monitoring connects to deployments
AI monitoring operates at the deployment level. A specific AI Agent Version is linked to an AI Endpoint through an AI Agent Deployment, which is a complex relation. One or more AI Monitor assets can be linked to each deployment.
An AI Agent's MLflow experiment ID does not vary by version, and Collibra uses that experiment ID as the stable join key for monitoring data. As a result, operational trust data is anchored at the AI Agent level, not captured separately per version. A single AI Agent can have multiple versions, each with its own deployments and endpoints, but they share the same underlying monitoring signals.
| Relation | Public ID | Kind |
|---|---|---|
| AI Agent Version + AI Endpoint → AI Agent Deployment | AIAgentDeployment | Complex relation |
| AI Agent Deployment is monitored by AI Monitor | DeploymentMonitoredByAIMonitor | Explicit relation |
AI Monitor assets
In Collibra, each LLM judge configured on a Databricks agent is represented as an AI Monitor asset. AI Monitor assets are created automatically when the Databricks integration syncs; you do not create or configure them manually. The name of each AI Monitor asset corresponds to the name of the LLM judge in Databricks.
A single AI Agent can have multiple AI Monitor assets, one for each judge that evaluates the agent's responses. AI Monitor assets are created in the domain that you configured for the Databricks AI integration capability, the same domain used for the rest of the integration's synced assets. The following attributes are defined on the AI Monitor asset type.
| Attribute | Description | Public ID |
|---|---|---|
| Data Drift Detection Enabled | Whether data drift detection is enabled on this monitor. | DataDriftDetection |
| Prediction Drift Detection Enabled | Whether prediction drift detection is enabled on this monitor. | PredictionDriftDetection |
| Schedule | The frequency at which monitoring data is synced from Databricks into Collibra. | Schedule |
| Alert Configuration | The threshold conditions that trigger an alert, and the notification channels used to notify stakeholders when a threshold is breached. | AlertConfiguration |
| URL | The URL of the LLM judge in Databricks. | Url |
Connection to the AI Trust Score
Monitoring results feed into the Operational Health theme of the AI Trust Score for AI Agent and AI Agent Version assets. Because monitoring data is anchored at the AI Agent level, an AI Agent and its versions share the same score for this theme. The score for this theme is the 7-day average monitor pass rate: the percentage of monitoring checks that passed over the last 7 calendar days, expressed as a value from 0 to 100.
If no monitoring data is available for this agent for the past 7 days, for example because no interactions have occurred or no LLM judges are configured for it in Databricks, the Operational Health theme is excluded from the trust score entirely. This can vary per agent, depending on usage and monitor configuration. For complete information, go to AI Trust Score: contributing factors.
Pass and fail criteria
Each monitoring check corresponds to one LLM judge evaluation of one agent interaction. The result of each check is one of the following:
- Pass: The judge returned a passing verdict for the interaction.
- Fail: The judge returned a failing verdict for the interaction.
- No assessment: The judge ran but MLflow could not map its response to a pass or fail outcome. This typically happens because LLM judges are themselves non-deterministic. They can occasionally return unexpected output (extra text, an ambiguous answer, or a value outside the expected set) instead of the constrained verdict they were instructed to produce. A no-assessment result can also occur if the trace was present but the judge had not yet evaluated it at the time of sync.
The pass rate displayed in Collibra is calculated as follows:
Pass rate = passes ÷ (passes + fails + no-assessments)
This differs from the pass rate shown in Databricks, which excludes no-assessment results from the denominator. As a result, the pass rate in Collibra will always be equal to or lower than the pass rate shown in Databricks for the same judge and time period. The Collibra UI shows a tooltip on pass rate values to clarify this difference.
For example, if a judge evaluated 100 interactions and 90 passed, 2 failed, and 8 returned no assessment:
- Collibra pass rate: 90 ÷ (90 + 2 + 8) = 90%
- Databricks pass rate: 90 ÷ (90 + 2) ≈ 98%