AI monitoring gives you visibility into how well a deployed AI agent's responses meet quality standards. Collibra monitors agents by reading LLM judge (scorer) evaluations from Databricks. Each judge assesses agent interactions, or a sample of them depending on how the judge is configured in Databricks, and returns a pass, fail, or no-assessment result. Monitoring results feed into the Operational Health theme of the AI Trust Score for AI Agent Version assets, and are surfaced in the agent Quality tab and the Agents dashboard.
For a conceptual overview of how AI monitoring fits into the broader operational trust framework, go to Operational trust for AI agents.
What AI monitoring tracks
For each LLM judge linked to an agent, Collibra collects the following signals per hour from Databricks agent interaction traces:
- The number of interactions that passed, failed, or received no assessment from the judge.
- Prompt and completion token counts.
These metrics are used to compute daily pass rates and 7-day aggregate scores that appear in the Trust Score, agent quality tab, and Agents dashboard. Current-day metrics are updated on an hourly basis.
How AI monitoring connects to deployments
AI monitoring operates at the deployment level. A specific AI Agent Version is linked to an AI Endpoint through an AI Agent Deployment, which is a complex relation. One or more AI Monitor assets can be linked to each deployment.
However, because agent interaction traces carry only the MLflow experiment ID, not an agent version ID, monitoring data is aggregated at the AI Agent level, not the version level. A single AI Agent may have multiple deployments across different endpoints, each with its own set of LLM judges.
| Relation | Public ID | Kind |
|---|---|---|
| AI Agent Version + AI Endpoint → AI Agent Deployment | AIAgentDeployment | Complex relation |
| AI Agent Deployment is monitored by AI Monitor | DeploymentMonitoredByAIMonitor | Explicit relation |
AI Monitor assets
In Collibra, each LLM judge configured on a Databricks agent is represented as an AI Monitor asset. AI Monitor assets are created automatically when the Databricks integration syncs — you do not create or configure them manually. The name of each AI Monitor asset corresponds to the name of the LLM judge in Databricks.
A single AI Agent can have multiple AI Monitor assets, one for each judge that evaluates the agent's responses. AI Monitor assets can appear in domains of type Governance Asset Domain. The following attributes are defined on the AI Monitor asset type.
| Attribute | Description | Public ID |
|---|---|---|
| Data Drift Detection Enabled | Whether data drift detection is enabled on this monitor. | DataDriftDetection |
| Prediction Drift Detection Enabled | Whether prediction drift detection is enabled on this monitor. | PredictionDriftDetection |
| Schedule | The frequency at which monitoring data is synced from Databricks into Collibra. | Schedule |
| Alert Configuration | The threshold conditions that trigger an alert, and the notification channels used to notify stakeholders when a threshold is breached. | AlertConfiguration |
| URL | The URL of the LLM judge in Databricks. | Url |
Connection to the AI Trust Score
Monitoring results feed into the Operational Health theme of the AI Trust Score for AI Agent Version assets. The score for this theme is the 7-day average monitor pass rate — the percentage of monitoring checks that passed over the last 7 calendar days — expressed as a value from 0 to 100.
If no monitoring data is available for this agent for the past 7 days, for example because no interactions have occurred or no LLM judges are configured for it in Databricks, the Operational Health theme is excluded from the trust score entirely. This can vary per agent, depending on usage and monitor configuration. For complete information, go to AI Trust Score: contributing factors.
Pass and fail criteria
Each monitoring check corresponds to one LLM judge evaluation of one agent interaction. The result of each check is one of the following:
- Pass: The judge returned a passing verdict for the interaction.
- Fail: The judge returned a failing verdict for the interaction.
- No assessment: The judge ran but MLflow could not map its response to a pass or fail outcome. This typically happens because LLM judges are themselves non-deterministic. They can occasionally return unexpected output (extra text, an ambiguous answer, or a value outside the expected set) instead of the constrained verdict they were instructed to produce. A no-assessment result can also occur if the trace was present but the judge had not yet evaluated it at the time of sync.
The pass rate displayed in Collibra is calculated as follows:
Pass rate = passes ÷ (passes + fails + no-assessments)
This differs from the pass rate shown in Databricks, which excludes no-assessment results from the denominator. As a result, the pass rate in Collibra will always be equal to or lower than the pass rate shown in Databricks for the same judge and time period. The Collibra UI shows a tooltip on pass rate values to clarify this difference.
For example, if a judge evaluated 100 interactions and 90 passed, 2 failed, and 8 returned no assessment:
- Collibra pass rate: 90 ÷ (90 + 2 + 8) = 90%
- Databricks pass rate: 90 ÷ (90 + 2) ≈ 98%