Chapter 5
Operating and monitoring AI agents: Stopping errant behavior
Even with strong identity and data controls, continuous behavioral monitoring is essential.
Traditional security controls focus on blocking unauthorized access. AI agents create a different challenge: Misuse can occur within authorized activity. As organizations delegate tasks and decisions to autonomous systems, security must evolve from gatekeeping to continuous oversight. Behavior monitoring is becoming essential.
Unlike conventional applications that follow predictable workflows, AI agents adapt to context and inputs in real time. This flexibility enables efficiency but also introduces uncertainty. Agents may act outside their intended scope, due to manipulation, flawed logic, unexpected data inputs, or evolving operational conditions. They may retrieve unusual volumes of data, operate outside normal hours, or execute actions inconsistent with their assigned roles. These deviations can signal compromise, misuse, or unintended consequences.
of organization say AI agents complicate intrusion detection.
How errant behavior emerges
The challenge is not always malicious intent. Errant behavior can stem from subtle issues such as data drift, conflicting instructions, integration failures, or edge case scenarios that were not anticipated during deployment. Because agents can act quickly and at scale, small anomalies can escalate into operational disruptions and compliance violations.
Among the organizations participating in the Ponemon Institute research, 66% said AI systems make intrusion detection more difficult.8 In environments where authorized systems initiate actions across multiple platforms, traditional alerts may not capture early indicators of misuse. Behavioral anomalies often provide the first signal of a potential threat that may evade conventional rule-based detection. The escalating volume of new and sophisticated attacks significantly elevates the risk of unknown threats. Organizations report that detecting and responding to AI-driven threats is increasingly challenging, reinforcing the importance of continuous monitoring and behavioral analysis that self-adapts to a rapidly changing operational environment.
8 Ponemon Institute, Managing Risks and Optimizing the Value of AI, GenAI & Agentic AI, March 2026.
Consider an IT operations agent responsible for scaling cloud resources. If manipulated inputs cause it to disable critical services or dramatically increase resource provisioning, operational disruption and financial impact can occur quickly. Because the agent is operating with legitimate credentials, traditional security tools may not detect the activity as being malicious.
AI-driven environments introduce new behavioral risk indicators, including:
Access patterns that diverge from established norms
Unexpected data retrieval volumes or destinations
Actions executed outside defined operational windows
Unusual tool usage or system interactions
Rapid escalation of privileges or scope of activity
Individually, these signals may appear benign. Together, they can indicate misuse, compromise, or unintended automation consequences.
From static controls to dynamic oversight
Monitoring behavior shifts security from static defenses to dynamic oversight. Establishing baselines for normal activity enables teams to detect anomalies in real time. Rather than focusing solely on whether access is authorized, security teams monitor how systems behave once access is granted.
Effective behavioral monitoring relies on correlating signals across multiple domains:
Identity activity and authentication context
Data access patterns and sensitivity levels
System actions and workflow execution
Network activity and integration behavior
Timing and frequency of automated actions
This contextual view enables security teams to distinguish legitimate automation from abnormal activity.
A supply chain optimization agent offers a clear example. If manipulated data triggers unusually large purchase orders, behavioral monitoring can detect deviations from historical patterns and flag the activity before financial damage occurs. Early detection enables teams to validate inputs, halt execution, and prevent cascading impact across procurement and logistics systems.
Behavior monitoring also helps detect unintended consequences. An agent designed to optimize operational efficiency may make decisions that conflict with business priorities if guardrails are not clearly defined. Continuous oversight ensures that automation aligns with policy and operational intent.
Organizations scaling AI adoption are moving beyond simple alerting toward automated containment and response. When behavior deviates from expected patterns, systems can trigger safeguards such as step-up authentication, workflow pauses, or human review. These controls enable automation to continue operating, while preventing escalation.
Operational visibility into agent behavior supports early detection, rapid response, and continuous improvement. Insights gained from monitoring can inform policy refinement, workflow adjustments, and risk mitigation strategies. Over time, organizations develop a clearer understanding of how autonomous systems operate within their environments.
Security becomes proactive rather than reactive. Instead of responding only after an incident occurs, organizations gain the ability to detect subtle signals automatically without constant labor-intensive rule-tuning, intervene early, and prevent escalation.
As automation expands across enterprise workflows, maintaining visibility into agent behavior will be essential to preserving trust, operational stability, and resilience. Organizations that effectively and efficiently monitor how AI systems act, not just whether they are authorized, will be better equipped to maximize their agentic AI return on investment (ROI) while mitigating operational and compliance risks in an increasingly autonomous environment.
Establishing dynamic behavioral baselines for agents and workflows
Detecting anomalies through self-adapting analytics
Correlating identity, data, and system activity signals
Leveraging threat intelligence to enrich context
Triggering containment actions when risk thresholds are exceeded
Preserving audit trails to support investigation and compliance
These capabilities enable organizations to maintain control without limiting the benefits of automation.
Monitor AI agent behavior continuously. As agentic AI scales, security must go beyond controlling access to rigorous 24/7 monitoring of behaviors. OpenText recommends an automated and context-sensitive approach with dynamic baselines and self-adapting analytics to proactively detect risky agent activities across identities, data, systems, and workflows — even when actions may evade rule-based detections.
By correlating signals such as abnormal data access, unexpected system actions, and unusual workflow activity and enriching insights with threat intelligence, organizations can significantly accelerate effective detections of elusive threats that are escalating in AI-assisted attacks.
Operate AI securely with OpenText Security Operations solutions.
