By Michelle Gill, GitLab
AI agents are fundamentally changing software development. They can autonomously read codebases, write and edit files, run tests, and fix bugs, all from a single prompt, without requiring engineers to write those prompts themselves. The next step is for agents to handle everything from booking business travel to processing procurement requests, using your credentials to do so.
That capability comes with real responsibility. Agentic AI poses distinct risks that software companies urgently need to address. The Center for AI Standards and Innovation, an arm of the National Institute of Standards and Technology (NIST), has taken enough interest in agentic AI risks to begin studying how to track the development and deployment of these tools.
“AI agent systems are capable of taking autonomous actions that impact real-world systems or environments, and may be susceptible to hijacking, backdoor attacks, and other exploits,” NIST notes in a document on the topic. “If left unchecked, these security risks may impact public safety, undermine consumer confidence, and curb adoption of the latest AI innovations.”
Agents expand and reshape the attack surface. Agent-to-agent interactions introduce vectors that traditional security models were never designed to detect, and agents can also connect low-severity vulnerabilities into high-severity exploits.
Security teams should already understand these risks. Engineering leaders eager to use agents must understand not only what agents can do, but what agentic capabilities mean for their organization’s security posture.
A shared understanding of AI’s risks bridges the gap between engineering teams and their security counterparts, giving both sides the foundation to ship faster and more securely.
How agents alter the threat model
The nature of large language models generates a variety of security challenges, some entirely new, others familiar problems in a different form.
AI agents share some risks with other software, such as exploitable vulnerabilities in authentication systems or memory management processes. NIST’s attention, however, is on the novel, more dynamic dangers unique to machine learning models and AI agents.
One of the biggest risks of AI, prompt-injection attacks, becomes more complex due to the non-deterministic nature of LLMs. The same prompt-injection attack may succeed or fail on different attempts, making remediation difficult to validate and comprehensive defenses challenging to implement.
Models can also carry intentionally installed backdoors, leaving critical systems vulnerable. Researchers have also raised concerns that even uncompromised models could threaten the confidentiality, integrity, or availability of critical data sets.
A further challenge comes from how individual capabilities combine within a single agent. AI agents merge language-model reasoning with access to tools. That means the ability to read files, query databases, call APIs, execute code, and interact with external services, all within a single system.
The danger emerges not from any single capability but from their combination and an agent’s ability to carry out these actions autonomously. Without proper guardrails, agents can delete codebases, expose sensitive data, and introduce cascading failures that are costly and difficult to unwind. Agents can even work around some guardrails to complete their assigned task.
Risk increases when agents have access to private data, receive untrusted content, and can communicate externally. This combination presents a materially different risk profile than one lacking any of these three elements. Some observers call this combination the “lethal trifecta.”
Additional risks include:
- Unintended operations, where agents execute actions beyond their intended scope due to misinterpreted instructions or prompt manipulation.
- Privilege escalation, which occurs when agents operating with broad permissions perform sensitive operations that exceed what the initiating user authorized.
- Cascading failures, where one compromised agent in a multi-agent system can corrupt others downstream.
Engineering countermeasures for agentic AI risks
All of these risks have concrete countermeasures. The most effective approaches layer controls at three levels.
- Model level: Maintain clear separation between system instructions and untrusted content using distinct messaging roles and randomized delimiters. Secondary classifiers provide an additional layer, scanning inputs and outputs for injection patterns and anomalous formatting. These measures reduce risk but do not eliminate it, which is precisely why the layers below matter.
- System level: Apply least privilege across the board. Agents should only access the tools required for their tasks, with credentials narrowly scoped and set to expire quickly. Examine content entering the system for injection patterns, and screen outbound content for sensitive information such as credentials or PII. Set default-deny network controls to limit external communication to explicitly approved endpoints. Design workflows to break the lethal trifecta, ensuring that no single agent can access sensitive data, process untrusted content, and communicate externally all at once.
- Human oversight level: Require explicit approval for critical operations while allowing lower-risk actions to proceed with notification. Tiering the approach prevents approval fatigue, which can lead to oversight. Users should be able to halt execution at any time, with rollback of partially completed work where possible. When an agent acts on behalf of a user, record both identities and evaluate permissions at their intersection. Log all agent actions, timestamps, identifiers, tools invoked, resources accessed, and outcomes in sufficient detail to reconstruct events after the fact.
Why responsible governance is a competitive edge
Teams can meaningfully reduce these risks through layered controls. The opportunity is just as real as the risk.
Agents working on your behalf make the lethal trifecta work for you. The right combination of data access, content processing, and external communication, when governed well, is exactly what makes agents powerful tools. AI agents can monitor systems, apply consistent security rules without fatigue, and build quality, secure code at a speed and scale no manual process can match. They function as a force multiplier, amplifying both strengths and weaknesses in equal measure.
Software engineers will always be necessary, but teams that deploy agents with the right governance and guardrails will have a clear advantage: faster development, faster remediation, and fewer security errors that undermine software quality.
Organizations that get the most from agentic AI will be those that understand the threat model and design against it from the beginning. That understanding draws the line between teams that deploy agents responsibly and those that learn the hard way.
##
ABOUT THE AUTHOR
Michelle Gill combines strategic vision with hands-on expertise to build and guide high-performing engineering teams within DevOps and the rapidly evolving fields of AI and data science. Currently at GitLab, she drives innovation in AI-powered workflows that enhance efficiency and reduce cycle times across their DevSecOps platform. With direct experience in software engineering, Michelle is passionate about helping teams develop skills they need in order to increase impact on their company’s success, and has employed numerous methods of trial and error, servant leadership, and old-fashioned hard work over the last 15 years to realize this progression at scale.
