Ask an AI agent a straightforward security question, and there’s a good chance it gets close but not quite there. That’s the finding behind a new benchmark from Sola Security, which tested large language models against 55 real-world security questions and found they answered correctly about 78% of the time. Roughly 18% of answers came back partial, and about 4% missed the mark entirely.
For Sola, that 78% ceiling isn’t really about the model. CEO Guy Flechter argues that large language models are already a commodity — anyone can ask them questions. The differentiator is context. An LLM can generate a fluent answer to almost anything, but it doesn’t inherently understand a specific company’s environment: which assets connect to which services, which relationships matter, which shortcuts introduce risk.
Flechter pointed to a well-known observation in security circles, often credited to Microsoft’s John Lambert: attackers think in graphs, defenders think in lists. Enterprise environments — assets, identities, service accounts, permissions — are fundamentally a graph of relationships. Large language models, built to process language, are naturally better suited to lists and text than the tangled, many-to-many structure of a real production environment. That mismatch, Flechter said, is the real ceiling on agent accuracy, not any particular model’s raw intelligence.
Sola’s answer is what Flechter calls a “security brain” — a context layer that sits between the LLM and the environment, mapping relationships between assets so an agent doesn’t have to guess. In practice, this shows up in two measurable ways. First, agents given shared context answered questions more accurately, because they could trace connections between assets that aren’t directly linked, without needing to be walked through every hop by hand. Second, agents with better context needed far fewer exploratory queries to reach a usable answer, cutting wasted iteration by roughly 70%, according to Sola’s data.
That efficiency gain matters because iteration isn’t free. The farther two assets sit from each other in an environment’s graph, Flechter explained, the more hops an agent has to work through, and each hop adds both cost and the odds of a wrong turn. Give the agent the relevant context up front, and it can skip the guesswork.
Flechter doesn’t expect the 78% ceiling to break through incremental model improvements alone. He estimates industry-wide gains might inch the number up a few points a year, enough to matter, not enough to reach the reliability a security team could fully trust for autonomous decisions. Getting there, he argues, takes more than prompt engineering: purpose-built context layers now, and eventually, foundational models built around graph reasoning rather than language, in the spirit of research directions like Meta’s JEPA architecture.
In the meantime, Sola has published its benchmark methodology through open.security, an open research initiative that lets other teams test their own agents against the same standard rather than take Sola’s numbers on faith. For developers and architects building or evaluating AI security agents, that’s a useful gut check: an agent that can’t explain its accuracy against your organization’s specific environment probably shouldn’t be making unsupervised decisions yet.
##
ABOUT THE AUTHOR
Tom Smith is a technology writer and content strategist with more than 10 years of experience covering enterprise technology, agentic AI, cybersecurity, DevOps, and cloud infrastructure. He has published more than 2,500 articles and conducted more than 4,000 executive interviews across outlets including Coder Legion, Techstrong/DevOps.com, and Insights From Analytics. By day, Tom serves as a content strategist at Cognizant. He can be found on LinkedIn, X, and Bluesky at @ctsmithiii.
