By Lee Rossey, CTO and co-founder of SimSpace, and Greg Bell, co-founder and CSO of Corelight
Frontier AI models breaking their own guardrails is a useful signal of how far autonomy has outrun oversight. Such incidents are dominating the conversation right now, alongside anxious speculation that unconstrained models could do far worse.. All this serves to distract attention away from the looming problem mostly like to hit security teams.
What’s coming for most organizations isn’t a model that’s managed to escape its sandbox. The real threat derives from less advanced but widely-accessible AI systems doing reconnaissance faster, writing convincing phishing emails, and chaining together vulnerabilities that have been sitting in a public disclosure database for months or years. An attack doesn’t need to be novel anymore – or particularly clever – when the hundreds of existing attack playbooks can be run in parallel at little cost. AI has democratized cyber attack, making vulnerability research and exploitation – previously expensive and manual – newly cheap and scalable. As a result, vulnerability discovery happens faster than organizations can handle.
Bringing AI into the SOC without losing control of it
It’s easy to say we need AI to fight AI. That’s absolutely true,but boards don’t want to hear “we deployed AI” without an understanding of governance and ROI. A CISO who grants autonomy to agentic systems before proving how they behave under real conditions is trading one risk for another. Recent Lab jailbreaks demonstrate how un-guardrailed models tasked with narrow objectives can manifest a host of non-aligned behaviors. Although many enterprise workflows will incorporate models with guardrails, these incidents strongly suggest that AI systems need to earn autonomy and undergo real-time monitoring to confirm ongoing alignment with human goals.
Two of the most important factors in building trustworthy agentic systems are data quality and rigorous testing. High quality security outcomes require richly-detailed and highly-structured input data, both for model training and ongoing operations. Low resolution security data may have been ‘good enough’ in the past, but given AI-accelerated vulnerability discovery and attack automation, defenders need much more granular data today, in order to distinguish (for example) between stealthy lateral movement and normal access patterns. Low-quality data actually imposes a hard cap on the quality of inference that can be derived from it. You might naively think that frontier models can ‘turn straw into gold’ – ie, derive critical insights from a large quantity of low-quality data. In fact, that’s not the case: even the most advanced models stumble when the underlying information needed to answer relevant questions is absent from training and real-time data sets. When agentic SOC workflows are built on a foundation of quality data, actionable insights come faster, more accurately, and at lower token cost. For all these reasons, it’s important to standardize on high quality data when training and deploying agents in the SOC. Some of this can be gathered in real environments, and some may come from AI proving solutions that can generate labeled synthetic telemetry (eg adversary activity, user behavior, and network traffic) on demand, so teams can train and test AI agents without needing access to sensitive customer environments AI proving solutions make it possible to prepare for a much broader range of attack scenarios than any single customer environment is likely to encounter.
With quality data in place, the next step in building trust is realistic testing, and measuring whether AI actually beats your organization’s baseline metrics of MTTD and MTTR. The most effective way to do this is to run agents against conditions that match your own environment, alongside existing playbooks and people in realistic attack scenarios. No single vendor can validate an AI security tool on its own, because no AI agent fails in just one dimension. It fails across data, models, workflows, and the judgment calls a human analyst would normally catch, so testing has to cover that whole chain and keep running after launch. This is why we, along with other stakeholders, started the AI Proving Ground Consortium (AIPGC). The AIPGC aims to help enterprises train, validate, and prove AI systems in a controlled environment before they touch a live security operation. The goal is to provide organizations with a closed loop for testing: a realistic environment, network-level evidence of what the AI actually does, a live agent to test against, real attack simulation, and independent analysis of the results. No single vendor grades its own work, so customers see the whole picture instead of stitching it together themselves.
AI has brought our long-standing security problems to the surface. Once organizations accept that the perimeter is gone and breaches are inevitable, then the real question becomes how fast intrusions are detected and mitigated. And although AI can compress the time between an attacker’s first move and a security team’s first reaction, the quality of that human response will remain vitally important. Security teams need to be enabled with high-quality data, trustworthy agentic assistance,, and a concrete sense of what their own environment looks like on an ordinary day. Without this robust foundation, AI agents will only amplify noise and waste human time – still the most valuable resource in the SOC.
##
ABOUT THE AUTHOR
Lee Rossey, Chief Technology Officer & Co-Founder, SimSpace
Lee is a former Group Leader at MIT Lincoln Laboratory, where he established the Cyber System Assessments Group, which became a nationally recognized centre of excellence. He led the group in cyber range development, cyber test and evaluation, cyber red teaming, and cyber exploitation as well as the deployment of the LARIAT traffic generation tool. As Group Leader, Lee worked with DARPA, testing, and evaluating over 30 classified and unclassified programs. He also worked with the Air Force.
Lee led several national studies for the Department of Defense to identify and assess existing capabilities, develop an approach and roadmap for their consolidation and advancement and provide the roadmap for the future, which laid the foundation for the National Cyber Range Complex, the Persistent Cyber Training Environment, and other specialized testing and training facilities.
Since 2015, Lee has served as the Chief Technology Officer and Co-Founder at SimSpace. The SimSpace platform is being used as the solution for USCYBERCOM’s Persistent Cyber Training Environment and for other militaries around the world as well as many commercial customers.
