AI’s Cybersecurity Guardrails Become a Double-Edged Sword, Hindering Defenders and Researchers

For months, leading artificial intelligence developers have implemented stringent vetting programs and robust safety protocols, meticulously designed to prevent their powerful models from being exploited by malicious actors. However, this deliberate caution, intended to secure the digital landscape, is now inadvertently impeding the vital work of legitimate cybersecurity professionals and offensive security researchers. The very safeguards meant to thwart cybercriminals are proving to be a significant obstacle for those tasked with identifying and neutralizing threats before they materialize.

The tension between AI safety and practical cybersecurity application recently came to a head in June when the U.S. government imposed export control restrictions on Anthropic’s highly anticipated AI models, Mythos and Fable. This action was reportedly spurred, at least in part, by a confidential report suggesting that the models’ built-in guardrails, designed to prevent their misuse for malicious cyberattack development, could be bypassed. While the exact motivations behind the government’s decision remain a subject of debate, with some suggesting it was less about a technical "jailbreak" and more about broader geopolitical and safety concerns, the immediate impact was a significant restriction on access to advanced AI capabilities.

Anthropic had, in fact, heavily marketed Mythos as a potent, potentially dangerous tool – a "doomsday cybermachine" capable of sophisticated cyber operations. This framing necessitated the imposition of strict access controls and guardrails, allowing only carefully vetted users to interact with the model. The export controls on Fable 5 and Mythos 5 have since been partially lifted. Fable 5 returned to general availability on July 1, while Mythos 5 has been reintroduced to vetted U.S. organizations under a government review process, highlighting the ongoing scrutiny of these powerful AI tools.

This approach of selective access and strict limitations is not unique to Anthropic. Both Anthropic, with its other AI offerings, and OpenAI have established specialized programs designed to grant cybersecurity researchers access to AI models with fewer restrictions. OpenAI’s "Trusted Access for Cyber" program and Anthropic’s "Cyber Verification Program" require applicants to undergo a vetting process. If approved, researchers gain access to models that are less constrained by cybersecurity guardrails, ostensibly to facilitate defensive research.

However, these guardrails have drawn considerable criticism from the cybersecurity community, particularly from those whose professional lives are dedicated to uncovering novel vulnerabilities and developing countermeasures. Mark Dowd, a prominent security researcher with decades of experience in discovering and selling "zero-day" exploits – previously unknown software flaws and the methods to exploit them – to Western governments, voiced his concerns during a recent cybersecurity podcast. He articulated a sentiment shared by many in his field: "It’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not."

Dowd’s work, which involves selling vulnerabilities to governments for intelligence purposes rather than reporting them for patching, operates on the principle that discovered flaws remain open for a period, offering strategic advantages. He acknowledges that his perspective might be influenced by his professional activities, but his views resonate with numerous individuals engaged in offensive cybersecurity – those who proactively probe systems for weaknesses.

Chris Anley, Chief Scientist at the security consulting firm NCC Group, elaborated on the paradoxical nature of AI guardrails in a cybersecurity context. He explained that an AI model’s ability to assist in exploiting a bug is a crucial step in validating its severity and the necessity of a fix. However, when guardrails compel the AI to refuse to engage with such requests, they become counterproductive for defenders.

"This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base," Anley stated. "So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked." He drew an analogy to a hammer, a tool essential for building but also undeniably a weapon.

When faced with these AI-induced roadblocks, Anley and his colleagues sometimes resort to open-source AI models that come without any inherent guardrails, offering unfettered access.

Paolo Stagno, CTO at Crowdfense, a company specializing in the acquisition and sale of undisclosed vulnerabilities to government agencies, echoed Dowd’s sentiment. He characterized the vetting programs and guardrails as an infantilizing approach, suggesting that AI companies "essentially treat customers like children who need babysitting." Stagno further noted that while his team utilizes advanced AI models for tasks like reverse engineering, they refrain from employing them for vulnerability discovery or exploit development. The risk of leaking sensitive vulnerability data or having it incorporated into future training sets by cloud-based models is too significant. For these critical functions, they rely on open-source models run locally, ensuring data remains within their controlled environment.

Giuseppe Cali, another security researcher focused on zero-day discovery and exploit development, offered a slightly different perspective. He indicated that guardrails do not actively impede his work because he strategically employs AI. Instead of using it for offensive operations, he leverages AI for initial reverse engineering, code comprehension, and the development of supporting tools. He finds that AI can significantly accelerate these preparatory stages, allowing him to dedicate more time and effort to the core task of vulnerability discovery.

"I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow," Cali asserted. "I am jealous of my bugs, and I like this game too much to let models play it for me." This highlights a segment of researchers who value the intellectual rigor and control over their discoveries.

However, the practical limitations imposed by strict guardrails are felt acutely by others. A researcher at a smartphone-component manufacturer, who requested anonymity due to authorization constraints for speaking with the press, revealed that their employer’s decision not to participate in Anthropic’s Cyber Verification Program means their AI tools are of limited utility for vulnerability discovery. "If it catches wind we’re doing anything security related, it just stops and isn’t usable," the researcher stated, underscoring the pervasive impact of overly restrictive safety measures.

Chris Thompson, CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an event focused on offensive security and AI, shared his observations on the inconsistency of frontier AI model guardrails. He noted that even within the purportedly more relaxed environments of Anthropic’s and OpenAI’s vetted programs, the guardrails can behave erratically, changing their behavior from day to day.

"I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson explained. "Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output." This constant "negotiation" diverts valuable time and cognitive resources away from the primary security objectives.

Consequently, Thompson observed a growing reliance on, or a push towards, open-source AI models originating from China, such as GLM. These models are freely downloadable, can be run locally without any vetting process, and have no usage restrictions. "You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," he warned. "I think it’s more harmful than good to have these guardrails in place." This shift could have significant geopolitical implications, potentially ceding ground in AI-driven cybersecurity to international competitors.

Thompson advocates for a recalibration of the AI development strategy. Instead of progressively tightening restrictions, he calls for AI frontier labs to broaden access to their programs responsibly, implement mechanisms for holding users accountable for abuse, and foster a collaborative environment. He argues that without such a shift, the cybersecurity community risks losing the critical AI race against an escalating threat landscape.

"There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before," Thompson concluded with a sense of urgency. "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now." The current approach, he suggests, is not only hindering the progress of cybersecurity professionals but also potentially leaving the digital world more vulnerable to the very threats AI was intended to help combat.

The dilemma highlights a fundamental challenge: how to balance the imperative of AI safety with the practical necessities of cybersecurity research and development. As AI continues to evolve, so too must the strategies for its responsible deployment, ensuring that the tools designed to protect us do not inadvertently become our own greatest impediment. The ongoing debate underscores the need for a nuanced approach that empowers legitimate actors while maintaining vigilance against misuse, a delicate equilibrium that remains elusive.