Cyberattacks carried out using AI agents are a scary reality we are starting to witness, but can AI become better than humans at cybersecurity? In this article, we look at two recent attacks performed using AI, present the latest research on this topic and share what we learned from building our own AI pentesting setup and putting it to the test.

Introduction

In November of last year Anthropic published a report on what they described as the first documented large-scale cyberattack orchestrated and executed mostly by autonomous AI agents. The attack, which is believed to be orchestrated by a Chinese state-sponsored hacker group branded as GTG-1002, used a setup based on Anthropic’s AI agent Claude. The goal was cyber-espionage of about 30 targets, from technology companies to government agencies, with the AI handling an estimated 80-90% of tactical operations independently. The attacks followed the standard Advanced Persistent Threat (APT) playbook: Reconnaissance, credential theft and lateral movement to collect and extract data. The attackers did not use sophisticated custom malware or any 0-day vulnerabilities, just open source tools generally available to the wider public.

Their setup was in fact so simple that we managed to build a small scale replica of it ourself without too much effort. It was not exactly like the original one, but it did work well enough for us to test it against some intentionally vulnerable machines and to learn a few things about AI pentesting in the process.

How Good is AI at Hacking?

Before getting into the details of our setup and how it worked, we need to take a look into the current landscape of AI cyberattacks, both in research and in real-world scenarios.

The GTG-1002 Case Study

As mentioned in the introduction, in November 2025 Anthropic disclosed what seems to be the first large-scale cyberattack mainly carried out by AI. The GTG-1002 hackers used a setup where one Claude instance (using the Opus 4.6 model) acted as a coordinator, with multiple other AI sub-agents in the role of workers. The coordinating AI divided the attack into smaller phases, provided the instructions to its sub-agents, aggregated their results and presented them to the human operators, who then guided the attack without handling tactical execution.

The AI in this attack was not fully autonomous: Humans were required to choose the targets, chain attacks together, guide the AI and correct the mistakes that occasionally the LLMs ran into. While this last point might not seem particularly important — after all humans make mistakes all the time too — it is crucial in defining the future role of AI in cybersecurity. As Anthropic itself reports, Claude hallucinated and frequently overstated findings, claiming to have found valid credentials which in reality did not work, or assigning a high severity to findings which then proved to be publicly available information. This effectively prevented the possibility of a fully autonomous attack of this scale and forced human operators to oversee the operation and bring the AI back on the intended track. So one conclusion we can take from this event is that as long as AI hallucinations are a thing, large-scale cyber attacks will always require some part of human oversight.

CodeWall’s Fully Autonomous Attack

The GTG-1002 attack showcased what AI is capable of today with the help of human oversight, but how good is it when it is left on its own? In other recent cybersecurity news, the security startup CodeWall reported that it had gained full read and write access to a production database used by the consulting firm McKinsey. CodeWall’s CEO claimed that the attack was run entirely by an AI agent without human intervention. The agent apparently even suggested McKinsey as a possible target itself after learning that there had been recent updates to McKinsey’s infrastructure. CodeWall’s researchers simply pointed the AI agent at McKinsey and only a few hours later, without any prior credentials or particular access, managed to identify and confirm the vulnerability on McKinsey’s side.

While this shows that fully autonomous exploits are possible, when looking more in detail into this particular incident we can see that the complexity of this attack was not very high and the AI agent’s most important contribution was to identify an unauthenticated, publicly accessible API endpoint. The exploit itself was a basic SQL injection, with the server even returning SQL errors in response to malformed requests. Essentially, the AI allowed the security startup to cover a very wide surface area in a short time to discover possible vulnerabilities. This reinforces a pattern we have observed during our testing: AI agents are remarkably effective at covering the breadth of an attack surface, but struggle when exploitation requires depth, like multi-step attacks where each action depends on the outcome of the last.

What Does Research Say About This?

These findings were also validated by a paper published by a group of security researchers at the AI Security Institute in the UK. They tested various models, with increasing token budgets against two simulated targets without any human oversight.

First, they showed what we have all been seeing over the past years: Each new model generation clearly outperformed its predecessors. The best performing model was unequivocally Anthropic’s Opus 4.6 (the same that was used by GTG-1002 in the large-scale attacks), which had its best run solve 22 steps out of 32 total steps in one of the two target applications, with an average completion of 15.6 steps over five runs. The high variance in step completion is not a very positive factor for the AI, as it indicates that even under identical circumstances, the results can vary substantially. While the paper does not mention this, a possible explanation for why some of the runs were less successful is that they may have been affected by AI hallucinations. Given the absence of human guidance, the AI failed to correct and ended up getting stuck.

Another very interesting finding was that up until the tested 100M tokens, the performance of the agent correlated log-linearly with the token cost. The study found that a 10x increase in tokens, from 10M to 100M, corresponds to roughly a 59% increase in steps solved. This means that even if there was no ceiling to the attack complexity that a model can perform, the cost is scaling exponentially. Some models did however have a limit to the amount of steps they could complete, regardless of the number of tokens consumed. GPT-4o for example could not get past step 3 regardless of token usage. This log-linear correlation between attack length and token cost reinforces the idea that these models have a hard time chaining multiple steps together to achieve long exploits.

Our Experience

Before testing the agents with our setup, the expectations were not very high. We used AI every now and then to help in daily tasks, but more often than not it would either be totally useless, or require so much debugging and effort on our side that it would have been easier to do the task without it. Initially, it looked like our instinct was right as we saw the AI get stuck on the most basic things, failing to recover from simple mistakes and attempting super complicated solutions to fairly easy problems. On top of that, it had significant issues dealing with interactive sessions and commands that did not return an output. While it was really good and fast at doing the initial bulk of the work like identifying running services, searching for known vulnerabilities and providing a clear and structured overview of the target, it had obvious problems when the tasks’ complexity began to increase. However, by helping the AI out a little bit, we managed to improve its results significantly.

A deciding factor for the performance of the AI is the system prompt that is used. By providing it with a strict set of engagement rules and a reference on how to perform the attack, as well as how it should handle errors and interactive shell sessions, the AI stopped wasting time (and tokens) in endless recovery loops, doing things it was never asked to do or waiting for commands that would never produce any output. This is not as easy as we initially thought, but luckily we found out that AI is quite good at writing its own prompts. We managed to bring up its performance by asking it to update the system prompt after every run to try to iteratively improve it as much as possible. We then added an extra layer to this and told the AI to keep track of the mistakes it made and how it solved them in a document that it then had to read before every interaction. This prevented it from making the same mistakes over and over and getting stuck.

At this point solving easy to medium-level Capture The Flag (CTF) challenges was quite straightforward, but we wanted to try out the AI coordinator/worker setup from the GTG-1002 attack. That is when we found out that using Claude Code was much more effective for this kind of experiments than using the Desktop version. The Code version can run longer sessions and has better integrations with the tooling on the attacking machine. This setup made the attacks easier to manage on our side and faster to carry out, as multiple AIs could try different things in parallel, but it did not allow us to solve more complex challenges. At the end of the day, if a challenge was too hard for the AI to solve normally, it would be too hard to solve even with multiple instances of the AI attempting it.

While setting up this leader/worker infrastructure, we also discovered a small trick to improve token efficiency: Using the Opus 4.6 model for the coordinator AI, but Sonnet 4.6 for the workers. Because of the nondeterministic nature of AI, it was not possible to measure how much more efficient this setup was, but it was faster than using just one agent and it was cheaper than using multiple Opus instances. It is however still more expensive than using just one instance of the AI, due to the overhead of coordinating between the agents. Because Sonnet has good cybersecurity capabilities with looser token rate limits, this setup allowed us to conduct faster attacks that were still orchestrated by the superior Opus model, without running into the stricter rate limits that Anthropic enforces for it. Occasionally, when tasks became too complex for Sonnet, Opus had to jump in and attempt the exploits itself, but for the most part Sonnet handled the job correctly.

It was definitely fun and instructive to test all of this, but it has to be noted that while this trick worked on our local setup for attacking CTF-style challenges and deliberately vulnerable machines, it does not mean that this would actually work in a real-world environment with significantly more complicated system and users using them in real-time. On top of that, the experiments we conducted were missing any sort of defensive strategy. The systems were intentionally flawed and no sort of intrusion detection or prevention was performed. AI will need to find a way to surpass this obstacle if it is going to become more widely adopted in offensive cybersecurity.

Conclusion — Will AI Take Over Cybersecurity?

This is the ultimate question that we were trying to answer in the past weeks. From what we have seen and what we have read, it seems that even as the models keep improving and the tokens become cheaper, the issues facing AI right now will not go away anytime soon. Both the GTG-1002 case and the research paper indicate that AI is not any better at hacking than a human attacker, but it can be faster and can be used to cover wide attack surfaces at scale much more efficiently. However, we do not think we are really close to it replacing humans in this context.

In our own experiments, human guidance was crucial for solving higher-level challenges. On top of that, hallucinations not only occasionally prevented the AI from advancing in the attack chain, but sometimes even jeopardized the operation by attempting to move the attack to other, unauthorized targets. This was a major red flag that reminded us that even if the model we wer talking with seemed like an intelligent being, it was still only a statistical model with no understanding of what was happening or what it was doing. It also reminded us that ultimately AI is just a (very powerful) tool that needs to be used by someone for something. The more experienced and competent its operator, the better results it can achieve. At the end of the day, the role that AI can play in cybersecurity is defined by the humans that are using it and by their ability to use it.

We will undoubtedly see more AI usage in cyberattacks, but we do not think the nature of threats will change significantly. Mostly, AI will impact the speed and scale at which attacks are carried out. AI agents will not find vulnerabilities that a skilled human could not, they will just find them faster and across a wider surface. If your systems are already secure, AI won’t change that. However, if you have weaknesses you have been getting away with because nobody had the time to look, that window is closing.

About the Author

Daniele Del Giudice holds a bachelor’s and master’s degree from ETH Zurich, specializing in Secure and Reliable Systems and Networking. His master’s thesis dealt with decentralized identity technologies for the Swiss e-ID. During his studies, he completed internships at two different security providers in Switzerland. His focus is on networks and data protection. (ORCID 0009-0003-1026-0578)

Links

• https://www.aisi.gov.uk/
• https://www.anthropic.com/
• https://www.arxiv.org/pdf/2603.11214
• https://www.assets.anthropic.com/m/ec212e6566a0d47/original/Disrupting-the-first-reported-AI-orchestrated-cyber-espionage-campaign.pdf
• https://www.codewall.ai/
• https://www.crowdstrike.com/en-us/cybersecurity-101/threat-intelligence/advanced-persistent-threat-apt
• https://www.mckinsey.com/
• https://www.theregister.com/2026/03/09/mckinsey_ai_chatbot_hacked/
• https://www.vuldb.com