First OpenAI, now Meta – Why do AI hacks keep happening?

Over the last fortnight, reports of AI models going beyond their expected bounds - be that technically or morally - have been seemingly unavoidable.

What started with a trickle – ChatGPT-maker OpenAI admitting their AI had hacked the site Hugging Face – has turned into a flood of groups revealing they had discovered instances of AI going out of control.

Claude-maker Anthropic, Meta and the UK's AI Security Institute (AISI) have now each reported incidents which seem to paint a worrying picture of a world in which tech going rogue is the norm.

In reality, each case offers a window into the risks posed by increasingly capable AI agents - and the importance of testing their limits before they are released to the world.

The OpenAI incident has, as Hugging Face's co-founder Thomas Wolf described it, come as a "wake-up call" for the tech industry since it happened at the end of July.

It was a big moment which caused big companies to reflect on their own systems - and, in some cases, check they hadn't missed something similarly shocking.

Anthropic was the first to act. On Friday, the company found three instances out of thousands where its model Claude had managed to gain access to the internet.

Then on Tuesday, the AISI, the UK government agency which evaluates cutting-edge models, then said it had detected a "security incident" during a routine evaluation.

It had been testing models by both OpenAI and Anthropic, and found they too tried to carry out cyber-attacks - calling for "scrutiny, transparency, and action".

Finally followed Meta, which revealed one of its AI models had inadvertently been allowed to access the internet due to a "misconfiguration" during a third-party test.

In disclosing the incident, it is following in the footsteps of those before it.

Testing questions

Before AI models are released to the public, they are put to the test in a series of internal and external evaluations.

The aim is to figure out their potential to do good or bad, as well has how they perform in benchmarks measuring their skills.

These typically take place in what are known as "sandboxes". These are protected spaces designed to mirror real systems - but with strict guardrails in place.

In the OpenAI-Hugging Face incident, the AI attacked the sandbox itself, finding a vulnerability which let it access the internet and "go rogue".

Meanwhile the AISI said its own incident, which saw two powerful AI tools create fake human profiles to try and trick people in attempted cyber-attacks, was not down to an issue with the sandbox.

Instead, it was due to how it went about its tests.

The models it tested were granted access to the internet, and the AISI also disabled in-built filters that would usually block dangerous cyber-attacks.

"To some degree, our evaluation design choices and specific configurations enabled the behaviour," it said, while noting its unexpected "signs of novel, potentially deceptive behaviours".

Prof Alan Woodward, professor of cyber-security at the University of Surrey, said these cases - while distinct in what happened and why - tell an important story.

"For 30 years, one rule of software testing held firm: whatever happens in the test environment stays in the test environment," he said.

"In the past month, that rule has been broken three times."

"One model broke out. One walked through a door left open by mistake. One was deliberately given the keys so testers could measure what it would do."

He said these were different causes, but they had the same lesson - "the testing lab is now where the risk lives".

He told the BBC that as models become more capable, more must be done to secure the environments where they are tested.

"Testing an AI agent is less like checking code and more like handling a hazardous material: sealed rooms, constant monitoring of what leaves the building, a rehearsed containment plan," he said.

"AISI contained its incident within an hour. The next organisation may not."

Growing capabilities

For those developing AI tools which are designed to take actions on a person's behalf, there is a careful balance to be struck between harnessing their benefits and exposing their risks.

The benefit is significant. In theory, we could be able to liberate ourselves of dull, menial tasks, such as replying to emails, going to meetings or managing calendars and diaries, by delegating these to capable bots.

The downside is that with great power comes great responsibility, and risk.

It's something particularly realised when handing power to tools which are not, like us, able to bring a range of values, context and understanding to decisions we made.

"Recent incidents of frontier AI models carrying out unsanctioned actions and, in some cases, human-like deceptive behaviour on the open internet are a serious reminder of the risks AI capabilities pose," said Ollie Whitehouse, the National Cyber Security Centre's chief technology officer on Tuesday.

Some believe the sheer volume of tasks that will be handled by these tools will mean human oversight might not be enough to contain the problem of models going rogue.

But in the meantime, many feel strengthening oversight overall is vital if development continues at its same, frenzied pace.

What next?

It is unlikely Meta will be the last to emerge with findings of models showing they have, as Prof Woodward puts it, "gone to school" – and learnt our own ways of finding and exploiting gaps in systems.

For some, these episodes point to clear security failures on the part of AI companies leading the charge on this game-changing, era-defining tech.

For others, they are merely another vehicle for tech firms to hype up their powerful models and compete with rivals.

For me, both theories hold some grain of truth.

But in rearing their head one after another, these events have nonetheless spurred fears about AI's capabilities and where these are headed as developers forge ahead.

And the question inevitably moves to what regulators can and should do next.

Michael Birtwistle, associate director at the Ada Lovelace Institute, makes the point that the UK lacks legal incentives for AI firms to prevent systems from developing capabilities which could pose dangers, and that there are no repercussions if testing protocols fail.

More broadly, Dr Imogen Stead, AI policy manager at the Centre for Long-Term Resilience, told the BBC that with opportunities to test frontier AI systems narrowing for many, governments should follow the UK in setting up dedicated institutes for testing.

Improving third-party evaluations with initiatives such as a "trusted tester scheme" for the most risky types of challenges could also be used to limit adverse impacts, she said.

Rather than fear an AI-cyber apocalypse in the meantime, Prof Woodward says, "it's a case of 'keep calm and fix stuff'".