More than 300 scientists say AI oversight needed after 'preventable' hacks

In short:

More than 300 scientists from across the Americas, Europe, Asia, Africa and Oceania are calling for greater AI safety measures in an open letter.

The letter gave three recommendations, including establishing an international body to record and publish security incidents caused by AI.

What's next?

The federal government is finalising its national AI standards and plans to legislate new rules in 2027.

A group of 350 scientists from more than 30 countries has written an open letter calling for greater AI security by establishing an international body to record and publish security failures due to AI.

In several incidents in recent months, both overseas and in Australia, AI models have hacked into the systems of external organisations, including startup Hugging Face and multiple government websites.

"[These incidents] can also be seen as an irresponsible lapse in standard cybersecurity practices, preventable with stronger sandboxes and procedures," the open letter reads.

The scientists are from 31 countries, including Australia, and are based across Europe, Asia, Africa and Oceania, and said that without immediate change, future incidents could expose critical infrastructure.

It had three recommendations, including:

- adopting high-assurance containment systems, and initially defaulting to air-gapping, which means disconnecting from the internet for cyber-capable AI models

- creating an international body to record and track AI containment failures

- reporting security incidents to the body and making them public.

Toby Walsh, chief scientist at the University of New South Wales AI Institute, was among the signatories and said now was the time to act.

"We're lucky so far because the hacks that have happened have been incredibly modest cyber events,"Professor Walsh said.

"But you can imagine what would happen if an AI [model] was trying to hack into the air traffic control system. It could easily cause significant harms to our day-to-day lives."

Safeguards will ensure incidents won't repeat

Following the Hugging Face hack, where an AI model broke into the company's systems, OpenAI described it as a "warning shot", saying without proper safeguards AI agents could take "dangerous actions" without human direction.

But Professor Walsh said one of the recommendations in the open letter would have prevented the hack entirely.

"We wouldn't have had the Hugging Face attack if they just air-gapped the test environment from the internet,"he said.

"We wouldn't have had the Medicare hack if they'd been careful about giving the agents minimal permissions for the task it was needed to do.

"I mean, these are all incredibly basic things."

Professor Walsh said the most important element of the open letter was the creation of an international body to provide both information sharing and oversight of incidents.

He said similar bodies existed for industries such as banking, aviation and pharmaceuticals.

"Every time there's an aviation accident, independent inspectors go in and find out what happened, so that everyone can learn and not make that same mistake again," he said.

"We could do the same with AI because it's advancing rapidly and we need to make it as safe as possible."

International cooperation needed to prevent incidents

Last week, Assistant Minister for Science, Technology and the Digital Economy Andrew Charlton echoed those sentiments surrounding regulating AI similar to banking and aviation in a speech.

The government is finalising its national AI standards, and plans to legislate the new rules next year.

But Professor Walsh said there needed to be broad international consensus on AI because the technology was not being developed in isolation.

"So we need to cooperate internationally … in terms of perhaps reporting of incidents and sharing of information that doesn't mean that we have to regulate internationally," he said.

Cyber security improved after it became a requirement for companies to report incidents, Professor Walsh said.

"[Otherwise] no-one knew what was happening and we weren't able to learn from all of the mistakes that companies made," he said.

"Similarly here, we're relying at the moment on the goodwill of these companies to report what's going on."