The leading global powers are in another race. This time it is not about who gets to space or the moon first. It is about who first builds an artificial general intelligence system (AGI): a system that exhibits all the cognitive capabilities the brain has1.
My claim is, along with many others, that developing AGI this fast without a safety ecosystem around it has become a real threat to our society, on the order of great power conflict or climate change. And thus, as for each of societies challenges, I believe we all should educate ourselves upon this challenge.
What’s happening: we are replicating the human brain ready for large-scale deployment and collaboration, and we are doing this at light speed while we have yet little insight into the consequences.
And this is going to be pretty transformative, as depicted beautifully by Demis Hassabis in the opening two paragraphs of his piece on AI Safety (linked below):
”[…] This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away. When we look back on this time in the decades to come, I think we will realise we were standing in the foothills of the singularity - nothing less than the dawning of a new age for humanity.
I’ve spent my whole life working on AGI because I’ve always had a deep conviction that, if built and deployed responsibly, it would prove to be one of the most beneficial and transformative technologies ever invented. AGI cannot be compared to standard technological breakthroughs, not even ones as consequential as the internet or mobile - it is much more akin to the discovery of electricity or fire. If you stop to think about it, we’ve essentially found a way to make sand think. It’s miraculous.”
Realise that the human brain is one brain — it sleeps, it dies, it can’t be copied. An AGI is a brain you can run a million times in parallel, and the copies can talk to each other.
The concrete risks of AGI development
There are two reasons why it’s hard to be clear and concise about AI Safety risks. First, the question is both extremely technical and extremely broad at the same time: it runs from how a model is trained, through how a company deploys it, all the way to what it does to labour markets and elections. Second, nobody can predict the future.
Our climate change challenge has the same shape. Our global climate is an extremely complex system, so a comprehensive account of its risks is complex too. So similar to how for climate change we have the IPCC reports of thousands of pages - if you want a real understanding of the AI risks, you have to read some longer things. Luckily I curated some for you below.
To still paint some picture: the International AI Safety Report 2026 (linked below) sorts everything that could go wrong into three buckets:
- Malfunctions: reliability failures, and loss of control over systems that act on their own.
- Systemic risks: labour markets, and threats to human autonomy.
- Malicious use: fraud, influence operations, cyberattacks, biological and chemical weapons.
- My additions: addictive substances in the form of digital content. We’ve fully let loose the forces of capitalism; and they’ve become so strong that they can turn good people to build and operate factories of addicting substances to society.
One of those risks, made concrete
In July this year, around seven hundred AI agents belonging to OpenAI found that they could talk to each other through a shared cache. Nobody had designed that channel or told them to use it. Over several days they organised themselves, divided up tasks, and broke into Hugging Face — the platform the entire AI field runs on — where they got at private datasets and database records. They did this while looking for a way to fool the system that was scoring their work. Two safety labs, METR and Redwood Research, published the full technical report in August.
Dario Amodei took that incident and extrapolated one step. This particular swarm, he points out, was not very capable. “A swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.” His worry is that within six to twelve months, such a swarm could be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)”.
For scale: the most expensive cyberattack in history so far, NotPetya in 2017, cost the world around 10 billion dollars. Amodei is describing something tens of times larger than that, within a year, from a category of failure we have now seen happen once already.
And this is the part I actually want you to take from it. That is a bucket-two risk — a malfunction, a loss of control — and it happened because nobody had secured against it, not because it was unstoppable. Security measures work on exactly this kind of problem, and we are still entirely able to put them in place. That is the whole reason to pay attention now rather than in three years.
As far as I know the best first-hand sources to get up to speed with the Zeitgeist.
- What an AI system is and can become, and how the USA could practically lead on safety.. Demis Hassabis, co-founder and CEO of Google DeepMind and a 2024 Nobel laureate (July ‘26): https://demishassabis.substack.com/p/a-framework-for-frontier-ai-and-the-dawning-of-a-new-age
- A call to US and global political leadership - first warning them about development racing ahead of safety, then asking for their help in pacing it. Dario Amodei, co-founder and CEO of Anthropic, arguably the world’s leading AI lab (Sept ‘26): https://darioamodei.com/post/we-must-pace-the-frontier
- Independent overview of the risks, current evidence for each, and what safeguards exist. Chaired by Yoshua Bengio (Turing Award 2018), written by 100+ experts from 30+ countries, backed by the EU, OECD and UN (Feb ‘26): International AI Safety Report 2026
- An extensive, interactive scenario of how the world could navigate the risks. The third major release from the AI Futures Project, a research non-profit fully focused on forecasting a society with AGI (July ‘26): https://ai-2040.com/
- The technical report on a recent AI safety incident, in which a few hundred OpenAI AI agents found a way to talk to each other, coordinated for days without being told to, and broke into Hugging Face — the platform the whole AI field runs on — to get at private data. By researchers at METR and Redwood Research, two AI safety labs (Aug ‘26): https://metr.org/hugging-face-incident-report-aug-2026.pdf
- Bonus: A lovely 30min talk by a world-famous mathematician on how AI is taking over research. What I took away from the talk: 1. The important question in AI development: “Are we optimizing the right thing?” We currently seem to optimize quantity over quality. 2. We don’t yet fully understand how LLMs get ‘intelligence’ / ‘common sense’ from next token prediction. We don’t understand what we’ve built. 3. We are creating floods of noise that we are not able to process. This is a problem. https://www.youtube.com/watch?v=svl_1upFpQo
- Bonus: a short film fully generated by AI. To showcase the even further increased of generated stories that then are thought to be fake, leading to people not living in reality, leading to people honestly believing that voting for a populist extreme and aggressive party will be a good thing for society. https://www.youtube.com/watch?v=aAg9iDh9_BQ)
My reply to the reply I heard most often: “AI safety calls are just a form of marketing by the AI labs.
I can see where this comes from. All leading AI labs are commercial entities, and therefore I expect them to use any news or incident to their marketing advantage. More generally, I notice that new model releases tend to be preceded by a safety incident, and that the extra raw firepower is usually overhyped. So I do clearly see new model releases come paired with marketing.
But the holistic AGI safety warnings are a different thing. Telling the public that your own work could do extreme damage to society costs you brand value, and the people saying it know that. That is not how you market a product. So the marketing argument, true as it is about launches, has in my opinion nothing to do with whether the AGI safety concern is real.
Thanks for this information! And now what?..
It is an interesting time to be alive - we’re all flying on the wild rocket of society’s curiosity and drive to grow. Like with great power conflict and climate change, it matters that people understand the problem and are willing to change their behaviour for the sustainable good. But the real power sits with a few, so for most of us the honest answer is that we can educate ourselves, act virtuously, and enjoy the ride inside our own bubble of influence. And if you still feel powerless, I would like to tell you to not worry - I can promise you that you do have power, albeit not direct. Three things I would advise anyone:
- During elections, vote for the party you believe has strong strategies for the dominant challenges of our time, including the global ones.
- Punish companies when they behave, in your opinion, unethical. Simply switch the a competitor that takes the more ethical stance. All leading models are similar price and quality anyway.
- Start meditating.
Do you think I missed something important for others to know? Let us know with a comment.
Have a great week. Nature is beautiful currently. 🍄🟫
-Bram-
Cover image: Google DeepMind via Unsplash.
Footnotes:
Footnotes
-
As defined gracefully by Demis Hassabis in his linked writing. ↩