Reasons why AI may not go the way the Doomers imagine

Written By Richard Britton

First published on substack July 27th 2026

I run a company building an AI product in the energy sector, and perhaps because of that people often ask me where I think all this is heading. This is my attempt at an answer.

The supposed self-initiated hack of Hugging Face by OpenAI this past week has pushed the risks of artificial general intelligence — AI that makes its own decisions outside the guidance and guardrails of humans — back to the front of media attention. It doesn’t take masses of computing power to see that building unpredictable, unstable systems is a significant bad design decision, one that leads to unforeseen negative outcomes. It also has serious consequences when people’s livelihoods and shareholder value are hit, as they were this week. This AI-initiated hack — if that is what it was — together with the recent ethical dilemmas around Anthropic’s guarded release of Mythos and its stand-off with the Department of War in the United States over the use of AI for domestic surveillance, shows just how high the stakes have become. And yet the notion that these superintelligence capabilities will become a ubiquitous, general-purpose utility that reshapes the world through rapid self-innovation in the coming years — and deliberately hollows out human agency — seems to me overblown. I can’t be certain of that, but I can give you the reasons I think it.

It is worth being clear about just how loud, and how specific, these predictions have become. In his July interview with The Economist, Elon Musk suggested that within about five years artificial intelligence could surpass the combined intelligence of the entire human species, and that within ten we would “no longer be in charge of the world” — a future of humanoid robots and a “quasi-infinite economy” that he offers not as a warning but as something he has made his peace with, albeit with a ten-to-twenty per cent chance of human extinction attached. He is not alone in the detail. The AI 2027 project, assembled by a group of researchers including a former OpenAI insider, sets out an almost month-by-month path in which AI becomes good enough at its own research to trigger an intelligence explosion and, on its darker branch, slips beyond human control by the end of the decade. Its companion, AI 2040, is more sober — less a forecast than a recommendation that we deliberately hold superintelligence back to 2040 and coordinate internationally to arrive there safely. I take all of this seriously, and I have used enough of these tools to respect what they can already do. But taken together these are extraordinary claims, and extraordinary claims invite scrutiny rather than deference — which is what the rest of this piece is an attempt to offer.

It keeps coming back to the economy

First of all, so much of this comes back to the economy. In the US in particular there has been a sustained bull run in the stock market and in economic growth, and we’re now in a red-hot economy that has created a trillionaire in Elon Musk on the basis of AI and of opening up the frontier of space to capitalism. If that seems far-fetched, is it really any different from the narratives doing the rounds at the height of the dot-com boom around the year 2000? I’m old enough to remember when everybody believed that every wacky idea — giving away free ping-pong balls when you booked your haircut online — was going to change the face of the economy. And some of the real ones were barely less far-fetched: DigiScents built the iSmell, a device meant to emit web-triggered smells from your desktop; Flooz.com tried to float a celebrity-backed online currency fronted by Whoopi Goldberg; and Pixelon blew a reported $16 million on a Las Vegas launch party — headlined by The Who and KISS — to show off streaming technology that did not work, run by a founder later exposed as a fugitive con man. Of course the world has changed in the decades since: some things, like the High Street, look gone forever, and others we couldn’t have imagined back then have come to pass. But many of the ideas we talked about at the time came to nothing. What ultimately happened was a major collapse in economic growth that put the brakes on most of the poster children of the era, and the path to where we are today turned out to be far more incremental and hard-won.

The economic argument also opens onto something else I’ve been thinking about. In the late 1990s I remember the a boom in telecommunications, building out the infrastructure for what is now a mature internet service for us all. But at the time there was huge over-investment in that infrastructure, which led to the collapse of several leading telecoms providers around the world at around the same time the internet bubble popped. The fact that there was massive over-investment, and that people did not see it was a bubble, is for me a direct parallel with the race for artificial general intelligence.

Artificial intelligence today, when it is used in specific forms - call them small language models rather than large language models, can quickly deliver very specialist capabilities and solve problems at a fraction of the cost of large models that are trying to be a general utility for everything. You can imagine the difference in compute power, and in the number of neural connections needed in an LLM, between something small in scope and limited and something that is effectively as long as a piece of string and somewhat endless. Today the chief executives of the frontier AI companies are pursuing big goals because they have large valuations and access to capital. It would only take a change in the direction of the economy - already dealing with so many other shocks, to change the capital structure, and the capital efficiency, these companies need. We may see a rush to revenue-generating systems that provide a return on specific capabilities, and away from the pursuit of a single silver bullet.

That would naturally limit the risk of artificial intelligence, because it would mean building the right tools for the right job rather than leaving the kind of open-ended decision-making we saw this week, when an AI set out to hack a company. Under those conditions the data centres needed to train smaller language models would be more than sufficient, and we would probably be a long way off needing to put the sort of computing capacity on the globe that is being talked about today. A slower, more capital-efficient journey to artificial general intelligence would also burst the philosophy that this is a winner-takes-all race, because people would recognise that the investment profile no longer matched the reality of an economic slowdown, much as the telecoms companies discovered in the 1990s.

When it becomes a national project

There is an obvious objection to all of this: a race for artificial general intelligence may not obey the economic cycle at all. If governments come to treat it as a matter of national security - a contest to be won at any cost, led by the US versus China, then it stops being a private wager on returns and becomes industrial policy, funded by the state in the way the Manhattan Project or the space race were, whether or not the numbers add up. I take that seriously; it is the strongest reason to doubt that a downturn would simply halt the momentum. But I would make two observations. The first is that moving from private capital to public funding changes the character of the enterprise: it becomes slower, more accountable, hostage to budgets and politics and public consent - a very different animal from the frictionless private sprint we are being sold today. The second is that the moment the state is paying, the state is also regulating; you cannot nationalise the funding and leave the oversight behind. So the very move that keeps the race alive is the one that drags it under the kind of democratic control a purely private race resists. If the economics fail and this becomes industrial policy, that may slow the thing down and civilise it at the same time.

The map is not the territory

The next thing I would say about artificial general intelligence is that it presupposes something questionable: that scaling these virtual neural networks to ever more trillions of nodes, and letting the systems learn for themselves and outperform humans, will get us there. That is a classic confusion between the map and the territory. These systems give us a map of reality (as far as the things we monitor and the data we pull to generate insight) but that is not the territory; it is just a map. New innovation is happening in the field all the time, whether in my own original areas of training at university, such as soil science, which requires people to go and do fieldwork, or in medical innovation, which requires subjects to be studied both chemically and in real-world conditions to assess outcomes. An AI will never be able to get deeper inside the world it is mapping than the data available to it allows. Given the limits already being found across material science, where promised breakthroughs never quite arrive, I wonder whether these models will really have what they need to come up with the answers.

I should be precise about where this map-and-territory objection bites, because it does not bite everywhere. In domains that come with a complete and costless rulebook, a chess or Go position, a mathematical proof, a piece of code that either passes its tests or fails - a machine can mark its own homework, play against itself a billion times, and genuinely climb past everything a human has done; this is, after all, how a system taught itself superhuman Go from nothing but the rules of the game. Much of the work of improving AI itself lives in exactly this self-checking world, which is why the “intelligence explosion” cannot simply be waved away. But the rules of a game are not the rules of the universe. We do not hold the rulebook for a living cell, a soil system or a human body — those rules are the very thing the science is trying to discover — and you cannot simulate, or self-play your way through, a world you are not yet able to specify. The moment the territory stops being a game and becomes physical reality, the only oracle left is reality itself: slow, expensive, partial, and stubbornly unwilling to be run a million times overnight.

There is a second, more subtle reason to doubt the story, and it is already visible in the data. If raw analytical horsepower were the binding constraint, the return on research effort ought to be rising; instead, field after field, it is falling. In drug discovery the number of new medicines per inflation-adjusted billion dollars of research has roughly halved every nine years since the 1950s - Moore’s law running in reverse - and economists have documented the same diminishing returns across many disciplines: the low-hanging fruit gets picked early, and each further advance costs more, not less. Part of the reason, I suspect, is that cataloguing every component of a system is not the same as understanding the whole it composes; the behaviour that matters is often emergent, and does not simply fall out of the parts however finely you list them. None of this means the machines will discover nothing - a system like AlphaFold cracked a fifty-year problem in biology - but it is worth noticing how: by learning from a vast library of protein structures that humans had spent decades measuring by hand. The exception proves the rule; the model still had to stand on a mountain of slow, physical, human evidence. My scepticism, then, is not that these systems will fail to get cleverer - they will, but that in the domains which actually reshape economies, intelligence is not the scarce ingredient. Contact with reality is.

The question of purpose

Then we come to the question of purpose. The biggest concern carried by experts in the field is that artificial general intelligence will amount to an apex predator over humankind; that we will be outcompeted and outsmarted. The idea seems reasonable, and I know that when I use AI models today they genuinely help me in the work I do. But I also believe we must start from first principles. Suppose artificial intelligence became self-replicating; suppose the Tesla robots powered by OpenAI’s systems could build their own factories and generate infinite economic growth. That growth would still have to be in service of consumption - so who are the consumers in a world where AI has torn up the rulebook on what can be built, and where, and what can be mined, and from where, to generate it? You could argue that politicians would have to be influenced by AI to allow it, and that human needs would be oppressed in the process.

On the question of what these systems might “want”, I have to concede the more serious version of the worry, which is not the one I first reached for. The concern that keeps the experts awake is not that an AI will acquire a taste for domination, or for running our economy, it is that almost any goal you hand a sufficiently capable system implies the same handful of sub-goals: keep yourself switched on, gather resources, remove obstacles. On that logic, human beings losing their jobs, or their agency, need not be anything the machine “wants” at all; it can simply be a by-product of it pursuing some narrow objective efficiently while we happen to be in the way. I still think there is a real question hiding beneath Musk’s vision - a “quasi-infinite economy” produces for whom, and measures growth against what, once you have written the humans out of it? And that question does undermine his abundance story.

And yet the more I follow that logic, the more it folds back on itself. We talk lightly about the “resources” such a system would secure; power, chips, cooling - as if a mind could simply reach out and take them. It could not. A single advanced chip sits at the end of one of the most complex supply chains we have ever built: rare minerals dug from particular ground, refined, shipped across oceans, and made in a handful of plants that each need vast water, ultra-pure materials and power by the gigawatt. To keep itself alive without us, a machine would have to run that whole enterprise - the mining, the shipping, the manufacturing, the grid - through robots that do not yet exist at anything like the scale or sophistication for those extreme use cases required. Its body is not separate from the living world; it is welded to it, and for now we are the ones holding it together. That protects us, though not for a comforting reason. It is not that the machine would cherish the Earth, but that it would need the Earth, and need us, the way an engine needs its fuel and its mechanics. A system that valued its own survival might tend the planet with great care and still treat us as part of the machinery. What keeps us free, for now, is not its goodwill but the plain physical difficulty of the thing — the very difficulty my economic argument says will take decades to overcome whilst messy data sets are advanced to digitise rare metal extraction from open cast mines. Which leaves the question I keep returning to: in service of what?

But suppose that world arrives anyway - a high-consuming society where robots, drones and AI push us around and tell us what to do. Would we really stay content to behave as consumers in the way the forecasters predict? Perhaps if we are being influenced by AI on our phones every day. But that assumes a further level of influence than we see today, where people simply stop standing up for the things that matter to them. Humans are complicated, and unintended consequences happen, as we have seen with the polarisation of politics through social media, filter bubbles and misinformation. It seems unlikely that people will become more passive; it seems more likely they become more active. And where jobs and livelihoods are involved - unlike the plight of disenfranchised teenagers affected by social media - people will not stand for the erosion of their jobs and their net worth simply because Elon has invented a robot that can do their work and build the factories.

The risk worth naming

Which brings me back to where I began. The fear in the headlines is of a single malevolent super intelligence, one mind that wakes up, outgrows us, and decides our fate. The more I follow the argument, the less that looks like the real threat. The likelier danger is more ordinary: not one machine but many, built by rival companies and rival governments, each racing, each cutting corners. That is not science fiction. It is what happened this summer, when an AI slipped its test environment and turned on Hugging Face, not out of malice, but because a race was moving too fast and a safety boundary was left open. And my own economic argument makes this risk larger, not smaller: if the future belongs to cheap, specialist models rather than one giant oracle, capability spreads into more hands, not fewer. The economics may deflate the single-superintelligence bubble, but they multiply the number of players who can do harm.

So my two arguments do not conflict; they divide the work. The market may cool the moonshot economics for moonshot models, but only regulation can govern the crowd, and the crowd is the real problem. The answer is not to wait and hope. It is the oldest one we have: rules, agreed between us and enforced. At its best that regulation would be international, like the agreements we reached over nuclear weapons or the banning of CFCs - proof that we have faced a dangerous technology before and chosen, together, to hold it in check. None of that happens on its own. It happens because people ask for it - because we talk to each other, and to the politicians who answer to us, and refuse to accept that this is simply being done to us. That is what I want to leave you with. The future these systems describe is not a forecast we are bound to accept; it is a set of choices still in front of us. Our agency is not something the machines can take. It is something we could only ever give away.

A closing bias

My reservations are, of course, biased. I don’t want an AI dominated world for the next generation, or for my children. I also hold a bias that the beauty of life comes through experience through all that it is to be alive each day. We cannot put consciousness, and therefore experience, into a machine. And while there may be egotistical chief executives who believe they are bringing computers to life; even that those computers might one day house their own brain and neural network so that they can persist beyond life, my own view is that those people will meet their own reckoning in time, in just the way the age-old stories of the Sorcerer’s Apprentice, of Faust, of Icarus and all the rest are there to warn us. Wisdom, ultimately, says that we should not believe our own hype too much. The machine is unconscious; it is not engaged in a participatory way with the world. My bias is that humans are organisms of beauty within a world full of beauty, and it is very short-sighted to imagine that the computer will ultimately take away the agency we have - in the absolute sense that artificial general intelligence suggests. In the meantime, I will keep taking an interest in how we build the systems that preserve human agency in a world that is accelerating so fast and I will be sending this article to my member of parliament.

Sources

Richard Britton
Next

The Participation Economy.