The Future of AI Is Not (Yet) Written
at least not by Mechanize
(with apologies to Vitalik)
Does history show that technologies are inevitable? Are there some technologies that are just so powerful that we cannot resist pursuing them?
It’s a view of technology that animates much of the leadership at frontier AI companies, such as, for instance, Demis Hassabis’s claim that “the history of science shows that if something is possible, humans will build it“; or Dario Amodei’s recent argument that the creation of powerful AI systems “was probably inevitable the instant humanity invented the transistor, or arguably even earlier when we first learned to control fire.”
Often, the underlying argument isn’t stated or justified in detail. But there are exceptions. For instance, take a short but influential essay from last year by Matthew Barnett, Tamay Besiroglu, and Ege Erdil, of software company Mechanize (’our goal is to enable the automation of all valuable work‘). In ‘The future of AI is already written’, they propose that the future of AI technology is set, and that it leads straight to the full automation of human labor:
Autonomous agents that fully substitute for human labor will inevitably be created because they will provide immense utility that mere AI tools cannot. The only real choice is whether to hasten this technological revolution ourselves, or to wait for others to initiate it in our absence.
As such, Barnett, Besiroglu & Erdil (hereafter ‘BBE’) maintain that the future is already fixed. Due to ‘hard physical constraints combined with unavoidable economic incentives’, humanity will always develop the ‘same technologies, in roughly the same order, in roughly the same way, regardless of what choices we make now’.
In support of the idea that complete technological determinism is some form of historical imperative, they invoke a range of historical cases–e.g. assumed convergence on basic technologies amongst isolated civilizations like the Aztecs and Spanish; the track record of simultaneous invention and discovery; the failure of historical export controls on the printing press or encryption software; the (seeming) failure of attempted restraint around nuclear weapons. On this basis, they conclude that:
Rather than being like a ship captain, humanity is more like a roaring stream flowing into a valley, following the path of least resistance. People may try to steer the stream by putting barriers in the way, banning certain technologies, aggressively pursuing others, yet these actions will only delay the inevitable, not prevent us from reaching the valley floor.
Finally, since ‘we have far less control over our technological destiny than is often thought’, BBE commit their company to fully automating all human labour as soon as possible. Crucially, their argument does so not only in light of what they see as the potential overwhelming benefits of automation, but also because of its inevitability: if they didn’t do it, someone else immediately will; so they are to be absolved of adverse consequences.
Planning a long-delayed deep-dive: the History & Mechanics of Tech Contingency
Big if true. But is this how technology actually works? Is the trajectory of technological development set in, er, silicon? And are we powerless to shape its direction, destination, or turbulence along the way?
The BBE essay struck a nerve, at least in my corner of the internet. It received a remarkable amount of attention and critical response (given its brief length and relatively obscure web placement), including from other self-identified techno-determinists. Some of these critiques have challenged the essay on the grounds of (1) its historical examples; (2) its use of the metaphor of a ‘roaring stream running down to a valley’; (3) its oversimplification of the actual choice space and levers that are available for differential technological development; as well as (4) the authors’ understatement (ironically) of their own replaceability and agency, and their consequent abdication of ethical responsibility from the societal impacts and even catastrophic risks which (they themselves predict or acknowledge) may occur as a result of accelerating full automation.I especially recommend Luke Drago’s Technocalvinism , which ties together historical and normative counterarguments with concrete strategic suggestions for how to achieve differential technology development towards human-augmenting (rather than human-replacing) automation, by leveraging restricted talent and countermarkets. And see also Vitalik Buterin’s critique of the essay, arguing that it represents a form of inevitabilism ‘low in galaxy brain resistance’, and that BBE unrealistically assume an infinitely liquid market where they are fully replaceable if they were to step back (whereas the AI industry today is, if anything, the opposite, highly talent-restrained). [ED: and see also Tao Burga’s Do not Surrender to the Tech Tree, which I have some quibbles with but on the whole really like).
Anyway, 3+ months after everyone else has moved on,In my defense, in the meantime I had another kid, published a book, and finished another one (book, that is, not kid). I’d like to offer my thoughts on this essay in particular, as a rationale for collecting some broader sets of empirical data, and writing up my own reflections on general themes and memes in debates over technological determinism or choice, both in general and in the AI context.
As it happens, I’ve long been enamored of macrohistory, technology timelines and tech trees; some years back I did a lot of (so far largely unpublished) work on the history of abandoned technologies and technological restraint; the epistemology of looking at tech history only through the visible success stories; and what lessons it all might hold for debates around whether or how we could shape AI.See e.g. this brief blog, and the slides to this talk. Moreover, for my recent book,Did I mention already that I wrote a book? I spent quite some time investigating the historical scope of technological contingency, and the conditions for (and constraints on) deliberate technological choice.
So, in this and some future still-in-draft blog posts, I’ll review BBE’s arguments for technology determinism–focusing mostly on their historical claims. In brief, while I think they get a few interesting things right, I will challenge their interpretation of many of the historical cases they present, and moreover survey many (dozens to hundreds) of other cases of technological delay, abandonment, or restraint across history.
In doing so, I hope to give not just a direct commentary and (qualified) rebuttal to this particular essay; but to use it as a jumping-off point for more general reflections about the mechanics of technological contingency; the common epistemic pitfalls we face when we try to track and interpret the historical course of technological developments looking back; and what history does, and does not, say about our leeway to steer the future of AI.
The Inevitability of Tech Inevitability Debates (or: sincerely taking the bait on neo-tech-determinism)
Briefly, though, why does this all matter? A lot of the initial reaction to the BBE essay dismissed it (not implausibly) as an advertising and recruitment stunt by a Mechanize facing a very constrained AI talent recruiting landscape. For instance, Jasmine Sun cautions people not to take the ‘bait‘:(sorry, Jasmine).
Mechanize, … doesn’t go for the splashy ads. They prefer plaintext blog posts, tweeted out as screenshots. … Are they true believers in full automation? Probably. But other incentives are in the mix. There are a million startups creating RL environments, recruiting is really hard, and hot takes are one reliable way to stand above the fray.
However, I think that the essay is worth engaging with on its substance and on its own terms. For one, I agree with Stefan Schubert that it is important to engage with arguments on the merits, rather than with dismissal and sarcasm. This is especially the case because the essay is a particularly frank, straightforward and pure statement of what many authentically but believe (certainly BBE do; but also many others without a direct financial stake in the narrative).
As noted, this strongly informs the worldviews and strategic decisions of the heads of frontier AI companies. Moreover, it is rapidly becoming a dominant expectation in AI, tech and in Silicon Valley, which, as Jasmine herself argues, has begun mixing AI status anxiety about escaping the ‘permanent underclass’ and the ‘great lock-in’ with a form of ‘technological nihilism’.Jasmin Sun: “These are mostly jokes, of course. But they express young tech workers’ latent anxiety about who will win and lose in the age of AGI—or at least, the assumption that there will be winners and losers, rather than AGI bringing about widespread abundance in a massively positive-sum game. Even the people building AI don’t feel insulated from precarity. Nobody knows if they’ll end up above or below the API: whether you’ll be the automaters or the automated. And despite CEOs’ attempts at optimism, I hear the “gentle singularity” and “personal superintelligence” deployed more often as punchlines than earnest visions of utopia.”
Most importantly; this is the domain where I’ve long been very frustrated by the state of the debate on technology determinism, especially as I simply have my own genuine uncertainties.
On the one hand, many determinist arguments–in AI debates, and elsewhere–are genuine cases of what L. M. Sacasas has called ‘manufactured inevitability‘, where people make sweeping claims about the past and about the future of technology, based on a few cherry picked examples and without even a token effort to explore or understand contrary cases. It is genuinely the case that many predictions or forecasts of the future end up having self-fulfilling, hyperstitional effect (whether or not by design). On the other hand, I think that many historians of technology and academics in Science and Technology Studies have gotten too complacent about the terms of the debate, and too used to dismissing any and all versions of technological determinism out of hand, using it simply as a (poorly defined) pejorative, even as they puzzle over the question of why, given that determinist ideas are ‘dead’, so many people continue to find them so plausible.
Yet, while some classic versions of technological determinism are clearly implausible, others, I think–such as Dafoe’s military-economic adaptationism–offer at least more interesting accounts of how technological determinism might work, that warrant substantive discussion. As I will argue, I still don’t think these forms of determinism support the kind of technological inevitabilism and no-agency pessimism that’s at stake in the BBE worldview. However, it does highlight the exact challenges we face in attempting to influence the course of powerful technologies; offers ground for more useful exploration over when and where we have greater or lesser scope for exerting technological choice; and how we should view our current situation with AI.
So, yes, let’s take the bait.
Setting out the Mechanize claims, and scope of my critique
As I understand them, the basic macrohistorical claims underpinning the BBE argument are, roughly, as follows.
Tech Tree Choice Dependency: technological development is like a ‘tech tree’, with inventions dependent on previous innovations made and available; this strongly constrains the sequence in which technologies can be (globally) developed, and therefore the actual range of technological ‘options’ open to societies at any point;
Spontaneous Invention: technologies routinely emerge ‘spontaneously’ after they are possible (i.e. when the necessary industrial and material conditions are in place), as shown by the historical record of simultaneous inventions;
Convergent evolution: historical development of technology is like evolution, which implies convergence towards one particular optimal solution, as shown by repeated cases of convergent evolution in biology
Historical technological convergence: there is often only one efficient technological method of solving societal problems, as shown by isolated civilizations converging on the same basic technologies (modulo the availability of local resources) independently
Futility of restraint: we do not control our technological trajectory: technological restraint is not possible, especially when technologies provide unique, irreplaceable capabilities; this means that all unique, irreplaceable technologies are inevitable; as shown by the failure of attempted export- (printing press; chips) or arms control regimes (nuclear weapons)
To these historical views, BBE marry a few forward-looking technical assumptions about AI agents, and about the impacts of full automation:
Technical feasibility of full AI agents: AI that could fully replace human labor is technically feasible;
Adoption pressure given economic competitiveness of full AI agents: agents that fully replace human labor would be far more competitive than any conceivable alternative arrangement (e.g. human-augmentative AI)
Societal impacts and cost-benefit distribution from full automation: in fact, full automation is not just inevitable but also desirable (in the long run)
Finally, they involve a few normative assumptions about how we should act if we were to find themselves in exactly the world they describe:
Axiological Destination-ism if a development is eventually inevitable, then there isn’t really much use or moral value in trying to delay it, as that’s ‘just buying time’ (even if we agree it is risky or has adverse side-effects for many).
Incentives excuse all: acting under a very strong set of incentives mostly excuses going along with them; you’re just making the best out of a poor hand.
(Career advice): as such, you should come help Mechanize build this technology, accelerating (and profiting from) its inevitable arrival.
In my coming writing, I’ll initially mainly focus on the historical claims (#1-5), and either grant or (more commonly) bracket many of the technical claims (#6-8) or the normative assumptions (#9-11), although many of those are not uncontroversially or obviously true (see Appendix for some brief reflections).
Open questions about the Past of Technology and the Future of AI
In response, I hope to spend the coming year-or-so to work out some of the following themes in the history of technology:Titles provisional, sequence and pace of posting to be determined.
What kind of tech determinist are you? Is the worldview encapsulated here, and in AI debates, actually ‘tech determinism’? What form? What is the most plausible and interesting account of tech determinism, and what does it imply about these arguments?
Is innovation truly like a ‘tech tree’? If so, what does that imply about technological choice?
Did isolated (premodern) civilizations really converge on the same technologies? Or is there instead a track record of premodern technology pluralism? (~120 counterexamples surveyed)
Why these differences are not (just) explained by environment, available materials, or irrelevance;
Why we can’t (easily) explain away these differences by claiming that technologically diverging civilizations were ‘not true civilizations’. (‘No True Scotland’)
How about modern societies? Are there optimal ‘purity effects‘ to high technology? How often, and why, are modern technologies convergent? What does that imply? (a few dozen counterexamples surveyed)
Is historical innovation really like (biological) evolution? What does that imply about the conditions for technological convergence? If ‘sociotechnical selectionism‘ describes the scope conditions (i.e. scale of analysis) for technological determinism–helping us understand how determinist technology trends might ‘emerge’ through historical competition, then… what are the historical and geopolitical scope conditions for such selectionism to meaningfully operate and drive technology into coherent directions?
Do technologies emerge ‘spontaneously’ once they are possible? What is the actual track record of simultaneous invention?
The Fragile Cultural Roots of the Tech Tree: my turn to think about the Roman Empire (and the Sapient Paradox, the Gossip Trap, and absent revolutions)
What is the historical overhanging fruit on the tech tree? What historical inventions appear unusually delayed? (e.g. the inverse of simultaneous invention). (>25 initial cases surveyed);
Is the AI revolution, in fact, already historically late? Considering historical trends in various inputs and pre-requisites?
When and where have technologies been lost? Is there a record of meaningful technological loss? What does this imply about the resilience of the tech tree? (>45 cases reviewed)
The Shorn Tech Tree: what technologies were abandoned early on in their development, or during their initial testing? Which have not lived up to their societal impacts? Which have receded from use (for reasons unrelated to their obsolescence by substituting Pareto-superior technologies)? (raw list of 500-600 cases across various categories)
Shadow of the tech tree: why is it epistemically and methodologically difficult to accurately identify, evaluate, and weight the importance of abandoned technology paths? How do we accurately evaluate the actual (at-the-time and present-day) technical feasibility and (military or economic) competitiveness of unpursued or abandoned technologies, in a way that avoids hindsight and survivorship bias?
What is the track record of technological restraint? Where have we actually failed? Where have we succeeded? Is this related to whether a technology was ‘irreplaceable’—and what does that mean?
Is full automation the final boss of tech determinism? So, what does the historical record imply?
An orbital slingshot, not a river valley
Practically, the historical track record of failed or abandoned technologies does not mean that full AI agents capable of automating all jobs aren’t a possible node in the space of AI technology (or even a live possibility on the horizon); nor that it will be easy for the trajectory to be bent away from that destination today, given a certain degree of ‘technical momentum‘ has kicked in.For instance, as Harry Law notes: “Political orders embed themselves in the design and allocation of tools, while some tools push back by generating new political structures. We might say the techne of deep learning was socially selected, just as we might acknowledge that our place in the tech tree means the basic shape of frontier AI systems is unlikely to change much in the medium term.”
However, in all this, my view is increasingly that the history of technology and invention simply does not support the view of technology as an inevitable, overdetermined, unstoppable stream. While the history of technology does demonstrate many important trends and structural forces that we deny or ignore at our peril, it also has an important degree of contingency. By itself, historical contingency does not equate to choice; but it affords the space for choice and, sometimes, highlights the machinery for bringing it about.
As such, a good metaphor for technological development is perhaps not that of a river uncontrollably descending towards one central valley. Instead, it is better analogized as a process of orbital mechanics and navigation on a rocket already in flight. Yes, the forces already acting upon us are great and the margin of error small; there are some parts of the engine you no longer can safely swap out; even small shifts in trajectory may take great forces, and there are only some stable orbits you can get to….
…yet, on the scale of planetary distances, even small corrections to our course or speed can amount to large changes in which destination we slingshot to, our speed upon arrival–and whether we’ll arrive well or instead overshoot our target or burn up on re-entry.Yes, this is just a metaphor. But that part of an argument that is asserted by analogy can be countered by analogy.
Appendix: thoughts on the technical assumptions of the full automation scenario
On the technical feasibility of full automation (#6):
although there are significant outstanding cruxes over whether AI progress might be fast or slow, and whether LLMs could be a road to AGI, I generally actually concur with BBE that very transformative AI capabilities (e.g. ‘full AI agents’) are possible at least in principle; and that something like ‘full automation’ would, in principle, be a coherent destination on the map
However, I expect that near-term progress towards this will be somewhat jagged and uneven: for whatever reason, LLM-based agents continue to encounter many performance challenges in real-world settings (e.g. achieving an automation rate of 2.5% in one experiment, etc.). Moreover, we have reason to at least be sceptical about what is indicated by most (!all) benchmarks, given e.g. risks of hidden benchmark contamination and lab incentives around how they construct and present them.
Critically, Mechanize is betting that generalisation from training LLMs with reinforcement learning on tasks with a verifier (e.g. maths or programming) will generalize to good performance on tasks without such an easy verifier (e.g. writing the best ever novels, etc). Indeed, this is one of what Helen Toner has flagged as one of the two big questions for AI. Yet to date, there is no clear sign yet of such generalisation.
To be sure, I don’t think arguments grounded in the near-term limits of LLM-based AI agents are decisive evidence against BBE’s thesis: I am given to understand they have longer-than-usual ‘timelines’; and at any rate, all they are claiming is that this technology will emerge eventually, not that it needs to do so within the current paradigm. Even so, this situation creates at least some problems for any inevitability argument, since under longer timelines (and more roundabout development trajectories) towards full AI agents, we will have additional opportunities for leverage and trajectory shift.Also, under a longer timeframe to full automation, the case for betting your career on *Mechanize* being the specific firm to outlast all competition and deliver this solution looks worse, all else equal, as a trajectory with a more staggered or uneven uptake may see intermediate rounds of investors losing their nerves and/or the start-up of more- and more well-resourced competitors.
On adoption pressure and economic competitiveness of full AI agents (#7):
- I don’t have strong views on this; but will note that the trend in increasing length of task performance shown by METR, appears to also show that the per-’hour’-task inference costs of AI agents are rising exponentially with them–in some cases to ‘hourly’ costs that are close to, or even in excess of, the rates you would pay humans.Toby Ord: “For instance, Grok 4 is at $0.40 per hour at its sweet spot, but $13 per hour at the start of its final plateau. GPT-5 is about $13 per hour for tasks that take about 45 minutes, but $120 per hour for tasks that take 2 hours. And o3 actually costs $350 per hour (more than the human price) to achieve tasks at its full 1.5 hour task horizon. This is a lot of money to pay for an agent that fails at the task you’ve just paid for 50% of the time — especially in cases where failure is much worse than not having tried at all.” While these costs might fall over the time with the fall of compute, it implies, at least, that real-world applications of AI agents–and the arrival of a stage of full automation of the economy–will lag significantly behind the capability trendline.
On the societal impacts and cost-benefit distribution from full automation (#8)
I’ll maybe explore this more in a future post, but in brief:
I expect both the trajectory and destination to look more confusing and strange than BBE make out by a basic comparison. Putting aside the fact that predictions of LLM’s job impacts have been quite challenging, it may also be the case that attempting to derive predictions about the labor market- and societal impacts on the basis of raw capabilities is senseless since, as Jimmy Alfonso Licon suggests, a Hayekian view of the economy implies that “predictions about the labor market are attempts to describe a future knowledge landscape that requires knowledge that does not yet exist but will be generated by millions of small decisions in the future.”
Personally, I am not intrinsically married to a role of work in the future. e.g. I have found John Danaher’s Automation and Utopia to offer a compelling, appealing account of how we could organize a post-work, post-scarcity society in a manner that would offer people a good life, with new sources of meaning. However, that does not mean that such a Banksian Culture outcome–of fully-automated luxury space anarcho-communism–would automatically follow from full AI automation of all jobs: post-work does not necessarily entail post-scarcity.
I’m not so well positioned to add to the live and interesting debates over what would be the human impact of full automation (whether direct, or indirect e.g. through its effects on political leverage). Instead, I’m mostly curious to focus on (a) the mechanistic claim of technological inevitability, and (b) its assumed historical underpinnings. After all, even those who (like Scott Alexander) are more sanguine about the future conditions of life in the ‘permanent underclass’, still expect that all roads will converge on the attractor of full automation.
Nonetheless, as other responses have noted, some of the Mechanize authors themselves have painted a picture of rather significant, even existential economic and societal upheaval for most people, even under best-case scenarios. For instance, Barnett’s own previous research (published as late as one year ago) estimates that the arrival of AGI will lead to below-subsistence wages for people within 20 years (1/3 chance) or within the coming century (2/3 probability), although he argues that this will not necessarily mean a crash in the quality of life, so long as people could receive other sources of income to sustain themselves, such as dividends, government welfare, and charity.In fact, in another post, the Mechanize authors aim to explore how exactly life after work might still be good; though the argument partially invokes a historical comparison to child labor that I think (a) somewhat misstates the established scholarship on the historical sequence in which the UK passed legal restrictions on child labor (vis-a-vis the contemporary rate of automation & wealth), and (b) doesn’t really engage with the continued prevalence of child labor in many developing countries today, even those that have surpassed the average level of wealth of 19th century societies. Elsewhere, though, Barnett notes he still believes there is a 1 in 20 chance of AI killing everyone,Again, in another blog post, BBE push back on recent arguments for AI existential risk. It is unclear to me whether this represents an update in their estimated probability, or whether the 1/20 estimate has already priced that in. Though, ultimately, I’m less interested in exploring this risk angle here. and that it is “highly likely” that humans would at some point trade with AIs that do not share our values.
More generally, even if Mechanize might be right to claim that full automation is ultimately desirable and worth any intermediate job losses, there is still something weird going on: if that is the actual crux of the debate, then why frame the original essay around a strong claim of the inevitability of this development (a claim which, if taken at face value, could imply that you also aren’t to be strongly credited for bringing it about)?
Acknowledgements
My thanks to Janna Tay, Luke Kemp, Gavin Leech, Jacob Slosser, and Rune Nyrup for helpful feedback on ideas, essays, and cases in this series.
Comment on Substack · Subscribe to Critical Maas · All essays