More Human, Fewer Humans
The promise of AI is seductive: all the benefits of human interaction without the inconvenience of another person. But what do we lose when we accept that bargain?
I read a sci-fi novel a few years ago about a near-future world with incredibly advanced nano-tech. In that world, there was no need for doorways: the wall would simply let you pass right through and re-form around you. Same with furniture: whenever you needed a chair, one would just appear before you, and would mold itself to fit your body and your posture. I highly doubt I’ll see this in my lifetime, though I do think this technology will eventually exist. For now we’ll have to make do with architecture that does its best to predict our desires, does an okay job of putting doorways and chairs where we need them, and is totally inflexible when wrong.
The same has always been true of software. Like physical world architecture, software architecture and UX design has always been about building virtual spaces for people to engage and interact. And just like physical world architecture, digital architecture was always subject to more or less the same set of limitations. Designers do their best to understand the user’s needs and build a space around those needs. When things go wrong, the user’s only recourse is to ask for support.
I was reflecting on this today as I was playing around with several sleep tracking apps, which I’ve begun using lately in an effort to improve my sleep. Ironically, I’m using them because AI suggested them: they’re the gold standard in sleep tracking and implement a strict, battle-tested clinical protocol.
While helpful, that protocol is precisely the problem. Software, at least the way we’ve always conceived it, by definition has no recourse to deviate from the protocol, i.e., the code as written. Generally speaking that’s a good thing: protocols are the reason the modern Internet, and software more generally, work. And applications like games and spreadsheets work pretty well, despite being inflexible.
But it’s a big issue when it comes to things like health. I felt myself becoming extremely frustrated trying to answer the questions the sleep apps were asking me: What time did you get into bed? What time did you begin trying to sleep?
Those questions sound simple enough on the face of it, but how do I explain to the app that, well, yes, I got into bed at 9pm, but my four-year-old son was jumping up and down on top of me for the next 20 minutes? When did I actually “begin trying to sleep,” according to the app and the protocol? How do I explain to the app that my sleep schedule was thrown off by travel last week, but it was an anomaly and we should ignore it? How do I explain that I have to wake up earlier one day a week to join an early group workout, and why this is so important to me and worth deviating from the protocol?
In their defense, all of these apps say the same thing: that you should be working with a real clinician, and that the apps are only intended as one simple tool. Fair enough. A doctor wouldn’t be surprised to hear about jumpy four-year-olds or morning workouts, and would take them in stride, factoring them into our sleep plans as needed.
Traditional, inflexible software can’t do this—but AI tools can. I’m testing the sleep apps so I have a baseline understanding of the protocol and how they work, but I’ve already built a much more sophisticated sleep tracking tool in the form of an AI agent, and I’m relying less on the apps and more on the agent every day. The tool is conversational: my smartwatch tells my agent when I wake up, and then my agent casually messages me and asks how I slept. I can respond with a sentence or a paragraph, depending how I feel, and the agent takes it in stride. I can say “issues with my son again” and it knows exactly what I’m referring to. Then it reviews my planned workout for the day, calculates a “readiness score,” and suggests possible tweaks based on both subjective and objective health data.
This got me thinking about the overall shape of software and about the risks and opportunities associated with software becoming more human-like. Software ate the world, AI is eating software, and to the extent that we all rely heavily on software in our daily lives, it matters very much how, and for whom, that software is designed. It matters where and how decisions are made, how conflicts are resolved, and to what extent there are humans in the loop when that happens. There are bitter lessons to be learned here from previous paradigm shifts, such as the shift from Web 1 to Web 2.
More Human 🎭
Software has been written a particular way for a long time. Traditional software requires rigidity of both the developer and the user. It requires the developer to resolve nearly all ambiguity before an application is shipped. And it attempts, imperfectly, to fit the user into one of a specific set of rigid categories, perfectly represented by the options in a drop down box.
There are good reasons for this. Most obviously, it makes the problem of writing software tractable to begin with. Computers are machines that are designed for running repetitive, deterministic programs that are clearly and completely specified. Computers and programmers have always struggled with randomness and ambiguity since they violate some of these fundamental principles. There have been niche experiments here and there over the years to build “fuzzy” software that didn’t quite follow these rules—fuzzy logic, natural-language interfaces, even the famous 1960s chatbot ELIZA, which held surprisingly plausible conversations without understanding a word—but until quite recently they mostly produced games and toys, and not much that was genuinely useful.
There’s another reason determinism is useful. It allows a system to make credible commitments. These commitments are embodied by protocols that allow systems and users that don’t know each other, don’t trust each other—and may not even live in the same country or speak the same language—to transact and communicate with one another. Modern internet protocols allow computers all around the world owned and operated by billions of different people running different software, using different internet services, etc., to communicate and exchange data. Bitcoin takes this a step further by allowing people around the world who don’t know or trust one another to not only exchange information but to exchange value in precisely the same fashion—another huge breakthrough.
I’d argue that global protocols like this are one of the biggest, most positive, most impactful achievements of the previous era of computing. That we’ve been able to develop, maintain, and govern these protocols as long as we have is a truly remarkable human accomplishment, and that we’d be able to do this isn’t obvious in retrospect. It wouldn’t have been possible without deterministic software! Bitcoin works precisely because it’s extremely rigid and inflexible. It has only a handful of different transaction types and an intentionally crippled scripting language. There are only so many ways to express yourself in Bitcoin, by design: it makes the system simpler, more secure, easier to reason about, and more efficient.
But as the example above illustrates, rigid software isn’t always optimal. In reality, people don’t fit into neat little boxes. There are always exceptions: a fixed form can’t understand a four-year-old jumping on the bed, the demands of a business trip, the importance of a weekly morning workout, or why you don’t have a “permanent mailing address.”
By contrast, AI can hear and understand the big picture. For the first time, software can meet us where we’re at rather than requiring us to contort ourselves to match the way it was designed. In this respect it moves some of the designer’s judgment from design time to runtime: rather than designing dropdown lists, buttons, and form fields, the AI user experience designer is designing rough conversation flows and high level goals and constraints. This sort of “meta design” is much more human than what came before. It’s less like traditional software design and more like preparing for an interview or a debate.
I can’t overemphasize the importance of this shift. When we talk about why we’re excited about AI, we almost always focus on things like raw intelligence and work automation. These are important aspects, too, but what’s at least as important is that AI both enables new categories of software that were never possible before, due to this fuzziness, and also, at least in theory, makes all software more human and more usable. It allows software to be better for the user, not just more efficient.
My experience so far with AI native software has been pretty positive. AI can listen, ask questions, understand context, and generate an interface, data structure, or workflow on the fly. AI support tools, done right, are actually more helpful than most human support agents. They’re immediately available around the clock, have greater context, can respond more quickly, and can help me get to the bottom of issues, or escalate as needed, an order of magnitude faster than a human can. They free me from the frustration of needing to explain the case again and again to a long line of humans.
There’s no doubt that AI makes our software more efficient and flexible, but at what cost? Nothing comes for free.
Fewer Humans 👤
To understand the cost, it’s helpful to look back to the previous transition from Web 1 to Web 2 and the rise of social media. I’m old enough to remember when social media was new and exciting. Being able to see and communicate with your classmates in a digital space, on the first version of Facebook, was legitimately exciting and groundbreaking. The promise of social media was real: democratized access to publishing, frictionless connection, and even unstoppable political organization, culminating in the high-water mark of the Arab Spring.
We all know what came next. All along, social media served its true masters, the tech giants, and increasingly leaned into dark design patterns and attention hacking, giving rise to unfortunate phenomena such as infinite scroll, clickbait, and even intentionally deceptive psychological practices. The profit motive was too great and attention too valuable for there to have been any other practical outcome. And that frictionless political organizing we were promised? We ended up with the dystopian version instead: splintering the commons into endless echo chambers, posing a very real threat to liberal democracy that persists to this day.
But the key thing to understand is that social media didn’t fail us because the technology failed to live up to its promise. Quite the contrary. It failed us because it succeeded so completely and so quickly that society reorganized itself around platforms before we fully understood the incentives and second-order effects. As a result, we found ourselves in the unfortunate situation of having unaccountable, for-profit tech giants and their algorithms increasingly disintermediating old-fashioned, direct human-to-human interaction. Those algorithms intentionally highlight the most extreme, outrage-inducing content because it gets the most attention, and attention is money.
There are lots of parallels to AI technology today. A small number of unaccountable tech giants increasingly controlling the public discourse and perception of the truth. Disintermediation by algorithm. Rapid social reorganization and a struggle to understand the incentives and effects. History doesn’t fully repeat itself but it certainly does rhyme.
But there are some differences this time around. The same technology that makes software feel more human increasingly allows actual humans to be removed from the system. The support representative turns into an AI agent (this transition is largely already complete). The clinician is replaced by a digital clone. The moderator, caseworker, teacher, editor, trainer, and eventually perhaps even judge are, one by one, systematically replaced by systems that convincingly perform their part, are always on, are cheaper and more accessible than humans, and act as if they understand you and are meaningfully able to pay attention to you.
Where social media mediated communication between humans, today we’re increasingly communicating with AI rather than with fellow humans. Where social media algorithms began to control the content we see and consume, today generative AI is not only producing content but reviewing it, interpreting it, responding to it, curating it, even rendering judgment on it. Where old fashioned software forces us into fields, profiles, likes, and data points, LLMs appear convincingly to reverse that flattening by allowing us to interact more naturally.
But the human-like interface may conceal an even more powerful, more sinister intermediary.
The Web 1 to Web 2 transition and the rise of social media is the historical warning that we should heed. As Jaron Lanier so powerfully wrote of that transition, digital systems reshape people and society itself to fit their representations, and concentrate wealth and power in whoever owns and controls the network. The AI version of this dynamic may be subtler and less obvious because the interface feels more open, contextual, and humane, but that softness and subtleness may belie real danger.
Judgment 👨⚖️
A few years ago, while working in Ethereum governance, I began thinking quite a bit about the role of autonomous systems in things like conflict resolution and justice. The context, of course, was blockchain and smart contracts, not AI. At the time the first really complex smart-contract based applications were just emerging, and with the rise of these applications there were very thorny questions about governance.
Blockchains and blockchain applications, i.e., smart contracts, are unique in that they’re by definition and by design outside the jurisdiction of any particular, existing legal regime—in theory if not always in practice. This raises the question: in this space, what’s the final source of authority and law? To whom, or to what, do we appeal when things go wrong?
This isn’t a theoretical question. It’s a question that was very much on the mind of the Ethereum community in the wake of the DAO hack and the subsequent Parity multisig compromises. Is code truly law, as many cypherpunks, and the DAO attacker, claimed, or is there instead actually a social layer that, as in the default world, can render judgment in case of conflict?
The question remains unresolved today. Different communities approach this question differently. In practice, it’s largely moot because for various reasons the vast majority of blockchain projects and organizations have in fact chosen to unambiguously place themselves within one or more existing jurisdictions and subordinate themselves to existing legal regimes.
In a sense that work was prescient, though it wasn’t obvious at the time. Outside of a few niche use cases blockchain-based autonomous applications mostly didn’t take off, but autonomous AI systems are starting to. The nature of the risk is the same, but in some ways the risk today is much greater than it was six years ago. In 2020, autonomous software looked obviously mechanical. It was rigid and could only handle a very narrow set of pre-programmed situations. By contrast, autonomous software today can truly speak human language: the language of understanding, empathy, mercy. To put it in the terms of my previous piece, it can make the autonocratic system feel anthropocratic without actually restoring human accountability.
Are such systems a wolf in sheep’s clothing? Can they be trusted?
Films like Minority Report have shown the risks of automation taken too far, but we don’t need to turn to futuristic sci-fi to see the risks. We can already see them emerging in real-world examples.
In January 2025, in a widely shared case, a student teacher in Illinois became frustrated with a colleague and, in a private message on Snapchat to a friend, joked about whether she should “kill him.” An automated system designed to evaluate threats at schools flagged the message and an hour later sheriff’s deputies, acting on an FBI tip, arrested and interrogated her.
The remarkable thing about this case is the way the human actors—the FBI, the sheriffs, the school staff—were disempowered by the automated system. The humans, who should’ve been the decision makers, arrived with no real context, and not as a result of human judgment. They were acting as the hands of the algorithm: a robot’s certainty wearing a badge.
What’s really shocking is that this dangerous automated system made humans worse at being human. Once the humans were finally looped in, they sorted out the situation quickly and professionally. They determined that there was no real threat. The student teacher was released the next day after questioning and no charges were filed. But the problem, and the risk, is that we’re only in the very earliest stages of the deployment of automated systems like this. Far greater automation is to come. The allure is too great, because of the potential efficiency gain and cost reduction.
This isn’t just about replacing labor, or about making apps easier to use. It’s a much deeper, more fundamental question that gets to the very heart of what it means to be human and to participate in human systems like society and government. Can we trust automated systems, even, or especially, those that have substantial context and feign understanding of innately human concepts like justice, power, and mercy, to render judgment upon us?
We have to be extraordinarily careful when delegating decisions that affect liberty, employment, reputation, access, and dignity. With the authority to make these decisions comes great power, and we must always be extremely mindful about where we install power, and how we constrain it through means like checks and balances, transparency, and accountability. What sort of checks and balances do these systems need?
No system is perfectly designed. There are always exceptions. And when there are exceptions, someone or something has to resolve them. As discussed above, there are clear advantages to AI in such scenarios. It’s faster, cheaper, and arguably even more impartial and less corruptible than a human judge. But my strong sense is that even the most powerful AI systems will never be able to truly understand those innately human concepts, not in a deep sense. Ultimately the AI system, like any system, is just following rules, and these human ideas by definition transcend rules and defy codification.
I concluded six years ago that we still have no choice but to ultimately appeal to human arbiters. That’s as true today as it was then, and arguably even more important. Blockchain spent the past decade experimenting with automated conflict resolution mechanisms, decentralized courts and the like (Truebit, Kleros). The decade taught us that, while autonomous systems can self-manage for very narrow tasks (read: Bitcoin), they abjectly fail at anything more ambitious or complex. Traditional systems of justice and conflict resolution may be slow, expensive, and imperfect, but for now they’re the best thing we’ve got.
Still, AI has a role to play. It likely can help resolve simpler, more well-defined, more common cases. While it should have broad latitude when acting in a helpful, time-saving capacity, it should have sharply limited authority when judging, punishing, excluding, or coercing us. And in every case we still need to be able to appeal to human judgment and authority. Most importantly, as powerful and compelling as AI is, it cannot bear moral responsibility or take real accountability for its actions. For now, and hopefully for a very long time to come, that still requires a human in the loop.
Historical metaphors can also help. The fruit of knowledge delivered what it promised. The cost was transformation, alienation, and exile: after eating it, humanity could not return to its previous condition.
AI is dangerous for the same reason. It works, almost too well. It offers real knowledge, power, convenience, and freedom. We’ll eat its fruit enthusiastically, and society will reorganize itself around its new capabilities. But by the time the costs become clear, “going back” may no longer be a meaningful option.