Fixing the Web, Part III
Three Things #103: January 21, 2024
This is part three in a series on the biggest ways in which the modern web is broken. We’re nearing the end of the list and we’ll be talking about solutions soon, hang in there!
Thing #7: Complexity 🪢
When I think of the complexity of the modern web two remarkable anecdotes stand out.
The first is an article that I read about three years ago. It was written by an experienced web developer who attempted to document every step in the process of setting up a modern website: all the tools, libraries, and platforms, all the different technologies, and all of the decisions that had to be made (see image above). It was written in an extremely detail-oriented, pedantic fashion, and the responses on Hacker News were mostly people trying to figure out whether the article was serious or tongue-in-cheek. I read the whole thing. It was both. Everything included in the article was really a part of setting up a modern website. And I’m certain things have only gotten worse since it was written a few years ago.
The second is a post on X that I read a few months ago about the total size of the specifications behind the web (unfortunately I’ve lost the link but will update this if I find it again). How big do you think it is, in pages? It depends how you count, of course, but if you include just HTML, JavaScript, and CSS it’s on the order of tens of thousands of pages, orders of magnitude more than I expected. This post pointed out that it’s orders of magnitude greater than the US tax code, which is famous for being massive and overcomplex. Frankly I don’t understand how a spec for anything can be this complex, but there you have it: the modern web in all its complex glory.
As I keep saying in this series: it wasn’t always this way! I’m old enough to remember when setting up a website really just meant one or a small handful of HTML documents. You could write the HTML by hand (it wasn’t hard), you could use a WYSIWYG editor like FrontPage, or you could even use a word processor to convert a document into a web page. It felt a bit like editing a wiki or writing markdown today: just text, some basic syntax that was easy to learn and remember, and inter-page links, and everything just magically came to life and started working.
Things are orders of magnitude more complex today. Complexity isn’t always bad and it sometimes buys you nice things, a lesson we’ve learnt at Spacemesh. The early web could display documents with text and some basic formatting, and later images, but that’s about it. “Multimedia” just meant text and grainy, static GIF images. There were no animations and no embedded scripts or apps. Web pages weren’t interactive or dynamic at all; the early web was read only. There were no applications to speak of. Web content was tiny because most people with Internet access (myself included) had puny dialup connections. Browsers were slow to load and render even these simple pages. There were no headless browsers or fancy libraries for consuming web content.
If you started using the web more recently, you don’t know how lucky we are today. Today we have smooth, dynamic, interactive experiences on the web. We have full-featured applications like the one I’m using to write this. We have voice and video calls. We have pages and apps that are responsive and work across a huge array of devices. We have SPAs and PWAs. We have gigabit broadband and streaming. We have CDNs and edge caching and edge compute and a blazingly fast experience. We even have full 3D games. There’s almost nothing you can’t do with the web today (P2P and persistent local file storage are a couple of the very few things you can’t do with the web). But the price of all of this is a lot of added complexity.
And the price of this complexity is unfortunately hidden. It’s not something we think or talk about very often. For any web developer under age 30 this is just the way things have always been and, by extension, the way they’ll always be. But the cognitive burden associated with this complexity is enormous. A friend put it best when he said that developers today can’t write apps any more because it’s become too complicated. A few years ago a motivated solo developer could build and operate an app. That’s simply not possible today for all but the very most basic apps. In a sense every web page is now an app and maintaining any app is quite literally a full time job—typically for a whole team! This takes a lot of money, which usually means VC dollars and a business model. A solo developer can’t manage to write and run an app anymore. That’s incredibly frustrating and tragic.
This complexity also severely limits choice and innovation. A decade or two ago there were dozens of upstart web browsers. Today we’re reduced to three: Chrome (Chromium), Safari (WebKit), and Firefox. Even tech giants like Microsoft gave up maintaining their own web stack. To be clear, there are still lots of browsers, but they’re all based on one of these existing stacks because building one from scratch is simply too hard. So there are really only three.
It means that choosing a web browser today is akin to choosing an airline or a mobile phone company: all the choices are sort of unappealing and more or less the same because the startup cost and barrier to entry is too high. When your protocol or standard has gotten so complex that no one can afford to or is capable of building a new implementation, that’s a good sign that something’s wrong and it’s time to start over from scratch.
The web as we know it isn’t going anywhere anytime soon, complexity and all. There’s simply too much invested in it. Projects like Urbit are trying to reimagine the web (and, in the case of Urbit, the entire Internet) from scratch, which I think is a little too ambitious as a starting point. I’d start with a New Web that, as with the early web, just tries to do a few things and does them well—hopefully without most of the problems outlined in this series.
Publishing something on the New Web should feel like publishing on the Old Web: like editing markdown or a wiki. That necessarily means limited styling and limited interactivity but I think that’s a good place to start. Those things can come later, once we get the basics right. Let’s start with books and excellent content, not flashy ads and companies monetizing your data.
It’s possible that a New Web that starts out as simple as the Old Web would end up just as complex as the web but in different ways, but I sort of doubt it. The web itself isn’t going anywhere anytime soon. That doesn’t mean we shouldn’t try to build something simpler and more specialized in parallel.
Thing #8: Privacy 🕵️
In the early days of the web there was no such thing as privacy. Nothing was encrypted. When you logged into a site with a username and password (the only form of authentication available in the beginning) your credentials were sent in plain text across the Internet and could be intercepted at lots of points: by someone else on your same local area network (maybe not so bad at home, but potentially catastrophic at a company or university), by someone “snooping” at the destination network, at any point in between, or in seemingly random places like webserver logs. This lack of privacy made sense in the beginning because computers were only just beginning to be connected to one another and the threat model wasn’t fully understood, and because in the beginning people mostly weren’t passing around sensitive information or doing sensitive things like shopping.
Over time more and more web and other Internet traffic has become encrypted. SSL and TLS were developed and support was added to major web browsers and servers. This was a key piece of the infrastructure that was required in order for online shopping to take off. It took a long time for the standard to become widely adopted, but thanks to recent initiatives like Let’s Encrypt and HTTPS Everywhere almost every web page and application now offers transport-layer security. This is not without flaws—there are still various forms of MITM attack possible, including those that use DNS hijacking—but it’s a lot better than the way things used to be and it makes it very difficult to snoop on the information moving back and forth between your device and the server.
Of course it does nothing to protect the data once it arrives at the destination. There are still a myriad of ways your data can be compromised, and it happens all the time. Really: the list of exploits, intrusions, and data compromises, even just among Fortune-500 companies (which should know better and have the expertise and resources to do better) is depressingly long. Sadly, the vast majority of companies, including tech companies and Fortune 500 companies, and even governments simply do not prioritize network or system security or take these issues seriously enough. Lots of these organizations leave extremely sensitive information including names and addresses, birthdays, social security numbers, credit cards, purchase/transaction history, and even protected health information sitting around on unencrypted, Internet-connected servers. This violates the very most basic security principles, but companies do it anyway out of laziness, a lack of understanding, and bad incentives.
End to end encryption helps but it only really works for point-to-point messaging and file sharing applications like email and chat. With E2E encryption, only the parties to a conversation can decrypt and access the contents. It doesn’t work well for other types of applications where the counterparty isn’t an individual or a small group of people but is rather one of those big, vulnerable servers run by an irresponsible company. And we can’t rely on closed source E2E encrypted apps (which most are) because there’s no way to verify for ourselves that they’re doing what they say they are.
Blockchains take a totally different approach to privacy. Contrary to popular belief account data, transactions, etc. on major platforms including Bitcoin, Ethereum, and Solana are not encrypted. Privacy, such as it is, can be achieved through other means such as not reusing addresses, but it’s nuanced and difficult even for experts, most wallet software isn’t privacy friendly, and most privacy software isn’t user friendly. There are applications and even entire chains that bake in stronger privacy guarantees, but governments and regulators have made using them very difficult due to concerns about money laundering, terrorist financing, and other illicit use cases.
Privacy is very much on the list of “goals that are critical but really hard,” along with most of the other problems discussed in this series. Fortunately, for many kinds of applications, I think less than perfect privacy is fine as long as people know best practices and the software they’re using supports these practices. Perfect privacy is almost never possible in practice but we can still go a long way without it.
A great example is pseudonymity. The worst thing you can do in blockchain is to use a single account for absolutely everything: for getting paid, for gambling, for remittances, for trading NFTs and shitcoins, for joining DAOs, etc. This by definition means that you’re leaking tons of information each time you engage in one of these actions: the way blockchains work today—at least until they offer better out-of-the-box privacy guarantees—everyone who’s party to any of those transactions can immediately see all of the others! A much better approach is to compartmentalize your digital life into multiple identities with a wallet address for each one: e.g., one that you use for your day job, one that you use with your family, one that you use for degen trading, etc. If you do this correctly it offers much stronger privacy guarantees than an “out of the box” blockchain experience.
It can also offer better privacy guarantees than the web. Why? Because the vast majority of web applications today require you to identify yourself: with a name an address, a telephone number, a credit card number, a birthday, etc. Sure, you can create lots of email addresses and in many cases use fake names, but unless you’re an expert these email addresses can be easily traced back to your real identity. And the reality on the web today is that most users are lazy and choose to authenticate with an account on an existing, centralized service (Google, Facebook, Microsoft, GitHub, X, etc.). These companies can literally shut down your identity and block you from accessing anything you’ve signed up for at the click of a button! This happens all the time, without cause or justification and with little to no accountability or means for appeal.
By contrast, in a crypto native world you can (and should!) truly have many different identities that you control. If set up and used correctly, they cannot be linked to one another. Crypto native applications should never require KYC or real world identification—if they do they are by definition not crypto native! If you combine these two ideas you have a very powerful foundation for strong privacy. Add cutting edge technology like zero knowledge proofs and things get even better.
Privacy is a fundamental human right and outside of a very narrow range of sanctioned, high stakes transactions that impact other people (marriage, high-value financial transactions, criminal and judicial proceedings, running for office) you should be permitted to use a pseudonym and maintain your privacy everywhere. (And even in many of those high-stakes transactions you should be able to maintain relative privacy with zero knowledge proofs: by providing only the minimum required information relevant to the application.) You should be allowed to have and transact with as many accounts or identities as you want. If you believe as I do in the rights of privacy and self expression this conclusion follows naturally.
Crypto software can, should, and will make identity management, and by extension privacy, far easier than it is on the web today.
Thing #9: Interoperability 🪡
One way to look at the Internet is as a tool intended to solve interoperability once and for all. Before the Internet it was very difficult for computers to interface and talk to one another. There were a number of different, incompatible protocols and applications. It didn’t happen overnight but eventually the Internet solved this problem through the introduction of standards including Ethernet, BGP, the TCP/IP stack, and the rest of the OSI model. The eighties and early nineties, the period immediately prior to the invention and popularization of the web, saw an explosion of applications that made use of these new standards: email, newsgroups, gopher, BBS, FTP, telnet, etc. Using common protocols meant that for the first time in history any computer (and, by extension, any user) anywhere in the world could talk to any other anywhere else, and be reasonably sure that the message would be received as intended.
In retrospect it’s remarkable how the first decades of the Internet saw the emergence of so many successful, widely adopted protocols: SMTP, POP3, NTP, NNTP, HTTP, FTP, to name but a few. In some senses the Internet would’ve been much better if its development had stopped there or in any case if global protocols such as these continued to emerge, but we all know what came next: centralization in the hands of and even capture by several large companies, as discussed several times already in this series. HTTP is an open protocol but Facebook is not; email is open in theory but Gmail is not. There are open standards for streaming video but TikTok and YouTube aren’t open, etc.—the list of such examples goes on and on.
As these open protocols were gradually replaced by proprietary, closed alternatives—or dominant closed, walled gardens were built on top of them—interoperability and portability became harder and harder. The final push for interop and portability came in the mid to late nineties with the introduction of standards like RSS. RSS put up a good fight but it and similar protocols were eventually killed by social media, as evidenced by the fact that over time more and more websites and browsers have dropped support for it. Sadly, it’s in the interest of social media companies to keep users and content within their own ecosystem: to trap them, raise switching costs, and make it hard or impossible for users to take their data elsewhere or integrate with other applications. As a result there are scant global data standards, APIs can’t talk to one another, and many services provide no way to export your data, or give it to you only in a strange, useless format.
Indeed, as I wrote last week the original killer app of the web was composability: any web page anywhere could link to any other web page permissionlessly. That seems trivial today but it was a huge breakthrough at the time. And as with so much of the web this has gotten harder and harder over time. Walled gardens and paywalls break the composability and openness of the web: you can’t easily link to a Facebook post or a paywalled article. If you do, it often simply doesn’t work or the experience is intentionally terrible outside the “garden” (i.e., the app). Social media apps often do allow linking but they force you to download their app and create an account in order to view content. Some, like X, recently stopped displaying any link metadata and began disallowing embedded links. These forms of hostile behavior are sadly inevitable with the closed, walled garden model.
APIs help a little bit, sometimes. In theory a well designed API allows a clever, motivated developer to build anything at all on top of a platform. It should allow an application to act on a user’s behalf, accessing their data and exporting, modifying, or displaying it creatively. In practice it’s rarely this simple. For one thing APIs are permissioned: you need keys to use them. For another they’re not free (and tend to get expensive quickly). What’s more, they change all the time in arbitrary, unpredictable, unaccountable ways, and the platforms behind them are unafraid of deplatforming anyone they think might compete with them. There are hardly any universal data standards (this is something else that died around the time RSS did) so even APIs from similar services are likely to provide data in radically different formats.
The exact opposite of this is a platform like Ethereum and the radically open, permissionless, composable model that it introduced. The vast majority of smart contracts on Ethereum are open source and publish their interfaces. Any smart contract can permissionlessly, atomically, and synchronously interact with any other smart contract (for which it has an interface) at any time. There’s zero cost to composing such applications (aside from the gas cost, which the user pays for any transaction). This is a utopian version of the web—how the web could and should be if it were designed differently! It unlocks enormous scope for creativity. It’s a major part of what excites people so much about the potential of blockchain and “web3” applications.
The one downside to such a model is that it doesn’t easily scale since it requires a single, global, synchronized database. The web, by contrast, is decentralized and Ethereum is rapidly moving in the same direction with the introduction of rollups and many chains. Even with these challenges I think that Ethereum-style blockchain-based applications will continue to be far more permissionless, composable, and atomic than their web2 counterparts (anything is better than the existing API model!). And the Solana ecosystem is taking a different, more integrated approach that is also promising.
There’s a great deal that the web can learn from blockchains about composability. Open, permissionless composability is the only way to maximize creativity and ensure that the greatest number of high quality applications get built. That above all should be our goal for a modern, reimagined web.