Spacemesh in the Wild
Three Things #89: October 15, 2023
The difference between theory and practice has always fascinated me. It sort of blows my mind that, no matter how powerful our computers and how sophisticated our algorithms, it’s impossible to simulate the way things actually work, even relatively simple things. You can’t create perfect laboratory conditions to test how something will actually work in the real world no matter how you try. The thought is tempting and lots of smart, well intentioned people have tried many times in many fields, but it just isn’t possible. Theory and practice just aren’t the same. It’s like an immutable law of the universe.
What’s even more interesting, you can’t perfectly explain why. Because if you could explain the difference, i.e., if you could explain why your theory is wrong, then by definition you could fix your theory. And you cannot. So by definition your theory is actually wrong and by definition the reason is unknowable. It sort of breaks the brain, doesn’t it? A bit like Gödel's incompleteness theorem. I wonder if there’s an incompleteness theorem of theory.
One reason theory and practice differ is the observer’s paradox. This is true of all software and of all products more generally. Your users won’t behave the same way during testing that they do in production. In theory, there is no user. In practice, there is, and the simple fact of adding the user already means your theory is incorrect and incomplete. If there’s one thing theory is bad at, it’s predicting human behavior.
How does this relate to Spacemesh? We spent years designing, building, and testing Spacemesh in the lab. Then we finally launched it a few weeks ago, and we’ve learned a great deal since then. Here are some of the things we’ve learned and some of the things we got wrong—i.e., ways in which our theory didn’t line up with reality in practice.
Thing #1: P2P is Hard 👬🧑🤝🧑👭
“In theory, theory and practice are the same. In practice, they are not.” - Einstein
In theory, a network is a graph with nodes and edges: in our case, node software and P2P connections among them. Nodes have properties such as computational power and disk space, and edges have properties such as bandwidth and latency. It’s pretty easy to simulate such a graph, to build a simulator with many nodes that pass information around in peer to peer fashion.
In practice, however, not all nodes are created equal. Nodes in a real P2P network have all sorts of properties that aren’t captured by the reductionist model described above. Most importantly they have operators: human beings that get some things right and mess other things up. Humans that are running other applications on the system that’s also running the node. Humans that sometimes follow advice and do what they should do (from the system operator’s perspective), e.g., keeping their software updated, but more often do something else, something totally unpredictable. Sometimes they make strange changes to config settings. Sometimes they run many nodes on a single system, and occasionally they even run multiple instances of the same node at the same time, even though this is dangerous and you make it very difficult for them to do this. (Real humans have an uncanny ability, it turns out, to remove obstacles and safety checks and do just about everything you expect they never would or could.)
Every application on the Internet relies on exchanging data with another system. Most applications are client-server and most servers are pretty reliable: they sit in a data center on a fast, high-bandwidth Internet connection, they don’t run lots of other applications, and they’re managed by professionals.
None of this is true for a P2P network. Running P2P networks is hard for all of the reasons described above. The nodes are horribly unreliable. They come and go. They have unbelievably bad Internet connections: high latency, low bandwidth, and sometimes they disappear when it rains (seriously). Sometimes they have good days and sometimes they have bad days. All of this is nearly impossible to simulate in the lab, and it’s therefore impossible to know how your network will perform in real world conditions, especially when you have fragile things like consensus mechanisms with synchrony assumptions.
It turns out that despite originally being designed as a P2P network, the modern Internet was built for client-server use cases, not P2P. Most ISPs make it very difficult to run P2P applications at home, and some make it borderline impossible. They block inbound connections. They rotate IP addresses. They engage in a horrible, disgusting, no good, very bad practice called CGNAT that breaks the most basic tenets of the design of the Internet (that each node has its own IP address) and make running P2P infrastructure extraordinarily difficult. I’d never heard of CGNAT a few weeks ago; now I know more than I’d like to about it, and I wish I’d never heard of it. Various techniques such as hole punching exist to work around these issues, but in our experience even these don’t work in many cases.
To some extent all P2P networks, including other blockchains, run into these same issues. But most other blockchains don’t rely to the extent that we do on ordinary people running nodes on ordinary home computers and ordinary home Internet connections. Spacemesh is truly P2P in this sense, in the original sense. Most Bitcoin and Ethereum nodes are hosted on cloud infrastructure providers or in data centers, and for dPoS networks like Solana basically all are; as far as we can tell very few Spacemesh nodes are. This is a really, really good, exciting thing, since the goal of Spacemesh is maximal decentralization, but it makes running the network very hard, much harder than I expected. We didn’t expect that we’d be spending as much time troubleshooting the P2P network itself as we have been; we expected to be spending our time higher up the stack, on the protocol. Hard lesson learned: you simply cannot take the network layer for granted in the case of a P2P network, especially a really decentralized one with many home users.
For a long time I’ve long had an idealistic vision of a swarm of ordinary home nodes coming together to give rise to a newer, better Internet, starting with P2P projects like Spacemesh. I now understand that that goal will be significantly harder to achieve than I had imagined, because the network itself is so hard to get right and because we’re fighting an uphill battle against an Internet designed to serve us content rather than let us host it ourselves.
The upside is that, once we do figure out the network, we have an extraordinarily powerful new type of infrastructure that we can build incredible things on top of. We’re off to a great start.
Thing #2: Classes of Users 🐟🦈🐋
“Everyone has a plan until they get punched in the mouth.” - Mike Tyson
One of the most important things you need to do when building a product, and also one of the hardest, is knowing your customer. It sounds very Paul Graham-y but it’s true: if you had to come up with a single factor that differentiates successful startups from unsuccessful ones, you could do worse than: how well do they know their customer?
This is another important difference between theory and practice. In the lab, by definition, there is no customer. There is no user other than you, the designer/builder. Incidentally this is why it’s often a very good idea to build things for yourself and people like you, but this isn’t always possible. It’s also why it’s important to launch your product as early as possible, even before it’s ready, and to get it into your customer’s hands as soon as possible. Naively, this is so you can learn their preferences; in fact, it has at least as much to do with figuring out who your customer is in the first place. WIth very few exceptions, startups tend to get this wrong initially.
We got this wrong initially. As I explained previously, we envisioned two classes of Spacemesh users, at opposite ends of a spectrum. At one extreme we expected there to be small home miners, otherwise known as guppies or minnows. We expected these to be overwhelmingly young gamers, with some degree of tech savviness but without coding or devops expertise and without lots of resources. At the other extreme we expected some whales: large, professional, industrial operations with lots of resources and equipment.
What we didn’t expect is that the largest contingent, as far as we can tell, is somewhere in between: hobbyist home miners, affectionately known as sharks. They’ve invested a lot of time and money in setting up powerful mining rigs for other networks and they’re interested in Spacemesh to turn a profit but also as a learning experience, since the requirements to mine Spacemesh are different from other networks and are fairly unique. I’m thrilled that these users are active on Spacemesh and I’ve really enjoyed getting to know them on our community Discord, but we didn’t consider their needs very well when designing the Spacemesh protocol and our software. That’s led to a frustrating user experience for them, and has caused some to join pools.
Could we have done a better job of forecasting who our users would be and what their needs and preferences would be? Possibly, although it’s not totally clear how we would’ve done this. Most of us are not product or sales people and, even if we were, we didn’t meet many of these “shark” users at events or meetups over the years. A few have been with us for a while throughout many generations of testnets, but most showed up when the network launched and the economic incentives appeared.
The good news is we know who are users are today, and we like them a lot. We’re super happy they’re part of Spacemesh, and we’re belatedly doing as much as we can to make their lives easier. For me this is one of the most joyous aspects of practice, as opposed to theory, and ultimately it’s the reason I’m an entrepreneur, a builder, and a software developer (as opposed to a researcher or an investor): I want to build things that real people use, and I want to meet those users face to face and get to know them. We’re well on the way to doing that at Spacemesh.
Of course, this will likely change over time. A blockchain is a strange beast, not exactly a company and not exactly a product, and we’re still figuring out what product market fit looks like, what sort of applications will be built on Spacemesh, and who our users will be in the long term.
Thing #3: Pools and Canaries 🐦🏖️💰
“The best-laid schemes o' mice an' men
Gang aft agley,
An' lea'e us nought but grief an' pain,
For promis'd joy!” - Robert Frost, To a Mouse
I’m a huge fan of testnets. I’ve probably made more transactions on Ethereum testnets than I have on mainnet. As a developer, it’s really important to test your applications end to end on a testnet before deploying to mainnet, and the more realistic the testnet, the better. One of the great things about Ethereum testnets is that they’re really busy: they have millions of transactions and very complex state, and they’re long-lived. This means they’re very, very similar to mainnet and an app that works on a testnet is almost certain to work on mainnet.
But there’s one important difference on a testnet: the coin, by definition, doesn’t have any value. If it had value, it would be a mainnet and not a testnet. The whole point of using a testnet is that you can test things for free in a low-stakes environment. (There is of course something in between known as a canary network, but for the purposes of this conversation I think the binary distinction is more useful. A canary net is basically another mainnet if the coin has nonzero value.)
Spacemesh was in a testnet phase for over a year. During this time we saw a lot of interest in the project and thousands of users tested our software, but very few stuck with it. I don’t blame them because it wasn’t easy to run and, again, the coins they were earning had no value. (It still isn’t as easy to run as it should be, but it’s easier than it was and the coins do have some value now, so most miners are sticking around, which is exciting.)
This is a perfect example of the theory-practice dichotomy in action: you cannot have your cake and eat it too. Either you test on a testnet where the economic conditions are totally different than on a mainnet (e.g., slashing or other forms of economic punishment don’t actually matter because there’s no real value at risk), or you test on a mainnet where the economic conditions are realistic but your tests are expensive, possibly very expensive (especially when the network is busy and gas is high). A testnet is like playing poker for peanuts—which, for those who have never tried, doesn’t work very well. Take my word for it.
We had no idea how the Spacemesh economic incentives would work while we were in testnet mode. It turns out that the people we thought would be our miners, those elusive minnows and guppies, were not the majority once the mainnet launched. As described above the main class of miner is actually the sharks, i.e., the enthusiast home miner. And the other big class of user is the pool, which contains many sharks.
This has resulted in two things that we didn’t forecast. The first is that the network has grown much bigger much faster than anyone forecast. We’ve only been live for a couple of months, we’re in a deep, long, dark bear market, and we already have tens of thousands of miners on the network. There’s around 30 PiB committed to the network, a figure that’s growing steadily. On the face of it this is fantastic.
But it’s not quite that simple. As I wrote about last month, users in a pool are not exactly the same as ordinary home users. For one thing, they’re not actually running their own consensus clients. This means that, while they are contributing to the security of the network, they’re not contributing to the decentralization of the network.
For another, many or most pool users aren’t values aligned with the project. To be clear, this is okay. It’s just economics at work. As I’ve said many times, Spacemesh is open source, permissionless, and censorship resistant. Anyone in the world is free to mine for any reason, and there are clear economic incentives to do so, by design. But these users aren’t Spacemesh users, per se. Rather, they’re pool users who were mostly already running the pool software and just flipped a switch to allow their storage space to be dedicated to Spacemesh rather than another protocol. Many have probably never been on our Discord or read any of these articles or any of the articles in our blog. Many are probably totally unaware of the Spacemesh mission, vision, and values.
We didn’t account for that, and we haven’t fully figured out what this means for the project, the protocol, and our software. We’ll continue to improve the UX for miners, and on top of that we’ll be adding additional incentives to the protocol to encourage users to run their own nodes at home. This might have the effect of convincing some pool users to run their own nodes and manage their own keys, which would be great for decentralization and for the health of the protocol. Many likely won’t switch anyway, and that’s okay too. There’s room for many classes of users at Spacemesh and we need to do a better job of balancing their needs.