Decentralization is (Still) Hard
Three Things #127: July 7, 2024
Decentralization is one of my favorite topics. I’ve written and spoken about it countless times, talking about why it’s valuable—for things like censorship resistance and permissionlessness—and why it’s so tough and fragile. Building solid infrastructure for decentralization, especially for decentralized cryptoeconomic systems of value creation and exchange, has been my focus for the past seven years.
At Spacemesh, we don’t just talk about decentralization; we live it. It’s one of our core values, and we’re committed to the (admittedly long and hard) job of building the most decentralized network. This mission drives many of our design choices and engineering efforts. Today, we’ve got around ten thousand unique miners on the network. I’m proud of that, but we still have a lot to do.
That’s the thing about decentralization: the work is never really done. Like freedom, it’s not a battle you win once and for all. It’s something you have to keep fighting for since there are so many centralizing forces pushing back.
Decentralization is really, really hard. I’ve learned a ton about this since the Spacemesh network went live last summer. Here are some of the things I’ve learned about how to do decentralization, why it’s so challenging, and how we might do better as we move into our second year.
Thing #1: The Goal 🥅
"The greater the obstacle, the more glory in overcoming it." - Molière
Our goal is simple. It's not easy, but it's simple, and it hasn’t changed in years: to have one million distinct miners on the network.
What does “distinct” mean, and why is this important? I covered this in a recent talk, but here's the short version: for real political, economic, and logical decentralization, we need to make sure no single actor or small group of colluding actors can shut down the network, block transactions, or censor others. Plus, we can’t have a single point of failure that could take down the entire network.
The idea of “distinct” is admittedly a bit fuzzy. It'd be great to have many individual Spacemesh nodes running on many computers in many different data centers in many different places, but if all those nodes are controlled by the same actor, the network isn’t meaningfully decentralized. This is why mining pools can be a problem: the storage space (in the case of Spacemesh) might belong to many individual home miners, but if one pool operator controls the nodes tied to that space and decides how they vote, the network isn’t decentralized enough.
On the flip side, a network made up of nodes owned and operated by thousands of distinct actors, all running the same software on the same hardware in the same datacenter on the same Internet uplink, also isn’t meaningfully decentralized. This is due to the risk of common mode failure. A single bug in a single piece of software, or another single infrastructure failure, could take down the entire network. True, robust decentralization requires meaningful differentiation across these dimensions of decentralization and diversity, and many others.
To hit a million distinct miners, we need home miners. Making home mining as easy as possible has always been part of the Spacemesh mission. The entire protocol, from top to bottom, is designed around this idea. I know it’s much too hard to run Spacemesh from home today—more on this in a moment, and on what we’re doing about it—but it’ll get easier. While we’re only about 1% of the way to the goal, I do believe it’s achievable.
For Spacemesh, it’s also key that home miners don’t need to join pools to mine. Even small miners should get rewards often enough that there’s no economic advantage to joining a pool. This isn’t entirely true today due to some flaws in the existing protocol design and implementation, but we know how to fix these issues and we’re working on it.
In other words, Spacemesh miners should be able to control their own destiny. They should be able to mine without needing any intermediary: someone else to sell them coins (proof of stake), or hardware (proof of work), or to operate a pool for them. They should have direct, unfiltered, uncensorable access to the network and the protocol.
This is what Spacemesh is all about: millions of ordinary people, uncoordinated and unmediated by any central actor, running Spacemesh on their existing hardware, earning coins permissionlessly, and participating in the Spacemesh economic ecosystem, creating value for themselves, their families, and their communities.
Thing #2: The Problem 🚧
“In theory, theory and practice are the same. In practice, they are not.” - Einstein
Our goal is simple but challenging: to make things easy for the little guy. Specifically, we want mining rewards to be linear, directly proportional to the amount of space a miner commits to the network. Sounds straightforward, right?
The basic idea behind Spacemesh economics is that the marginal cost of mining for a small-scale home miner should be almost zero. This holds true if the miner uses a computer with a hard drive and an internet connection they already own and would pay for anyway. By mining on the network, they utilize otherwise underutilized resources.
However, an industrial-scale miner faces additional costs. They must acquire hardware specifically for mining, pay staff, including devops engineers to maintain hardware and operate nodes, and worry about redundancy, disaster recovery, and replacing failed drives. Thus, the marginal cost for a large-scale miner should be much higher than for a small home miner, preventing big miners from pricing small miners out of the network.
In reality, things aren’t so simple. Running a Spacemesh node is much harder than it should be. The original idea was that you’d download and install an app, minimize it, and basically “set it and forget it.” You’d check back a few days later and see that you’d begun earning coins.
But in practice, you need a modern, fairly powerful computer and a high-bandwidth internet connection to handle all the network traffic. Even then, syncing can be tricky due to the amount of data you need to download and process. You might need to fiddle with the node to get it to work, trying things like quicksync or changing the number of peers your node talks to. We’re working on making this easier, but that’s the reality today. Unfortunately, it means that all but the most technically savvy, motivated, and patient home miners will give up or seek an easier way to mine.
Pools have addressed many of these problems for miners. They have more resources and have developed more user-friendly software with features advanced miners want, like splitting data across multiple drives. For a small cut of the fees and your soul private keys, they’ll operate the node for you. While some users care deeply about decentralization and network health, most miners prioritize profits and ease of running a node. So, it’s no surprise many miners, large and small, have migrated to pools despite pools being less ideal for decentralization.
Even beyond pools, there are fundamental protocol constraints that make decentralization hard. I didn’t fully grasp this until recently. It has to do with how miners participate in the mining process.
In Spacemesh, each miner generates an eligibility proof, known as an activation or an ATX, once per epoch and broadcasts this proof to the network. Each of these proofs imposes a burden on the network. The network needs to transmit each proof so that every node sees it, and each node needs to validate and store it. The cost of a single proof or a handful of proofs is tiny, but collectively, the cost of millions of such proofs is enormous. Despite the network being only a year old, there are already tens of GB worth of this data that a full node needs to download and verify. There are a few million such proofs each epoch, which takes even a fairly powerful computer more than a day to download and verify. And that’s before even touching any transaction or block data!
The tension here is that we don’t want to impose a higher minimum size because that would make it difficult or impossible for home miners to join. However, small miners impose a much greater burden on the network per unit of space they contribute. Incentive compatibility dictates that the miner must pay for the cost their activation imposes on the network. This isn’t currently the case, which is one reason the network is overwhelmed by miner identities and activations.
If we impose a cost on an activation, small miners will be disadvantaged as rewards will no longer be linear. To put things in perspective, imagine a small miner with the minimum 4 storage units earning 1 coin per epoch, and a “whale” miner with 400 storage units earning 100x that. If the activation cost is set at 0.1 coins, this is 10% of the small miner’s income, but only 0.1% of the larger miner’s income.
There’s no perfect solution here. There’s a fundamental tension between network health on the one hand, and fairness, accessibility, and decentralization on the other. This is one of the core tradeoffs in computing and distributed systems! The most efficient network—just like the most efficient human system—is the most centralized one. But decentralization is essential. We must sacrifice some efficiency for decentralization, and finding the right balance is the art of building a healthy blockchain.
If we impose no cost on activation and maintain perfectly linear rewards, the network will continue to be overwhelmed with too many activations, simply because optionality always has value. Other things being equal, it will always be in a miner’s interest to maintain many identities rather than one. If we impose such a cost or reward miners for merging identities, it will negatively impact the smallest miners and encourage pooling. These simple economic facts are inescapable.
Economies of scale affect both the supply and demand sides. On the supply side, large miners benefit from expertise, amortizing costs over more storage space, accessing faster computers and higher-bandwidth Internet, utilizing existing hardware, etc. On the demand side, many small identities impose a larger burden on the network. We’re really swimming against the current by trying to limit these economies of scale.
Thing #3: The Solution 🗝️
"The secret of change is to focus all your energy not on fighting the old, but on building the new." - Socrates
Trying to defeat economies of scale is a fool’s errand. As I mentioned before, it seems to be one of the iron laws of the universe that big, well-endowed actors nearly always do better than their smaller, poorer peers. So, attacking the issue from the supply side—by trying to level the playing field perfectly for small miners—isn't the way to go.
We have more control over the demand side, i.e., the Spacemesh protocol itself. It’ll always be true that an additional miner imposes an extra burden on the network. So, from a network health perspective, it’s always more efficient if two smaller miners merge into one. But there are other things we can do.
Naively, the right approach seems to be to reward decentralization. Encourage miners to diversify: across geographies, hardware, software, etc. There have been a few attempts to do this, like the Ethereum Beacon Chain’s correlation penalties. But in some sense, miners already have an incentive to diversify: it helps them avoid common mode failure, avoid downtime, and earn more rewards when a bug or issue takes some miners offline! We don’t need to add special incentives for diversification. Any attempt to go further would be gameable—miners can always pretend to run different software or have different IP addresses. This would just collapse into proof of work and start another arms race. This isn’t the answer.
I think the right approach is threefold:
First: Reduce the per-miner cost to the network as much as possible. Spacemesh is unique among blockchain protocols in requiring every miner to recertify every epoch. Proof of work doesn’t require any eligibility proof ahead of time—successfully finding a nonce and mining a block is the proof of eligibility. In proof of stake, a miner only needs to broadcast such a proof once when they first join the validator set. After that, they’re an active validator until they voluntarily exit or commit a fault.
The protocol today also guarantees at least one reward slot per miner per epoch, which, as I wrote about recently, is a problem. If miner eligibility proofs were somehow durable—guaranteeing eligibility for a longer period—it would drastically reduce the burden each additional miner imposes on the network. We could move to a more proof-of-work-like setup where miners don’t need to prove their eligibility ahead of time and are randomly selected to participate in block production. But this would make it harder for the smallest miners since it would increase the variance of their rewards and mean they wouldn’t win blocks very often unless they joined a pool. This would also require deep changes to the protocol, which would take time to develop and test.
Second: Ensure rewards are as close to linear as possible. They’ll never be 100% linear, and even perfectly linear rewards wouldn’t be sufficient for the reasons discussed above, but we can get pretty close. Linear is much better than the alternative, which would disadvantage small miners even more. If we can substantially reduce the burden each miner imposes on the network, the remaining cost could be amortized over a very long period, making it insignificant. We may even reach a point where the value a miner brings to the network far exceeds the cost it imposes, eliminating the need for a per-miner or per-activation fee.
Third: Introduce multiple tiers to consensus. The upcoming Athena VM release is the perfect opportunity. We could increase the minimum storage space required to mine at the base layer significantly but simultaneously introduce one or more higher layers where smaller miners could still mine profitably. While economies of scale would play out at higher layers too, we might find an equilibrium where it’s not profitable for larger miners to mine at higher layers, but still profitable for smaller miners. That would be ideal.
We haven’t given up on our mission to be friendly to the little guy. It’s become harder to maintain this promise, but we’ve learned a lot about the challenges we face and how to overcome them. I’d love to hear your thoughts and ideas on how we can do better!