The Problem of Pools
Three Things #85: September 17, 2023
The growth in the Spacemesh network over the past few weeks and the past few epochs has been phenomenal, exceeding all expectations. The network grew from 2,800 smeshers in epoch 2 (the first epoch where people could mine and earn rewards) to 6,500 in epoch 3 to 32,000 in epoch 4, which started about a week ago. Much of this growth has been driven by small, hobbyist home miners, which was always our target demographic. These users have been very engaged on our Discord and other channels, asking smart questions and swapping information about their home mining setups.
However, much of the growth has also been driven by users who have decided to join a mining pool for one reason or another, something that caught us by surprise. Spacemesh was designed specifically to be friendly to small home miners, and in such a way that there should be no advantage whatsoever from joining a pool. The promise of Spacemesh is that mining should be accessible and fair to even the smallest miners, who shouldn’t be at a disadvantage relative to larger miners. In simple point of fact, that isn’t true today, and what’s happened recently has showed that we still have our work cut out for us to fully realize that promise.
This week I want to explore what’s going on and why it happened, and explain a few ways we might respond.
Thing #1: Fixing PoST
One of the core elements of the Spacemesh protocol is something called proofs of spacetime, or PoST for short. The way miners in Spacemesh establish eligibility to participate in consensus and earn rewards is by provably dedicating a certain amount of hard disk space to the protocol and submitting proofs on an ongoing basis (once per epoch) that they still have custody of this data.
The process of initializing and managing this storage, and of generating and submitting proofs, is unfortunately rather complicated and cumbersome. As with so much in Spacemesh, in the most straightforward case a miner can simply run the Smapp application, use it to perform initialization, and then let it mine on an ongoing basis. The application and the underlying node software will manage all of the details on behalf of the miner.
However, this only works well for the simplest possible mining setup: a single node connected to a single miner identity on a single system, initializing a relatively small amount of storage on a single drive using a single GPU installed on the same system. This is the specific use case we had in mind when we designed Spacemesh, i.e., that of the “small home miner,” and we always assumed it would be the majority of the network.
As it turns out, however, many users, perhaps most at this stage, are interested in going further. Many of them have built custom Spacemesh mining rigs with arrays of many drives. Many want to use arrays of GPUs to perform initialization and run many nodes side by side, and many want to perform initialization in one place and mine on an ongoing basis somewhere else. This is a positive development that we’re excited to see but it’s just not the use case that we designed for, which is why it’s currently pretty hard to do these things.
As much as we care about usability, our first priority at Spacemesh is network health and safety. In order to keep the network secure we need to defend against various types of theoretical attacks. In one such attack, a sophisticated adversary could claim to have more storage than they’ve actually initialized and maintained. The proofs of space that miners have to generate on an ongoing basis are actually probabilistic so that they can be compact; rather than sending all of your data to the network once per epoch, which would be totally infeasible, you read all of your data and perform some proof of work while doing so, looking for certain elements of the data that satisfy a particular difficulty threshold. Then you submit only those elements as your proof. So in theory a miner could store, say, only 90% of the data they claim to have and still succeed in generating such a proof most of the time. (For those who are inclined see Proof of work and PoST proof generation for more details.)
In order to prevent this sort of attack we had to make this proof of work costly. Security is always a balancing attack because everything is a tradeoff: while this makes the network more secure it also increases the burden on every miner. We did lots of analysis and testing prior to genesis and picked a set of parameters that we thought were a reasonable compromise: they provide a high enough degree of security without imposing too burdensome a penalty on all miners.
Unfortunately there were a few flaws in this design. In the current design it turns out that miners that want to commit very large amounts of storage, on the order of tens of TiB or more, are actually better off if they split their one large identity into many smaller ones. The way Spacemesh works today, this provides several benefits: it allows miners to get storage online and begin earning rewards more quickly, it means they can more safely generate proofs during the window each epoch when that work needs to be done, and it means they can cut a few corners, do more work in parallel, and earn greater rewards than they could if they only had a single identity.
While in general miners should be free to split their storage however they wish for privacy or other reasons, at scale this is actually a problem because each identity imposes a cost not only on the miner who owns it (since they need to manage that identity) but also on the entire network (which has to receive, process, and store each miner’s proofs forever). By correcting the flaws in the PoST design, by enabling miners to grow their storage plots and combine existing identities, and by charging miners for the actual cost that their proofs impose on the network, we know how to get the incentives right and fix this but it will take some time. Miners who have already set up many identities won’t likely do additional work to combine them, and those already mining as part of a pool won’t likely exit the pool, unless the incentives to do so are quite strong.
Unfortunately, until this work is done, and in spite of our promises to the contrary, some miners will continue to see benefits from split identities or using pool software that manages all the gory details for them. The good news is that we know what needs to be fixed, we more or less know how to fix it, and there are some low-hanging fruit fixes that should improve the situation pretty soon. Stay tuned for more.
Thing #2: Better UX, Better Tools
There’s a type of user that cares deeply about what Spacemesh stands for, things like decentralization, censorship resistance, financial freedom, and a fair, level playing field. Those users do exist—I know because I’ve met them—but the reality is that they’re a minority, at least for now, and that’s okay. We’re building Spacemesh for all people and we need to be okay with the fact that many of those people may not share our values.
Speaking of building—there’s a lot left to build. We’re a small, under-resourced team that’s doing our best, but the reality is that the experience of using and mining Spacemesh today isn’t great for many users, to say nothing of trying to build applications on the platform. Our software is still buggy—hopefully there aren’t too many of the sort of bug that would crash the network, reduce stability, or cause security issues, but we have plenty of the sort of bug that makes it hard to, say, know your account balance at any point in time. There are also tons of other issues, features we haven’t gotten around to adding or things we haven’t had time to design and build or fix yet. For instance, our API was designed three years ago. Both the protocol and the state of the art have advanced considerably since then, and we had to cut a lot of corners implementing the API. This is seriously hampering efforts to provide better tooling and to improve the apps and tools we already have (such as the explorer), but we haven’t yet had the resources to redesign the API from scratch.
Users who are strongly values aligned may be willing to put up with a certain degree of frustration. Most users, however, are not so patient. I totally understand this. People have busy lives and can only dedicate so much time to Spacemesh even if they do align with the project’s goals and values. There’s only so many hours you can dedicate to fiddling with GPU settings, nonces, CPU threads, etc., not to mention apps that are just buggy and don’t work as they should out of the box.
Another thing that pools have done is provide a better user experience out of the box. In theory mining on Spacemesh should be as simple as downloading and running an app and pointing it at your GPU and free storage. This is the experience we’ve always promised users, and to be fair, it is more or less this easy if you’re running the “simplest possible mining setup” described above, i.e., the one use case we optimized our design for. But most miners today probably don’t perfectly fit this description. For anyone else, mining isn’t as straightforward as it could or should be. It turns out that there are a lot of variables, more than we accounted for—in terms of hardware setup, system resources, etc.. The complexity of the protocol doesn’t help. So our “five minute easy setup” has actually become fairly complicated for a lot of users. Software from pools has, it turns out, been easier. For instance, it’s not easy to figure out how to divide your storage so as to optimize your rewards and make the most of your hardware. Our software doesn’t do that for you; pool software does.
The obvious thing we need to do is improve both our tooling and the UX of our existing apps. It should be easy for all users, or at least most users, not just users with the simplest possible setup, to use official software to initialize, say, multiple identities or multiple drives using multiple GPUs, to move storage files around, etc. And Smapp should be simplified and streamlined, with a lot of confusing configuration options hidden for all but power users.
We’re beginning to work on a Smapp redesign to accomplish this, which may entail splitting it into several applications. We’re also working on splitting the node software into several modular pieces so that you could, e.g., perform several PoST initialization jobs in parallel, connect many miners to a single node, host a node in the cloud, etc. These are all big projects and they’ll take time, but the different interface that pool software has offered to the same underlying protocol has both inspired and informed future designs. We’re protocol designers and hackers, not product people, so it’ll take us some time to catch up.
Also remember that we have a superpower that pools don’t: doing things in-protocol. Today, by letting a pool perform initialization and mine for you, you have no choice but to give them your private key. This is not only antithetical to values such as personal responsibility and self sovereignty and bad for network health, it’s also dangerous because it requires trusting a third party to act honestly on your behalf, not get your identity banned, pay you correctly and not steal funds, etc. (in other words, from the perspective of money, it’s no better than a bank, which is sort of not the point of cryptocurrency!). This will never be the case with official Spacemesh software—you’ll always have full control of your own keys and your coins, as you should.
Thing #3: Education
Fixing PoST and improving UX are both important but, in my opinion, the most important thing we can and must do is to better educate Spacemesh users. The protocol is complex and, as explained above, today mining is also too complicated. The vast majority of our resources have been focused on shipping the network, responding to bugs and issues as they arise, and adding critical technical fixes and improvements. We’ve had scant time to think about education and it shows in the sorry state of our documentation.
One of the things that makes education hard is that among Spacemesh miners and users there are several personas and they are quite different from one another. As I mentioned we’ve always been focused on the small home miner with off-the-shelf hardware and without a ton of technical expertise. But since launching we’ve discovered that there are at least three other distinct classes of users: hobbyists with more technical expertise and lots of spare hardware, large scale semi-professional or professional mining operations, and mining pools. While there’s some overlap, the reality is that each of these groups of users has different needs: they need different tools and different docs, and they have different sets of questions. We initially prioritized the first at the expense of all of the others, and even now we’re struggling to help all of the groups. (One of the amazing things about the Spacemesh community is the way in which more experienced users have offered a hand to newcomers, something that’s delightful to watch and for which I’m incredibly grateful, but more needs to be done.)
As a result, we need a diverse set of educational materials. We already have basic explainer videos targeted at home miners. We’re working on more technical documentation for hobbyist and professional miners. We have Q&A forums and a FAQ in Discord, and we’re working on building a more complete, universal documentation site. Eventually we’ll add things such as full protocol documentation (targeted less at miners and casual users and more at app developers and researchers), an integration guide (for building apps and services on Spacemesh), and a lot more.
While Spacemesh is a blockchain and shares many of the same features as more familiar chains like Bitcoin and Ethereum, it’s also quite different in some key ways. I’ve attempted to explain some of those differences here, but this isn’t the best format for people to find or consume that information, so we need to do a better job. And answering one-off questions one by one on Discord also doesn’t scale (and, sadly and frustratingly, isn’t indexable by search engines). The good news is that, the more time we spend answering questions and discussing things with the community, the better sense we get of the sort of questions people have and the issues they’re encountering, which will ultimately lead to better docs and other resources. Between this and dogfooding the network myself (by, e.g., mining at home), I feel pretty confident that I can explain not only the high level protocol outline but also the practical, low-level technical details that miners and application developers are looking for today. By both improving the system itself and simultaneously improving documentation and other educational resources, the situation will rapidly improve.
For the time being, however, we have to contend with these limitations and with pools. The emergence of big pools on Spacemesh is frustrating to some people and in some ways it contradicts the Spacemesh mission of making mining easy for home users without needing to join pools. By the same token, however, pools are a legitimate application and they might always make sense for certain classes of users. As long as they act in a responsible fashion and in the network’s best interest, I have no problem with them, although a single, large, monopolistic pool does have centralization risks.
At the end of the day Spacemesh is a permissionless, censorship resistant protocol which means that we couldn’t ban pools from joining even if we wanted to. We need to help our users understand why this is the case and why it’s a good thing, and rather than “fighting” pools we need to figure out how we can respond more constructively through better incentive schemes (which at the end of the day is all a blockchain is at its core) and we need to communicate this to our users. Hopefully after reading this it’s a bit clearer how that might happen.
We have our work cut out for us to get Spacemesh to a place where it lives up to its original promise and potential but we’re aware of what’s going on, we know what needs to be done, and we’re working on it. The best thing you can do to support that work today is to join the network, mine, use it, and let us know what’s working and what isn’t. All of our work is open source and permissively licensed and we welcome contributions.