Three Things #44: November 20, 2022

Spacemesh is Hard

As I’ve written before, designing and building a new layer one blockchain, especially one with a novel consensus mechanism like Spacemesh, isn’t trivial. It involves technical, social, and financial risk and uncertainty. Here are three of the biggest challenges we’re facing right now as we approach genesis.

Working on Spacemesh these past few years has felt a bit like walking up a very long staircase. There’s a door at the top, visible now, and we’re not quite sure where it leads, but we’re excited to walk through it. Photo by Belinda Fewings on Unsplash

Thing #1: To Ship or Not to Ship?

There’s a universal tension that arises when building software. On the one hand engineers almost universally want their software to be perfect. If it were up to them, they’d spend forever and a day tinkering, testing, tweaking, and improving things: basically, doing everything other than shipping. After all, they’re in it for the technical challenge and they’re after an abstract sense, never quite achieved, of technical beauty and perfection. Engineers like engineering and not all the peripheral things software teams have to deal with: marketing, sales, community, talking to users, etc.. In my experience the average talented engineer would be happy if they could tinker with their software forever, even if it never shipped and was never used by an actual user.

On the other hand you have business and market realities. R&D is expensive. Your investors won’t let you tinker forever without releasing something. And they’re right: the best way to build something great is to start small and iterate by responding to users and the market. The only way to do that is to ship. No engineering team ever built something amazing by locking themselves in a room for years and not talking to the people who would actually use their software. (YC has piles of advice on talking to users, precisely because engineers hate doing this.)

This tension is bad enough if you’re building an app for sharing cat photos. It’s more complicated by the unique nature of blockchain software. These days most software isn’t too hard to upgrade. You can easily release a new mobile or web app every week, or even every day if you like, and expect upgrades to happen more or less automatically. You can force your users to upgrade. Using technology like automatic app updates or over-the-air updates, the process can even be totally seamless for the user.

Distributed systems in general and blockchain software specifically is much harder to upgrade. For one thing, each upgrade requires the thousands of people running your software to coordinate. If you upgrade via hard fork and only some of your users upgrade, you end up with a network split which is very messy to fix. Automatic or forced upgrades are frowned upon for reasons of governance and culture. Even collecting analytics data is unpopular, so oftentimes we don’t even know which versions of our software people are running.

Then there’s the fact that in blockchain bugs can be a really, really big problem. A bug in your average application isn’t such a big deal. A bug in a blockchain core protocol can cause the entire chain to crash, applications to stop working, and millions or even billions of dollars to vanish. If the bug causes one or more users to get exploited, undoing the exploit can be costly or impossible. Blockchain software is unlike web and mobile apps and more like critical embedded systems, e.g., in airplanes or hospitals. It has to work right the first time, it has to be able to run with very little maintenance for a long time, and it has to be backwards compatible.

For all of these reasons, we have to make sure that Spacemesh is in good shape before we ship the first complete version and go to genesis. Today, our code is more or less complete but we haven’t finished setting all of the parameters and we haven’t tested it as thoroughly as we’d like. In an ideal world we’d do extensive testing and simulation of various failure modes to maximize the information available to us before setting the parameters. We’d test various combinations of parameters. And we’d do long-running tests in the wild for months to be absolutely certain the network is stable and secure.

But we don’t have that much time. We’ve been working on Spacemesh for nearly four years and it’s time to ship. Of course we’ll do as much testing as we can before genesis, and we’ll continue that testing long after. We fully expect there to be issues post-genesis and we’ll be prepared to do emergency fixes and upgrades. Hopefully they won’t be catastrophic issues, but we’ll be prepared for those, too, just in case.

The question of when to ship remains a controversial topic on the team. There are team members who insist that we need a few more months or even years before everything is fully tested, buttoned up, and ready for primetime. There are others who think we should launch as soon as we have a minimally viable beta. I’m in the latter camp: unfortunately, we don’t have the luxury of that much time. We have limited funding and our community has already been extraordinarily patient these past few years. We can’t keep them waiting forever. And it’s essential that we begin getting real world feedback on Spacemesh “in the wild”—launching is by definition the best way to do that!

Knowing when to launch—to avoid analysis paralysis, but also not expose our community members and early adopters to undue technical risk—is an extraordinarily difficult balancing act. We’ll ship Spacemesh as soon as possible, and no sooner.

For more: Read the latest monthly updates to get a sense of recent progress and where the project stands. See this GitHub project board for a more technical breakdown of the remaining tasks.

Thing #2: Choosing a VM

In 2018, Spacemesh began a project called SVM (“Spacemesh Virtual Machine”). The original plan was to build a WebAssembly-based VM with a custom smart contract programming language on top. We worked on this for over three years and developed a pretty sophisticated prototype, but we made the mistake of doing the work independently and not actually integrating the prototype VM into the go-spacemesh node. The two projects were worked on by different teams using not only different code but also different languages (Go vs. Rust).

We ended up more or less throwing out this prototype and all of the work that was done over those years (although the things we learned from it were invaluable and inform our current design). In addition to the issue that the two projects developed independently and without a common roadmap or a clear integration plan, a few big things have changed since we initially planned our VM strategy. One of the big things that changed is that EVM has emerged as a clear and dominant winner among smart contract VMs and environments. This wasn’t so obvious a few years ago, when a number of projects (including Spacemesh) chose non-EVM environments such as WebAssembly (Wasm) instead. Spacemesh has the advantage of being a fast follower and learning from the successes and mistakes of other projects, so we reevaluated this decision in light of present information.

Another thing that’s changed is Ethereum’s transition to a rollup-centric roadmap and the relative maturity of the rollup ecosystem. Combine this with the idea of a modular blockchain and the menu of options available for a new smart contract platform today is quite large. We could choose from a menu of fully-featured VMs at the base layer including EVM, Wasm, RISC-V, and Move (to say nothing of the custom VMs used by blockchains such as Solana, Stacks, and Tezos). We could choose to support one or more of these at layer two, inside custom rollups or execution environments, rather than enshrining them at layer one. We could mix and match: e.g., EVM-compatible rollups or sidechains running on a Wasm base layer VM. If we choose EVM, it’s not monolithic: there are varying degrees of EVM compatibility with performance tradeoffs. We could enshrine a single rollup or execution environment at genesis, or have a small set of them, or open it up as a free-for-all. There are at least thousands of possible permutations of the design.

While it’s exciting that there are so many options today, it’s led to a sort of analysis paralysis. Each of these ideas is independently quite complicated and it takes months to study, understand, spec, build, and test each of them—and testing them in combination is harder than testing in isolation. By the time this process is done for a single item, the entire landscape has changed, since the space is developing so quickly. We want to “skate to where the puck is going to be,” but that’s quite hard to forecast given the breakneck pace of development.

The good news is that we have a new plan, and I’m quite excited about it. We’ll announce it very soon. For now, suffice it to say that it involves the following: one, we’ll initially launch with a very small set of enshrined smart contracts (e.g., multisig, vesting vault); two, EVM compatibility that’s as robust as possible; and three, given the complexity and rapid evolution of the design space, we want to maintain optionality. This last point means that we don’t want to enshrine a single VM now, change our minds in a year or two, and end up having to support two VMs forever. The best way to do this is to keep the base layer as simple as possible and encourage experimentation at layer two.

For more: Watch my talk on modular blockchain, and this talk on the Filecoin VM.

Thing #3: People

Blockchains are people. We say this often enough, usually half in jest and usually referring to the role that social consensus plays in blockchain governance. But blockchain software is people, too. I think a lot of engineers have a naive mindset about writing software: that if you hire a bunch of smart people and put them in a room together, they’ll sort of figure things out. Nothing could be further from the truth, as we’ve learned the hard way.

Engineers are stereotypically good with computers and bad with people. Computers have sharp edges and software is deterministic; people, by contrast, are fuzzy and complicated. Spacemesh is a heavily engineer-driven project (which is one reason we focus on shipping code and not on making noise), and we historically haven’t done the best job managing people.

People are expensive, and different people are motivated by wildly different things. A lot of smart people, and a lot of great programmers, are assholes, or aren’t ethically motivated, or simply aren’t team players. Some people who have all of these positive qualities just don’t get along with others who also do, for idiosyncratic reasons. Some people are highly rational, others are highly emotional, and people who are far apart on these poles really struggle to communicate and be effective on a team together. Most people don’t know their own personality profiles and cannot explain them. People have vastly different needs and expectations with respect to compensation, communication, work style, etc.. And all of this is exacerbated when those people are working remotely (by necessity or by choice).

At this point, I think we’ve made most of the people-related mistakes a startup can make: hiring too quickly, hiring too slowly, firing too slowly, bad onboarding, bad offboarding, not hiring for cultural fit, failing to properly assess technical skill, poor management, cofounder issues, poor communication, lack of coordination, too much coordination, assigning tasks to the wrong people, lack of accountability, you name it. The bad news is that these mistakes cost us a lot of time and money.

The good news is that we’ve learned from them and they’re behind us. I can say with confidence that the group of people working on Spacemesh today is the the right group to take us to genesis and beyond. It’s the smartest, most professional, kindest, most driven and most hardworking group of people I’ve ever worked with, and I feel honored every day to be part of this team and have the opportunity to work with and learn from these people.

What Spacemesh has taught me is that, when a group of people come together and create a thing, they inevitably imbue it with a piece of themselves. This is like a variation on Conway’s Law: organizations design systems that mirror their values. The reverse is also true: if you want a system that values X, hire people that value X to build it. If you want to build an ethical system, you have to hire ethical people. There is simply no way around this.

You cannot create an open, decentralized, subversive, permissionless system that’s also inclusive and encourages radical participation if you don’t live these values and really believe in them. Why? Partly because these ideas are complex and nuanced and if you don’t understand them, you can’t design for them. Mainly, though, it’s because doing this is really hard, and if you don’t really, really care, you’ll give up or take shortcuts.

Finding the right people to work on Spacemesh has been extraordinarily difficult for a number of reasons. We’re not looking for people that are primarily financially motivated (so the vast majority of big tech refugees need not apply). Spacemesh is extraordinarily complex, both in concept and in implementation, so you also have to be abnormally smart and creative to be able to understand it and contribute to it. Quite a few apparently brilliant new hires have nevertheless not worked out because, after a few months, they didn’t grok the system and the underlying design and motivations (or didn’t put in the required effort).

Most importantly, we are extraordinarily values- and vision-driven. In my experience, nearly every company and every team talks about vision and values, but vanishingly few actually put them into practice. Over time we’ve found ways to hire for our values, and we’ve found that this is, rightfully, the biggest obstacle for onboarding otherwise talented candidates. As one example, one of our values is thinking, designing, and engineering for the very long term. This has interesting implications on hiring since most teams turn over every couple of years. We don’t hire people at Spacemesh to work for a year or two. We hire people that want to work on this for a decade or two because that’s how long it will take to get Spacemesh to where we want and need it to be. The modern economy, and crypto in particular, encourages chasing new, shiny things. Most people don’t have this sort of attention span.

And then there’s remote work. Our team came together during the pandemic and under financial constraints. We decided to hire the best people we could find regardless of where they’re from and where they live. In a sense, this worked out brilliantly because we have a truly amazing team. But having people spread out around the world across many countries, timezones, cultures, and languages has also been extraordinarily challenging, and we’ve had to part ways with quite a few people to arrive at today’s all-star roster. Call me old fashioned, but I find that it’s almost impossible to build trust, rapport, team spirit, and a straightforward sense of common humanity over Zoom and Slack—and no team functions well without those things. If I had to do it all over again, I’d try harder to build a team in a single place, or at least get people together face to face a lot more often. Maybe we’ll get there someday with Spacemesh.

For more: Take some time to study personality profiles. I recommend the Big Five. Understand your own personality traits, especially those that are most pronounced. Reflect on what this means for your work and communication style, and how easy or hard it is to get along with people with very different personality types.