Earlier this year, I left my engineering role at Ripple Labs, which recently killed the labs and rebranded as Ripple. I'm often asked why I left but it'll suffice to say that it was an incredibly toxic environment.

This is the story about what I and dozens of others did to fix that.

About a year ago, I started out on the Codius team. We were a small group of four people. One was the CTO, and the other someone else who later went on to become the co-inventor of the interledger protocol. Early on, we didn't really have much of an idea of what Codius would be used for. It was an attempt at implementing smart contracts, which itself is an idea that hasn't really materialized past the heavily-invested nebulous vaporware stage, even to this day.

In reality, we were given a lot of free reign to develop new ideas, try them out, and see how it could be used within the wider cryptocurrency ecosystem. Before I joined the team, codius was the name given to the smart oracle system that was being developed. It wasn't obviously clear outside the team, but Codius was a fascinating research project. As time went on, I developed the core of the virtualization software. At first in C++, but later in rust.

At some point, two of my teammates quietly drifted away to build interledger. Meanwhile, another engineer was moved to my team. Formerly, the CTO was the lead engineer, but after he had drifted away I did my thing and started to fill the vaccuum.

Thats essentially how I became the team lead and manager. This invisible promotion never actually happened, but I ended up taking on that role through the tyranny of structurelessness within this small team. Eventually we collectively decided that Codius wasn't actually futhering Ripple's original goal of a fully decentralized payment network. One engineer, S, organized us into a discussion where we consensed that Codius isn't really going to help that goal nearly as much as going out and direclty implementing some more radical changes to the way Ripple worked, both technically and culturally. We discussed our idea with those above us and decided to become the Decentralization team. As I transitioned more into this newfound leadership role, I decided that we should coninue our original role as a research team.

The first thing we did was determine what was strictly opposed to the subject of decentralization:

  • A core rippled team lead by a toxic Rockstar engineer
  • Agreements within the team that special backdoored versions of the software needed to be produced as security through obscurity.
  • A developer-turned-operations that refused to cooperate with the existing operations team, instead becoming violently defensive towards any attempt to share deployment knowledge
  • Zero consideration for operations engineers outside of the one developer-turned-operations on their team, leading to an incredibly user-hostile and undocumented deployment process.
  • Every deploy was performed by hand, with all attempts to automate via even a shell script prone to a profane outbursts from the team.

Open Source but in name only, and with an unacceptably high bus factor for a financial system aimed at processing trillions of dollars a day with financial institutions paying millions to use.

Becoming an internal research team gave us the leeway to implement radical transparency within the company and attempt to eradicate the toxic walled garden culture the core team had enshrined itself in. S and I were pissed about the adverserial culture between the core team and the rest of the company, and immediately began a "reverse engineering" campaign against the core ripple daemon developers. We had no documentation to work from and began a scientifically rigorous approach to documenting our own system. Tools were thrown together to spin up virtualized systems, docker images built, and a DevOps revolution thrown in motion.

This resulted in a number of very radical changes in the company:

  • We built a test network called the Altnet for internal and external devs, so playing with the network wouldn't cost you actual real money
  • A thousand pages of build and deployment documentation added to the Confluence wiki.
  • The first public list of known processing nodes in the network, to illustrate Ripple's share of control.
  • Docker images and RPM packages were produced, eliminating the nead for every single developer in the 80+ engineering department to build and maintain their own binaries.
  • Hard evidence that the current design of the ripple daemon was unsustainable by every metric
  • An internal scientific research journal dubbed The Decent Journal was published every two weeks, containing a curated company-wide summary of what we learned and developed in the last two weeks.
  • A release process was developed, and a release manager put in place to prevent releases causing network-wide meltdowns and coordinate with the public, though the later was never realized before I left.

Unfortunately, it wasn't enough to turn the tide. The core team pushed back visciously and I gave my two weeks notice after 6 months or so of trying.

Regardless, those points are just a few of the things we did with me leading a science focused engineering team. I'm proud to have made an impact and see it as an intense learning experience. My best experiences were with the rest of the team co-editing our Decent Journal and wanting to share it with the rest of the company. We wanted the world to learn what we did.

Yes, they're still there. Yes, employee turnover is high. Yes, money is really just a social construct that we all kinda go along with for no real reason.