On joining PlanetScale and the vision of open source database infrastructure
By Shlomi Noach |
This is a personal account on why I joined PlanetScale to work on Vitess and PlanetScaleDB, and what I perceive Vitess can become in the MySQL open source ecosystem.
Background
I am a software engineer, drawn into the database world. I’ve been working with MySQL for two decades now, and am authoring or have authored various open source tools in the MySQL ecosystem: orchestrator for high availability, gh-ost for online schema migrations, freno for throttling and others. In the past decade I’ve worked mostly on infrastructure and distributed systems, work that produced those open source solutions.
The MySQL community enjoys a plethora of open source solutions, and an open discussion between its members. Many companies run MySQL infrastructure as means to enable their product, and are happy to share advice and experience on all things database infrastructure.
Yet, throughout the years I’ve experienced a growing frustration. As much as we, the community, collaborated and shared solutions, we were still all re-inventing the same things. Writing an open source tool and sharing it with the world is great, but deploying and automating it on another platform always hits issues. Different companies have different infrastructure, different deployment mechanisms, different network setups, different availability zones, different expectations on how to address availability issues, different versions, different topology layouts, etc. The effort to integrate a 3rd party tool is sometimes measured in months. Companies would just resort to developing their own in-house solutions, tightly integrated with their own infrastructure and flow.
We’ve had some discussions about that. We tried to run some community collaboration on a project or two. But people, myself included, have their own workload and schedule, and this didn’t take off. There were ideas for better integration between some of the open source tools. Some such integrations exist. But there’s nothing to the extent that solves or seriously empowers infrastructure and deployments.
Enter Vitess
Many consider Vitess an “open source sharding framework for MySQL”, and, indeed, Vitess solves some of the hardest problems in the database world: sharding, live resharding, data locality, and geo-placement. But Vitess is more than that. Architecturally, it provides the mechanisms to run a full blown infrastructure for a database deployment.
Vitess runs in a proxy/agent/controller setup. Clients connect to the proxy (vtgate) which routes their queries to the correct agents/shards/backend databases (vttablet). A controller (vtctld) can refactor and operate on topologies. This puts Vitess in the position to be able to automate away, and hide, much of database operational complexity.
Vitess’ design puts it in a position to further address more of the hardest problems: high availability, consistency, disaster recovery, online schema migrations, consistent reads, throttling, automated and tested backups and restores, rolling upgrades, data retention, local serving, load balancing, service discovery, and more. Some of these are already in the works, or already exist and are being improved at this time (more to come in future technical blog posts). And it is this design that allows Vitess to interact with, or fully integrate, existing open source tools in the MySQL ecosystem.
In my personal vision, Vitess is not a sharding framework, but an infrastructure framework for MySQL. Setup Vitess and get online schema migrations for free, no setup required. Get failovers and service discovery out of the box. Customize your HA/consistency rules if you like. Get throttling and consistent reads built in. I envision Vitess as the Grand Unified Theory for database infrastructure, that integrates existing tools, knowledge and experience in one well coordinated solution.
Fortunately my vision is compatible with the vision that the PlanetScale team has for Vitess and MySQL. We will work to make Vitess itself simpler and easier to install and use, while bringing in more infrastructure solutions. We plan to write and share more about the engineering challenges and developments we do here.
Shlomi Noach Software Engineer and database geek github.com/shlomi-noach @ShlomiNoach