⚡ Blazing fast NVMe drives with unlimited IOPS now available. Read about PlanetScale Metal ⚡
Sign in|Get started
Navigation

Benchmarking PlanetScale Metal vs Amazon Aurora and Google Cloud SQL

PlanetScale Metal uses blazing-fast local NVMe drives for storage, providing unlimited IOPS at ultra-low latency. Customers who switch to Metal almost always see improved performance and cost savings. Here we show several performance and pricing comparisons between PlanetScale Metal, Amazon Aurora, and Google Cloud SQL.

We get it, it isn't called "BenchMarketing" for nothing. There are a million ways a benchmark can be run, leading to unfair comparisons. Read this article as a launching point, but don't just take our word for it. You can read how Block, Intercom, and Depot have drastically improved their performance on Metal.

500GB TPCC - M-320 vs Aurora db.r6i.xlarge

Benchmarking

To start, we will run the following comparison:

First, let's look at the QPS results. The graph below, as well as the rest of the graphs on this page, are interactive. Click the thumbnails to add or remove data from the main graph. Hover the graph to see values at a given time.

We can see that Metal takes a clear lead for all connection levels. Even with the overhead of running the VTTablet process with mysqld, it is able to out-perform the Aurora database both with and without a proxy. This is due to the low I/O latency and unlimited IOPS capability of having an attached NVMe drive. During the test, the M-320 instance topped out at around 40k IOPS. Such IOPS bandwidth is quite costly on AWS, though using Aurora I/O-Optimized helps with this. Aurora with the proxy enabled resulted in a lower QPS than without the proxy, as might be expected.

Now for the p99 results.

In addition to better QPS, the M-320 has significantly better p99 latency than the other two configurations. In this configuration, the PlanetScale M-320 is a clear winner in raw performance, latency, and consistency.

Cost analysis

All of the benchmarks in this article run against the single primary node. However, for a highly-available system, we must have several database nodes available. PlanetScale gives you a highly-available cluster by default with proxy (VTGate), 1 primary, and 2 replicas. In Aurora, all of these are configurable.

For Aurora I/O-Optimized, each db.r6i.xlarge instance costs $0.754 per hour, and storage is $0.225 per GB/month. For Aurora Standard, each db.r6i.xlarge costs $0.58 per hour, the storage is $0.10 per GB/month, and $0.20 per 1 million requests. Here, a request refers to single IO operation, but we will be approximating this to be 1 IO operation for every one query to simplify the math. If we assume a ~10k QPS workload as above, that comes out to 10,000 * 60 * 60 * 24 * 30 = 25,920,000,000 requests per month. Here is the pricing assuming each has a primary and two read replicas for us-east-1:

The PlanetScale instance comes out significantly cheaper. We can see that for a high QPS / IO workload, Aurora's I/O-Optimized pricing plan makes more sense when comparing it to Aurora's standard pricing model. The PlanetScale instance also comes with additional unused storage, so there's room for the data to grow before needing to pay more. These costs, as well as the other ones shown in this article, are not including the VTGate upgrade for PlanetScale, nor the cost of the proxy services for Aurora or Cloud SQL.

500GB TPCC - M-640 vs Aurora db.r6i.2xlarge

Benchmarking

Now for larger instances. This time we'll use a M-640 and an Aurora db.r6i.2xlarge instance. Both of these have 8 vCPUs and 64GB of memory. We again up-size the PlanetScale VTGate, and we will test Aurora both with and without a proxy. We will run this benchmark using a 500GB TPCC database.

When we double the resources of our database, we'd hope to gain ~2x the capacity, so long as things like network bandwidth are not a bottleneck. This is roughly what we see for all three of our configurations:

The M-640 is performing ever-so-slightly better than the Aurora cluster without a proxy on the QPS side. In both instances, the Aurora database is less consistent (spikier graph).

Let's look at p99:

We can see that the Metal database not only has low p99, but it is extremely consistent compared to both of the other options.

Cost analysis

Let's again compare the cost of these two configurations.

For Aurora I/O-Optimized, each db.r6i.2xlarge costs $1.508 per hour, and storage is $0.225 per GB/month. For Aurora Standard, each db.r6i.2xlarge costs $1.16 per hour, and storage is $0.10 per GB/month and $0.20 per 1 million requests. If we assume a ~18k QPS workload as above, that comes out to 20,000 * 60 * 60 * 24 * 30 = 51,840,000,000 requests per month. The pricing:

Google Cloud SQL vs PlanetScale Metal

Another popular product in the cloud-hosted MySQL space is Google Cloud SQL. Let's also compare Metal to Google Cloud SQL.

Though Metal is available in Google cloud, all of the Metal numbers used here were benchmarked in AWS.

For Google Cloud SQL, we use an Enterprise Plus db-perf-optimized-N-8 instance with the multi-region option enabled. This configuration has 8 vCPUs and 64GB memory on the primary, and no additional proxy service is applied. The database and test server are in the same region and zone (us-central-1f), and the test machine is a c3d-standard-16 (16 vCPUs, 64 GB Memory). We'll use a TPCC 500GB benchmark. Here are the results:

The resulting QPS and p99 are very inconsistent.

If you look carefully in that QPS graph, you'll notice that the graphs starts to get extra spiky in the final 30 seconds or so. To see if this is something that would continue, the benchmark was run again but with 20 minutes of execution time per segment instead of 5. We also include 20-minute run data from Metal as a comparison, and to ensure we do not run into similar issues with a longer test.

PlanetScale Metal performs great. For Cloud SQL, we can see that the pattern continues for runs with a high number of connections, particularly the 150 and 200 connection lines. After 4-5 minutes, the workload gets inconsistent and does not recover. This could be an issue of needing to tune some MySQL knobs to improve performance. However, for this test, we are leaving Cloud SQL in its default MySQL configurations.

Cost analysis

You can use Google's pricing calculator to estimate costs. To make the comparison even, we selected 3 of the db-perf-optimized-N-8 Enterprise Plus instances and 800GB of storage (extra is needed for binlogs). 3 instances were chosen to match up with the three instances you get with PlanetScale.

Google takes a slight edge on cost at 800GB, though this comes at a cost of worse performance. When accounting for the full 1866GB you get with the M-640, PlanetScale comes out ahead. This is again, ignoring any VTGate upgrades, or Cloud SQL proxy costs a user may need for a production workload.

Metal, Aurora, and Cloud SQL

Here are graphs for PlanetScale Metal, Aurora (with and without a proxy), and Cloud SQL. Only the 100 and 200 connection lines are included to minimize clutter.

In terms of QPS, Metal and non-proxied Aurora are basically tied. Though as we saw before, as the database grows and RAM is a smaller-and-smaller % of the total DB size, Metal shines due to its I/O performance. Metal will likely perform better with data growth.

In terms of p99 latency, Metal is the absolute clear winner. On the performance consistency side of things, Metal is also the clear winner, as both its QPS and p99 performance is the most even and consistent of the bunch.

1TB TPCC - PlanetScale M-640 vs Aurora vs Cloud SQL

Now for a larger database. This time, we use a 1TB TPCC database with populations settings of tables=40 and scale=250. The QPS and p99 numbers are below:

The PlanetScale M-640 outperforms the other options in all categories.

Is Metal always better?

By now it should be clear: Metal is awesome for performance. But is it always faster? There are some cases where it might not be. One example of this is if you have a small database with large compute needs, and where a substantial % of the data can fit in memory. In this case, the unlimited IOPS and low disk latency is leveraged less.

To show this, here is an example comparing an M-320 to an Aurora db.r6i.xlarge but with only a 100GB TPCC database.

Though Metal has lower p99 and better consistency, it lags behind in the raw QPS performance of the Aurora DB. As IO becomes less-and-less of a bottleneck, the benefits of unlimited IOPS decrease.

Of course, there are lots of other good reasons to choose PlanetScale: cost, high-availability, branching, deploy requests, and sharding to name a few.

The fact of the matter is, PlanetScale now provides the best combination of performance, availability, and scalability of any MySQL database product on the market. If you'd like chat with us about how Metal can help your database, reach out to us.