PlanetScale vs Heroku benchmarks

This page includes benchmarks that compare the performance of Postgres on PlanetScale with Postgres on Heroku, along with all of the resources needed to reproduce these results. We also recommend reading our Benchmarking Postgres blog post, which covers the methodology used in these benchmarks and the steps taken to maintain objectivity. We invite other vendors to provide feedback.

TPCC Benchmark configuration

Heroku has an unusual plan system compared to PlanetScale and many other vendors, for several reasons:

• The number of vCPUs is not specified in the plans.
• Some of the RAM options are atypical, like the 30.5 above instead of 32. We are using 30.5 as the closest equivalent.
• IOPS are not specified.

Given these limitations, we chose the standard-4 plan as our guess at a comparable set of primary resources for our M-320.

• Configuration: All Postgres configuration options left at each platform's defaults, except for connection limits and timeouts which may be modified to facilitate benchmarking.
• Benchmark machine: Benchmarks were executed from a c6a.xlarge in us-east-1, the same region as both databases.

TPCC Benchmarks

TPCC is a widely-used benchmark to measure general-purpose OLTP workload performance. This includes selects, inserts, updates, and deletes.

Benchmark data: A TPCC-like data set generated with TABLES=20 and SCALE=250 using the Percona sysbench-tpcc scripts. This produces a ~500 gigabyte Postgres database. You can replicate the data following these instructions.

Benchmark execution: Using the Percona tpcc scripts running a load with both 32 and 64 simultaneous connections. We run the load on each database for 5 minutes (300 seconds).

Queries per second

Our first benchmark measures queries per second (QPS) at 32 connections and 64 connections, revealing a significant difference:

Click the graphs in the sidebar to toggle the number of connections. The PlanetScale database averaged ~17,000 QPS. Heroku averaged ~2300 QPS.

p99 latency

We also measured the p99 latency for the duration of the benchmark run:

Despite both being in us-east-1, PlanetScale shows much lower latency due to locally-attached NVMe drives with unlimited IOPS, 8th-generation AArch64 CPUs, and high-performance query path infrastructure. Heroku also experienced huge spikes in latency starting around 160 seconds.

Query-path latency

We measured pure query-path latency by running SELECT 1; 200 times on a single connection. This tests the overhead of any database query. You can replicate the data following these instructions.

Results compare PlanetScale + PSBouncer, standard PlanetScale connection, direct-to-Postgres on PlanetScale, and Heroku.

Direct connections to PlanetScale are significantly better than Heroku. Though, to be transparent, the direct connection to PlanetScale is same-AZ, which may provide an advantage depending on the AZ of the Heroku node.

Cost

A PlanetScale M-320 with 929GB of storage costs $1,399/mo. This includes three nodes with 4 vCPUs and 32GB RAM each, one primary and two replicas. Replicas can be used for handling additional read queries and for high-availability. The benchmark results shown here only utilized the primary.

In addition to the unusual instance specification provided by Heroku, the pricing is somewhat unclear. Heroku states the standard-4 instance has a max cost of $750 per month, but it is not clear if or how the monthly cost could be less than this. We will use this as the base cost for a single instance. To match the capabilities and availability of the 3-node PlanetScale M-320, we must add two replicas. This would lead to approximately a price three times the single-node configuration, giving a total of $2,250/mo.

PlanetScale offers better performance at a lower cost for applications that require high availability and resiliency.