PlanetScale vs Supabase Benchmarks

This page includes benchmarks that compare the performance of Postgres on PlanetScale with Postgres on Supabase, along with all of the resources needed to reproduce these results. We also recommend reading our Benchmarking Postgres blog post, which covers the methodology used in these benchmarks and the steps taken to maintain objectivity. We invite other vendors to provide feedback.

TPCC Benchmark configuration

• Configuration: All Postgres configuration options left at each platform's defaults, except for connection limits and timeouts which may be modified to facilitate benchmarking.
• Benchmark machine: Benchmarks were executed from a c6a.xlarge in us-east-1, the same region as both databases.

Note that Supabase gets double the CPUs so we could match the RAM. We cover the reasoning in the TPCC results section. 12k IOPS is the max usable by a 2XLARGE on Supabase.

TPCC Benchmarks

TPCC is a widely-used benchmark to measure general-purpose OLTP workload performance. This includes selects, inserts, updates, and deletes.

Benchmark data: A TPCC-like data set generated with TABLES=20 and SCALE=250 using the Percona sysbench-tpcc scripts. This produces a ~500 gigabyte Postgres database. You can replicate the data following these instructions.

Benchmark execution: Using the Percona tpcc scripts running a load with both 32 and 64 simultaneous connections. We run the load on each database for 5 minutes (300 seconds).

Queries per second

Our first benchmark measures queries per second (QPS) at 32 connections and 64 connections, revealing a significant difference:

Click the graphs in the sidebar to toggle the number of connections. The PlanetScale database averaged ~17,000 QPS. Supabase averaged ~5000 QPS. Supabase does not offer memory-optimized RAM:CPU ratios (8:1), so we bumped up Supabase to a 2XLARGE to give it 8 vCPUs and 32 GB of RAM. This gives Supabase the same RAM capacity as PlanetScale, while giving it double the vCPUs. We also maximized IOPS to 12,000, the max a 2XLARGE can utilize.

p99 latency

We also measured the p99 latency for the duration of the benchmark run (lower is better):

Despite both being in us-east-1, PlanetScale shows much lower latency due to locally-attached NVMe drives with unlimited IOPS, 8th-generation AArch64 CPUs, and high-performance query path infrastructure.

OLTP benchmarks

In addition to TPCC, we run the OLTP Read-only sysbench benchmark. OLTP workloads tend to be 80%+ reads, and this benchmark allows us to isolate performance for such queries.

Benchmark data: A simple OLTP data set generated with TABLES=10 and SCALE=130000000 using standard sysbench. This produces a ~300 gigabyte Postgres database. You can find instructions for replicating this data here.

Benchmark execution: Using the standard sysbench tool using the oltp_read_only and oltp_point_selects benchmarks. You can find instructions for replicating this benchmark here.

Queries per second

This benchmark contains only SELECT queries, including ones with range scans and aggregations.

The PlanetScale database averaged ~35,000 QPS. Supabase averaged a much lower ~18,000 QPS. PlanetScale not only excels in QPS, but provides a much more consistent performance over time, leading to better predictability.

p99 latency

While running this benchmark, we measured the p99 latency of queries (lower is better):

PlanetScale offers both lower latency and better consistency, which is desirable for predictable performance.

Query-path latency

We measured pure query-path latency by running SELECT 1; 200 times in a row on a single connection. This tests the overhead of any database query.

Results compare PlanetScale + PSBouncer, standard PlanetScale connection, direct-to-Postgres on PlanetScale, Supabase with TXPooler, and without.

Direct connections to PlanetScale are significantly better than Supabase. Though, to be transparent, the direct connection to PlanetScale is same-AZ, which may provide an advantage depending on the AZ of the Supabase node. The Supabase transaction pooler (TXPooler), however, adds significant overhead. PlanetScale's PSBouncer is much lower latency in comparison.

Cost

A PlanetScale M-320 with 929GB of storage costs $1,399/mo. This includes three nodes with 4 vCPUs and 32GB RAM each, one primary and two replicas. Replicas can be used for handling additional read queries and for high-availability. The benchmark results shown here only utilized the primary.

A single 2XLARGE Supabase instance costs $404.64/mo, $94/mo for 750GB of EBS storage, and $216/mo to upgrade to 12,000 IOPS, a necessity for demanding workloads. This means a single node costs $714.64. To match the capabilities and availability of the 3-node PlanetScale M-320, we must add two replicas. This would lead to a price three times the single-node configuration, giving a total of $2,143.92/mo.

PlanetScale offers better performance at a lower cost for applications that require high availability and resiliency.

FAQ

What about the OrioleDB benchmarks?

Supabase / OrioleDB reproduced our TPCC and OLTP Read-only benchmarks in a recent article. Their results show similar average QPS to PlanetScale, ~18k QPS for TPCC and ~36k QPS for OLTP Read-only. However, these used a different storage engine (OrioleDB) and used io2 EBS volumes which significantly increases costs.

A single 2XLARGE Supabase instance with 750GB of io2 storage with 12k IOPS costs $1,978.89/mo for a single node. Adding even just one replica at this configuration brings the total cost to 3,957.78/mo, over 2.5x the cost of the PlanetScale M-320.