Prometheus metrics for PlanetScale Neki

Overview

PlanetScale serves a Prometheus-compatible metrics endpoint for each Neki branch. Prometheus finds these endpoints through HTTP service discovery. The same discovery list covers every active branch in your organization, including Neki branches. If you use Datadog, the Datadog agent can scrape the same Prometheus endpoint.

Connect Prometheus

You need Prometheus 3.x, or a Prometheus-compatible engine such as VictoriaMetrics, that supports HTTP SD.

Create a service token in your organization settings. Save the token ID and the token.
Grant the token the read_metrics_endpoints permission.
Add a scrape configuration. Replace ${ORG}, ${TOKEN_ID}, and ${TOKEN} with your values:

scrape_configs:
  - job_name: "${ORG}-neki"
    http_sd_configs:
    - url: https://api.planetscale.com/v1/organizations/${ORG}/metrics
      authorization:
        type: "token"
        credentials: "${TOKEN_ID}:${TOKEN}"
      refresh_interval: 10m

Each discovered target carries a signature that expires after one day. Keep the refresh interval short so Prometheus always holds a valid signature. Each target returns the metrics of one branch. Prometheus attaches the planetscale_organization_name, planetscale_database_name, and planetscale_branch_name target labels, so you can tell branches apart. Every series also carries planetscale_database_branch_id. For managed Prometheus services that do not scrape, run Prometheus yourself and forward the metrics with remote_write.

Metrics

PlanetScale Neki emits the following metrics to be scraped. Every metric name starts with planetscale_. A histogram appears as three series with the suffixes _bucket, _sum, and _count. Counters and gauges appear under the name shown.

Router Metrics

Routers accept application connections, plan each statement, and send work to shards. These metrics describe the statements that routers handle.

Name & Description	Type	Tags
planetscale_neki_router_queries_total The total number of queries routed, by query type	Counter	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_router, query_type
planetscale_neki_router_query_duration_seconds The distribution of query latency, in seconds, through the router	Histogram	cluster, database, le, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_router
planetscale_neki_router_query_errors_total The total number of query errors returned by the router, by error class	Counter	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_router, error_class

Sidecar Pool Metrics

Each sidecar keeps a connection pool to its Postgres instance for each database. These metrics describe the pool. Use them to see when clients wait for a connection.

Name & Description	Type	Tags
planetscale_neki_sidecar_pool_capacity The configured maximum number of connections in a sidecar pool	Gauge	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_connections_active The number of open connections in a sidecar pool, checked out or idle	Gauge	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_connections_in_use The number of connections in a sidecar pool that clients currently hold	Gauge	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_connections_available The number of idle connections in a sidecar pool that a client can take	Gauge	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_connections_waiting The number of clients waiting for a connection from a sidecar pool	Gauge	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_waits_total The number of checkouts that waited for a pooled Postgres connection	Counter	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_neki_sidecar_pool_wait_seconds_total The total seconds that checkouts spent waiting for a pooled Postgres connection. Recorded only once a checkout has waited	Counter	cluster, database, planetscale_cell, planetscale_cluster, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard

Postgres Metrics

These metrics come from the Postgres instances in each shard. Series from a primary carry planetscale_role="primary". Series from a replica carry planetscale_role="replica".

Name & Description	Type	Tags
planetscale_postgres_connection_state The count and state of Postgres connections	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_connection_state, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_wal_archiver_succeeded_count The count of successfully archived WALs. Primary only	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_wal_archiver_failed_count The count of unsuccessfully archived WALs. Primary only	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_wal_archiver_last_age_succeeded The age of the last successfully archived WAL. Primary only	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_wal_archiver_lag_bytes WAL bytes written but not yet archived (current LSN minus last archived LSN). Primary only	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_wal_size_bytes The cumulative disk size of WALs waiting to be archived	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_settings_max_connections The current value of the max_connections setting	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_settings_max_wal_size_bytes The current value of the max_wal_size_bytes setting	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_replica_lag_seconds Replica lag in fine-grained seconds from Postgres. Replicas only	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_locks The count of current lock modes	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_lock_mode, planetscale_pod, planetscale_role, planetscale_shard
planetscale_postgres_database_xact_commit_total The total committed transactions on Postgres databases	Counter	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_database_name, planetscale_pod, planetscale_role, planetscale_shard

Backup Metrics

Name & Description	Type	Tags
planetscale_backup_restore_active Whether a backup restore is active, tagged with the current phase (1 = active)	Gauge	cluster, planetscale_component, planetscale_database_branch_id, planetscale_phase, planetscale_pod, planetscale_shard
planetscale_backup_fetch_percent The percentage progress of a backup fetch operation (0-100)	Gauge	cluster, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_shard

Infrastructure Metrics

These metrics describe the pods that run your branch. Router pods carry planetscale_router. Sidecar pods carry planetscale_shard and planetscale_role.

Name & Description	Type	Tags
planetscale_pods_cpu_util_percentages The CPU utilization percentage of router and sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_router, planetscale_shard
planetscale_pods_mem_util_percentages The memory utilization percentage of router and sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_router, planetscale_shard
planetscale_pods_iops_total The total IOPS (Input/Output Operations Per Second) of sidecar pods	Counter	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_volume_available_bytes The available storage space in bytes on Postgres volumes	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_volume_capacity_bytes The total storage capacity in bytes on Postgres volumes	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_pods_mem_rss_bytes The RSS memory usage in bytes of sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_pods_mem_mmap_bytes The memory-mapped file usage in bytes of sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_pods_mem_active_cache_bytes The active cache memory in bytes of sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_pods_mem_inactive_cache_bytes The inactive cache memory in bytes of sidecar pods	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_shard
planetscale_pods_container_status_restarts_total The total container restarts, across all restart reasons. Join planetscale_pods_container_last_terminated_reason to attribute them	Counter	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_router, planetscale_shard
planetscale_pods_container_last_terminated_reason The reason a container was last terminated (1 = the current reason). Join against planetscale_pods_container_status_restarts_total to attribute restarts to a reason	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_restart_reason, planetscale_router, planetscale_shard
planetscale_pods_container_waiting_reason The reason a container is waiting, such as CrashLoopBackOff or ImagePullBackOff	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_container, planetscale_database_branch_id, planetscale_pod, planetscale_role, planetscale_router, planetscale_shard, planetscale_waiting_reason
planetscale_pods_status_phase The pod status phase (Running, Pending, Failed, Succeeded, Unknown). Carries the pod’s role, so join it to break container restarts down by role	Gauge	cluster, planetscale_cell, planetscale_component, planetscale_database_branch_id, planetscale_phase, planetscale_pod, planetscale_role, planetscale_router, planetscale_shard

Example Queries

Replace your-branch-id with your branch ID.

Average router query latency

sum by (planetscale_router) (
  rate(planetscale_neki_router_query_duration_seconds_sum{planetscale_database_branch_id="your-branch-id"}[5m])
)
/
sum by (planetscale_router) (
  rate(planetscale_neki_router_query_duration_seconds_count{planetscale_database_branch_id="your-branch-id"}[5m])
)

Sidecar pool utilization

The fraction of the pool that clients currently hold:

planetscale_neki_sidecar_pool_connections_in_use{planetscale_database_branch_id="your-branch-id"}
/
planetscale_neki_sidecar_pool_capacity{planetscale_database_branch_id="your-branch-id"}

Average time a checkout waits for a pooled connection

rate(planetscale_neki_sidecar_pool_wait_seconds_total{planetscale_database_branch_id="your-branch-id"}[5m])
/
rate(planetscale_neki_sidecar_pool_waits_total{planetscale_database_branch_id="your-branch-id"}[5m])

Attributing Container Restarts

planetscale_pods_container_status_restarts_total counts every restart and carries no role or reason label. Join the metric that carries the one you want. To break restarts down by role, join planetscale_pods_status_phase:

sum by (planetscale_role) (
  increase(planetscale_pods_container_status_restarts_total[1h])
  * on (planetscale_pod) group_left(planetscale_role)
  group by (planetscale_pod, planetscale_role) (planetscale_pods_status_phase)
)

To break restarts down by reason as well, filter against planetscale_pods_container_last_terminated_reason one minute at a time and join the role onto the result:

sum by (planetscale_role) (
  sum_over_time((
    increase(planetscale_pods_container_status_restarts_total[1m])
    and on (planetscale_pod, planetscale_container)
    planetscale_pods_container_last_terminated_reason{planetscale_restart_reason="OOMKilled"}
  )[1h:1m])
  * on (planetscale_pod) group_left(planetscale_role)
  group by (planetscale_pod, planetscale_role) (planetscale_pods_status_phase)
)

The gauge only reports a container’s latest termination reason, so the query attributes each minute’s restarts to the reason current in that minute, then sums over the hour. Change 1h to your own window and leave the 1m alone. A container that restarts twice inside one minute under two different reasons has both restarts attributed to the later one. Router pods carry no role, so their restarts appear in a series without a role label.

Tag Glossary

cluster: The PlanetScale cluster identifier
planetscale_cluster: The name of the Neki cluster that serves the branch
planetscale_database_branch_id: The unique identifier for the database branch
planetscale_pod: The Kubernetes pod name
planetscale_container: The container name (for example postgres, walg-daemon, router)
planetscale_component: The Neki component that owns the pod: nkrouter (router), nktablet (sidecar and Postgres), or nkadmin (admin)
planetscale_router: The router group name. Present on router pods
planetscale_shard: The shard identifier. Present on sidecar pods and Postgres metrics
planetscale_role: The role of the sidecar and its Postgres instance (for example primary or replica). Absent on router pods
planetscale_cell: The PlanetScale cell identifier
planetscale_phase: The pod’s status phase (Running, Pending, Failed, Succeeded, Unknown), or the phase of a backup restore
database: The database that a router session connected to, or the database that a sidecar pool serves
query_type: The type of SQL statement, such as SELECT or INSERT
error_class: The two-character class of the Postgres error code (SQLSTATE) of a failed query
le: The upper bound of a histogram bucket (less than or equal to)
planetscale_connection_state: The state of database connections
planetscale_lock_mode: The Postgres lock mode
planetscale_database_name: The Postgres database name
planetscale_restart_reason: Reason for a container’s most recent termination (OOMKilled, Error, Completed). Carried by planetscale_pods_container_last_terminated_reason
planetscale_waiting_reason: Reason a container is waiting (CrashLoopBackOff, ImagePullBackOff, and others). Carried by planetscale_pods_container_waiting_reason

Need help?

Get help from the PlanetScale Support team, or join our Discord community to see how others are using PlanetScale.

Architecture

Pricing

Platform

Guides

Prometheus metrics for PlanetScale Neki

Overview

Connect Prometheus

Metrics

Router Metrics

Sidecar Pool Metrics

Postgres Metrics

Backup Metrics

Infrastructure Metrics

Example Queries

Average router query latency

Sidecar pool utilization

Average time a checkout waits for a pooled connection

Attributing Container Restarts

Tag Glossary

Need help?

​Overview

​Connect Prometheus

​Metrics

​Router Metrics

​Sidecar Pool Metrics

​Postgres Metrics

​Backup Metrics

​Infrastructure Metrics

​Example Queries

​Average router query latency

​Sidecar pool utilization

​Average time a checkout waits for a pooled connection

​Attributing Container Restarts

​Tag Glossary

​Need help?

Overview

Connect Prometheus

Metrics

Router Metrics

Sidecar Pool Metrics

Postgres Metrics

Backup Metrics

Infrastructure Metrics

Example Queries

Average router query latency

Sidecar pool utilization

Average time a checkout waits for a pooled connection

Attributing Container Restarts

Tag Glossary

Need help?