New container restart metrics for Prometheus
We've added two Prometheus metrics for container restarts and deprecated planetscale_pods_container_restarts_total.
The old counter had a bug. Its planetscale_restart_reason label described only the container's most recent termination, not the restarts being counted. When that label changed, or a failover changed the role label, the series split and rate() and increase() reported restarts that never happened or missed ones that did. The counter also didn't exist until a container's first restart, so that restart never showed up as an increase.
The replacements:
planetscale_pods_container_status_restarts_totalcounts all restarts for a container. It starts at0when the pod is created and carries no reason or role label.planetscale_pods_container_last_terminated_reasonis a gauge that reports why a container was last terminated.
To count restarts for a single reason, filter the counter against the gauge one minute at a time:
sum by (planetscale_container) (
sum_over_time((
increase(planetscale_pods_container_status_restarts_total[1m])
and on (planetscale_pod, planetscale_container)
planetscale_pods_container_last_terminated_reason{planetscale_restart_reason="OOMKilled"}
)[1h:1m])
)
The gauge only reports a container's latest termination reason, so the query attributes each minute's restarts to the reason current in that minute, then sums over the hour. Change 1h to your own window and leave the 1m alone. To break restarts down by role, join planetscale_pods_status_phase, which carries planetscale_tablet_type on Vitess and planetscale_role on Postgres. Example queries are in the docs.
Two things to know when you switch:
- The new counter reports a series for every container, including ones that have never restarted, so expect more series.
- An
increase()window that reaches back before the new series first appeared shows a one-off spike. It clears once that moment ages out of your window.
The deprecated metric hasn't changed and keeps working until we remove it. We'll announce a removal date before then.