Neki, sharded Postgres, is now available. Learn more
Navigation

Reference

Store the Neki Data Topology in Git and Apply It from CI with the PlanetScale CLI

This guide keeps the Neki data topology as a JSON file in git and applies it with a script that a CI job can run. Shard ids differ between databases, so the file in git names shards with placeholders, and the script fills them in from the database before it compares or applies anything. The script can show what would change, apply the file, and fail the job when the live topology has drifted from git.

What you will build

repo/
  topology/topology.json     data topology with @SHARD_1@, @SHARD_2@, @SHARD_3@ placeholders
  ci/topology.sh             render | plan | apply

topology.sh plan    renders the file, diffs it against pscale branch data-topology get, exits 1 when they differ
topology.sh apply   renders the file and runs pscale branch data-topology update, then plans again and expects no difference

AI Prompt

Using the PlanetScale CLI (pscale 0.344.0), psql, jq and git, keep the Neki data topology in git and apply it from a script that CI can run.

Requirements:
- Create the Neki database with --engine neki, --cluster-size PS-10-AWS-ARM-NEKI and --replicas 2, add two shards, and wait until every shard answers a direct SELECT 1 with SET __neki.shard. Use one database role for every step.
- Store the topology in topology/topology.json in a git repository, with @SHARD_1@, @SHARD_2@ and @SHARD_3@ in place of the shard ids. The shard ids are different in every database and the topology update rejects a shard name such as sh2.
- Write ci/topology.sh with three commands. render fills the placeholders from SELECT uid FROM __neki.list_shards() WHERE name = 'sh1' (and sh2, sh3). plan diffs the rendered file against pscale branch data-topology get, after sorting both with jq -S .data_topology, and exits 1 on a difference. apply runs pscale branch data-topology update and then plan.
- Show a change made in a commit, a rejected topology that leaves the live topology unchanged, and drift made outside git that the plan catches.
- Run the verification step below before finishing.

Prerequisites

  • A PlanetScale organization with access to the Neki Platform Preview. Create an account if you do not have one, and see Neki availability and access for how an organization administrator joins the preview.

  • pscale 0.344.0 with a service token that can create databases, psql 18.6, jq, and python3:

export PLANETSCALE_SERVICE_TOKEN_ID="<token id>"
export PLANETSCALE_SERVICE_TOKEN="<token>"
export PLANETSCALE_ORG="<organization name>"
pscale version
  • A Neki database with three shards. Create it with these commands:
pscale database create topogit --engine neki --org "$PLANETSCALE_ORG" \
  --region us-east --cluster-size PS-10-AWS-ARM-NEKI --replicas 2 --wait

pscale role create topogit main admin --inherited-roles postgres --org "$PLANETSCALE_ORG" --format json > role.json
export PGHOST=$(jq -r .access_host_url role.json) PGUSER=$(jq -r .username role.json) \
       PGPASSWORD=$(jq -r .password role.json) PGPORT=5432 PGDATABASE=postgres PGSSLMODE=require
rm role.json
until psql -At -c "SELECT 1" >/dev/null 2>&1; do sleep 5; done

pscale branch shard create topogit main --config-profile default --count 2 --org "$PLANETSCALE_ORG"
until [ "$(psql -At -c 'SELECT count(*) FROM __neki.list_shards()')" = "3" ]; do sleep 10; done
for u in $(psql -At -c "SELECT uid FROM __neki.list_shards()"); do
  until psql -At -c "SET __neki.shard='$u';" -c "SELECT 1" >/dev/null 2>&1; do sleep 5; done
done

SHARD_1=$(psql -At -c "SELECT uid FROM __neki.list_shards() WHERE name='sh1'")
SHARD_2=$(psql -At -c "SELECT uid FROM __neki.list_shards() WHERE name='sh2'")
SHARD_3=$(psql -At -c "SELECT uid FROM __neki.list_shards() WHERE name='sh3'")

Note: Output samples are trimmed to the relevant lines. Each shard bills as a three-node cluster. PS-10-AWS-ARM-NEKI lists at $30 per month per shard, prorated to the millisecond. Current rates are on the Neki pricing page. Delete the database when you finish with pscale database delete topogit --force --org "$PLANETSCALE_ORG".

This guide also uses git.

1. Create the tables and the repository

psql -q -v ON_ERROR_STOP=1 \
  -c "CREATE TABLE customers (customer_id bigint PRIMARY KEY, name text NOT NULL)" \
  -c "CREATE TABLE orders (order_id bigint PRIMARY KEY, customer_id bigint NOT NULL, total_cents int NOT NULL)" \
  2>&1 | grep -v NOTICE

mkdir -p repo/topology repo/ci && cd repo && git init -q
cat > topology/topology.json <<'JSON'
{
  "shard_indexes": { "customer": { "type": "xxhash", "columns": ["customer_id"] } },
  "shard_groups": [
    { "uid": "@SHARD_1@", "key_ranges": [ { "shard_uid": "@SHARD_1@" } ] },
    { "uid": "by_customer", "default_shard_index": "customer",
      "key_ranges": [ { "shard_uid": "@SHARD_2@", "end": "80" }, { "shard_uid": "@SHARD_3@", "start": "80" } ] }
  ],
  "databases": { "postgres": { "schemas": { "public": { "tables": {
    "customers": { "shard_group": "by_customer" }
  } } } } },
  "default_shard_group": "@SHARD_1@",
  "authoritative_shard_group": "@SHARD_1@"
}
JSON

The file is the topology you would write by hand, with a placeholder wherever a shard id goes. orders is not in it yet. The commands from here on run inside repo.

2. Write the CI script

cat > ci/topology.sh <<'EOF'
#!/bin/bash
# ci/topology.sh render|plan|apply <database> [file]
set -u
cmd=$1; db=$2; file=${3:-topology/topology.json}
uid() { psql -At -c "SELECT uid FROM __neki.list_shards() WHERE name = '$1'"; }

render() {
  sed -e "s/@SHARD_1@/$(uid sh1)/g" -e "s/@SHARD_2@/$(uid sh2)/g" -e "s/@SHARD_3@/$(uid sh3)/g" "$file" | jq -S .
}
live() { pscale branch data-topology get "$db" main --org "$PLANETSCALE_ORG" --format json | jq -S .data_topology; }

plan() {
  if diff <(live) <(render) > /tmp/topology.diff; then echo "no changes"; else
    echo "topology differs from git:"
    grep -E '^[<>]' /tmp/topology.diff | head -12; return 1
  fi
}

case "$cmd" in
  render) render ;;
  plan)   plan ;;
  apply)
    render | pscale branch data-topology update "$db" main --org "$PLANETSCALE_ORG" --format json > /tmp/topology.out
    rc=$?
    if [ $rc -ne 0 ]; then echo "apply failed: $(jq -r .error /tmp/topology.out)"; exit $rc; fi
    echo "applied"; plan ;;
esac
EOF
chmod +x ci/topology.sh
./ci/topology.sh render topogit | jq -c '[.default_shard_group, .shard_groups[1].key_ranges[0].shard_uid] | map(length)'

Real output:

[14,14]

render swapped the placeholders for real shard ids, which are 14 characters here. The script compares both sides after jq -S, which sorts the keys, so a difference in key order is never reported as a change.

3. Plan, apply and plan again

git add -A && git -c user.name=ci -c user.email=ci@example.com commit -q -m "Shard customers by customer_id"
./ci/topology.sh plan topogit; echo "exit=$?"
./ci/topology.sh apply topogit; echo "exit=$?"
./ci/topology.sh plan topogit; echo "exit=$?"

Real output:

topology differs from git:
>   "databases": {
>     "postgres": {
>       "schemas": {
>         "public": {
>           "tables": {
>             "customers": {
>               "shard_group": "by_customer"
>             }
>           }
>         }
>       }
>     }
exit=1
applied
no changes
exit=0
no changes
exit=0

Before the first apply the live topology differs from the file: in the diff, a line that starts with > is in git and not yet live, and a line that starts with < is live and not in git. After apply the script plans again and finds no change, which is the check that the topology Neki stored matches what git says.

4. Change the topology in a commit

jq '.databases.postgres.schemas.public.tables.orders = { "shard_group": "by_customer" }' topology/topology.json > topology/next.json && mv topology/next.json topology/topology.json
./ci/topology.sh plan topogit; echo "exit=$?"
./ci/topology.sh apply topogit; echo "exit=$?"
git add -A && git -c user.name=ci -c user.email=ci@example.com commit -q -m "Add orders to by_customer"
pscale branch data-topology get topogit main --org "$PLANETSCALE_ORG" --format json \
  | jq -r '.data_topology.databases.postgres.schemas.public.tables | to_entries[] | "\(.key) -> \(.value.shard_group)"' | sort

Real output:

topology differs from git:
>             },
>             "orders": {
>               "shard_group": "by_customer"
exit=1
applied
no changes
exit=0
customers -> by_customer
orders -> by_customer

A change to the file shows up in plan as a difference, and apply makes the live topology match. orders is now routed with the same shard group as customers. The topology update does not move rows that already exist, so apply a change to a table that has data only as the guide on resharding a live table describes. This guide's tables are empty.

5. A rejected topology changes nothing

jq '.default_shard_group = "no_such_group"' topology/topology.json > /tmp/bad.json
./ci/topology.sh apply topogit /tmp/bad.json; echo "exit=$?"
./ci/topology.sh plan topogit; echo "exit=$?"

Real output:

apply failed: data topology default_shard_group references unknown shard group "no_such_group"
exit=2
no changes
exit=0

Neki validates the topology and rejects an unknown shard group, the script exits with the CLI's status of 2, and the live topology is still the committed one.

6. Catch a change made outside git

# someone edits the live topology by hand
./ci/topology.sh render topogit | jq '.databases.postgres.schemas.public.tables.orders.shard_group = "'"$(psql -At -c "SELECT uid FROM __neki.list_shards() WHERE name='sh1'")"'"' \
  | pscale branch data-topology update topogit main --org "$PLANETSCALE_ORG" --format json > /dev/null

./ci/topology.sh plan topogit; echo "exit=$?"
./ci/topology.sh apply topogit; echo "exit=$?"

Real output:

topology differs from git:
<               "shard_group": "shapgq4j17tp68"
>               "shard_group": "by_customer"
exit=1
applied
no changes
exit=0

The manual change made orders use the authoritative shard group. The plan reported it with exit 1, so a CI job that runs plan on a schedule or before every deploy fails when the live topology is not what git says, and apply puts it back.

7. Run it from CI

A job that runs on a merge to the main branch needs the three PLANETSCALE_* variables from your CI secrets and a database role, because topology.sh reads the shard ids with psql. Install pscale 0.344.0, psql and jq on the runner first. This is the shape of the job; this guide ran the script from a shell and did not run it in a hosted CI service:

jobs:
  topology:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/topology.sh plan "$DB"
      - run: ./ci/topology.sh apply "$DB"
        if: github.ref == 'refs/heads/main'

Verify the result

./ci/topology.sh plan topogit; echo "exit=$?"
git log --oneline | wc -l

Expected: no changes, exit=0, and 2 commits.

How it works

pscale branch data-topology get returns the stored topology inside a wrapper object, and the topology itself is under .data_topology. pscale branch data-topology update replaces the whole topology from standard input. Because the update is a full replacement, git can be the single source of truth, and plan is a diff of two sorted JSON documents. A shard id is generated when a shard is created, so the file names shards by their position and the script looks the ids up with __neki.list_shards(). The Neki data topology guide describes the file format.

Common issues

Shard names are rejected

Putting "sh2" in a shard_uid failed with is not a shard the cluster has created. Use the id from __neki.list_shards(), which is what render does.

The get output has a wrapper

pscale branch data-topology get --format json prints an object with the topology under data_topology. Feeding that whole object to update fails. Take .data_topology first.

Applying a topology does not move rows

The update changes how the router routes new statements. It does not move existing rows, so a change to a populated table's routing returns wrong counts. Move data with a workflow.

The script needs database credentials

render calls psql, so the job needs the database role's PG* variables as well as the service token.

Next steps

  • Delete the database with pscale database delete topogit --force --org "$PLANETSCALE_ORG".
  • Read the guide on spinning up and tearing down a Neki database per pull request to test a topology change on a throwaway database before it reaches main.
  • Read the Neki data topology guide for placing tables in a shard group.

verification:
  status: verified
  tested_at: "2026-10-29"
  product_version: "pscale 0.344.0, Neki PostgreSQL 18.6 (Platform Preview)"
  command: "topology.sh plan after apply, and the number of commits"
  expected_result: "no changes, exit 0, 2 commits"