__neki.get_data_topology metafunction.
A new cluster starts with a topology stub naming the cluster’s initial shard:
Shards, topology, and logical objects
A Neki cluster contains shards and routers, a topology definition, and the logical database objects. The topology does not implicitly change when you add or remove shards or when you manipulate Postgres objects. The topology must be adjusted as necessary, using reshard workflows or the__neki.set_data_topology metafunction. For example:
- Newly added shards don’t automatically appear in the topology. Introduce them through a reshard workflow (to redistribute an existing shard group) or by declaring a new shard group that uses them.
- To delete a shard, first move remaining data off the shard and remove it from the topology. A shard can’t be deleted if it’s referenced in the topology document.
- The authoritative shard group can’t be removed or migrated to a different shard, so the cluster’s initial shard can’t be deleted. Leave it as a standalone shard and size it for catalog work, sequence reservations, and any unsharded tables that stay on it. Use separate shards and shard groups for large application tables.
- Newly added tables are resolved using topology defaults but don’t automatically receive dedicated topology entries.
__neki.set_data_topology metafunction called with the overwrite => true
option replaces the entire current topology document with a new one. Neki validates
the new document for syntax and certain structural issues,
but it doesn’t check if the logical objects exist or if their structure
aligns with the topology, and it doesn’t redistribute existing data to match the new document.
The ability to replace a topology document is useful for making additive or
subtractive changes that don’t break the existing logic: removing parts of the document
you no longer need, declaring routing rules for new tables, or setting up the
topology ahead of creating Postgres objects on a new cluster. To move existing
data around, use a reshard workflow.
Creating and removing logical objects
Neki doesn’t prevent you from creating or removing logical Postgres objects of your own accord. These operations fall into three main categories:- Changes that need a topology adjustment, such as creating a new reference table or a sharded table that overrides the default shard index.
- Changes that are covered by the existing topology without adjustment, such as creating a sharded table that follows existing topology defaults.
- Changes that don’t involve Neki topology at all, such as creating a foreign key constraint or changing the data type for a secondary column.
Familiar terms
Some terms familiar from Postgres have a different meaning in Neki.Shard index
A shard index turns values from one or more table columns into routing values. Neki compares those routing values with a shard group’s key ranges to choose a destination shard. Naming a column in the shard index doesn’t create a Postgres index on that column and doesn’t require that column to be indexed. An index can still be necessary for optimal performance.Primary index
A primary index is a per-table override of the shard index that the table is routed by. Without an override, the table is routed by its shard group’s default shard index. The per-table override is useful when different sharded tables contain columns that are named differently but contain the same logical data. For example, a data model for an event booking application might contain multiple tables that will be sharded by venue:venue_id
and an override for public.venues to use the id column.
A primary index declared in a Neki topology is distinct
and separate from a primary index used as a constraint on the underlying Postgres table.
The column named in a topology override doesn’t have to be a PRIMARY KEY
or a UNIQUE NOT NULL constraint at the Postgres level.

