Yesterday, we announced TIN, a fast, full-featured search index extension for Postgres. We got a question from several readers and customers: how can I build apps on this if I can't run it in CI? Here's our answer: Lead. It's another Postgres full-text search extension, with the same feature set as TIN. All the same SQL statements and all the same TINQL searches will work fine.
One big difference: it's slow.
What Lead is
Lead has the same features as TIN.
Search results are correct, even for complicated SQL and TINQL queries.
Tokenizing is the same. What's a word in TIN is a word in Lead. As we add fancier tokenizers to TIN, they'll come to Lead, too.
Writes work: CREATE INDEX, INSERT, UPDATE, and DELETE.
It's even ACID, with correct snapshot isolation. Really. If you insert rows in a transaction, TIN Lead searches in that transaction will see them, and concurrent transactions won't. If you ROLLBACK, the rows disappear.
It's durable across restarts. If your tests need to restart Postgres, that's OK.
BM25 scoring works, including tin.score and tin.max_score.
It has hit highlighting, including both HTML and ANSI.
What Lead isn't
Lead will never be fast. Every search is a full table scan. The 8.0 GB Wikipedia corpus that TIN can search at 10,260 QPS with a p99 of 2ms takes four minutes per query with Lead. We tried it so you don't have to. Keep your tables in CI under a few MB, and Lead will be fine. Unofficial benchmarking has it searching War and Peace -- 3.4 MB, around half a million words -- in 170ms.
We've left out all the parts that make TIN fast, but kept the rest.
Building on Lead
You can download Lead today. (And be sure to star the repo!) Build your production search apps on TIN and build your tests, CI, dev, and staging on Lead. If you find ways you need Lead to be more like TIN, open a support ticket or find us on Discord.