🐋 chore: add opt-in alternative local stack experiments (#14678)

* feat: add search stack PoC infrastructure (Track 1) Docker Compose stack for PLAN.md's new chat-search architecture: ferretdb 2.7.0 + its postgres-documentdb 17 backing store (wal_level=logical for the later CDC spike), a new dedicated chat_search_db (PostgreSQL 17 + pgvector, non-default credentials, three least-privilege roles per the Security roles section), and clickhouse (26.3 LTS). vectordb/rag_api are untouched, per decision 3. All four image tags verified against their registries via curl (ghcr.io manifest lookups, Docker Hub tags API) before pinning. Host ports checked against every existing compose file in the repo to avoid collisions. The role-provisioning init script and healthcheck script were both actually run against live containers once Docker became available mid-task: full Mongo-wire round trip against ferretdb, real INSERT/SELECT proving the writer's default-privilege grants and the reader's deny-by-default posture against a throwaway pgvector table, wal_level and pgvector/pg_trgm extension checks. One bug only surfaced at runtime and is fixed: psql does not interpolate :'var' inside dollar-quoted DO $$ ... $$ blocks, so role creation/idempotency uses \gset + \if/\else/\endif instead. * 🔐 chore: require operator-supplied FerretDB credentials The chat_search_db and ClickHouse services already refused to start without operator-supplied passwords; FerretDB's backing PostgreSQL still fell back to a working ferretdb/ferretdb pair, so the stack booted with a known credential even though it holds projected chat content. That is the same shape as the myuser/mypassword default the search plan calls out on the existing vectordb service. All four FerretDB credential references now use ${VAR:?} - compose, the healthcheck script, and the README examples - and .env.example ships REPLACE_ME placeholders instead of literals. The differential-test harness at packages/data-schemas/misc/ferretdb keeps its fixture credentials; it holds only throwaway test data.
2026-08-28 12:44:28 +00:00 · 2026-08-07 10:40:58 -04:00 · 2026-08-07 10:40:58 -04:00 · 5ee910e44c
commit 5ee910e44c
parent 51ed1fab4b
5 changed files with 759 additions and 0 deletions
--- a/search/.env.example
+++ b/search/.env.example
@ -0,0 +1,48 @@
+# Copy to search/.env (gitignored by the repo's `.env*` rule) and replace
+# every REPLACE_ME_* value before running `docker compose up`. Generate real
+# secrets, e.g.:
+#
+#   for var in CHAT_SEARCH_BOOTSTRAP_PASSWORD CHAT_SEARCH_OWNER_PASSWORD \
+#              CHAT_SEARCH_WRITER_PASSWORD CHAT_SEARCH_READER_PASSWORD \
+#              CLICKHOUSE_PASSWORD; do
+#     val=$(openssl rand -base64 24 | tr -d '=+/')
+#     sed -i "s#^${var}=.*#${var}=${val}#" search/.env
+#   done
+#
+# See search/README.md for the full credentials map and what each DSN is
+# used for.
+
+# --- FerretDB + its backing PostgreSQL (DocumentDB flavor) ---
+# FerretDB 2.x forwards these same credentials to the Mongo wire protocol
+# (see search/README.md "How FerretDB auth works"), so this pair is the
+# database password for all projected chat content. Generate it like the
+# others above - no default is supplied and compose refuses to start without
+# it.
+FERRETDB_PG_USER=REPLACE_ME_ferretdb_user
+FERRETDB_PG_PASSWORD=REPLACE_ME_ferretdb_password
+FERRETDB_PG_DB=postgres
+
+# --- chat_search_db (new, dedicated PostgreSQL 17 + pgvector) ---
+# Non-default credentials at provisioning (PLAN.md decision 3) - nothing
+# connects to this instance as `postgres`/`postgres` or reuses vectordb's
+# myuser/mypassword.
+CHAT_SEARCH_DB=chat_search
+CHAT_SEARCH_BOOTSTRAP_USER=chat_search_admin
+CHAT_SEARCH_BOOTSTRAP_PASSWORD=REPLACE_ME_bootstrap_password
+CHAT_SEARCH_OWNER_PASSWORD=REPLACE_ME_owner_password
+CHAT_SEARCH_WRITER_PASSWORD=REPLACE_ME_writer_password
+CHAT_SEARCH_READER_PASSWORD=REPLACE_ME_reader_password
+
+# --- ClickHouse ---
+CLICKHOUSE_DB=chat_search
+CLICKHOUSE_USER=chat_search
+CLICKHOUSE_PASSWORD=REPLACE_ME_clickhouse_password
+
+# --- Host port overrides (defaults already avoid every port used by the
+# repo's other compose files - see search/README.md port map) ---
+# FERRETDB_HOST_PORT=27021
+# FERRETDB_DEBUG_HOST_PORT=8089
+# FERRETDB_PG_HOST_PORT=5434
+# CHAT_SEARCH_DB_HOST_PORT=5435
+# CLICKHOUSE_HTTP_HOST_PORT=8123
+# CLICKHOUSE_NATIVE_HOST_PORT=9000
--- a/search/README.md
+++ b/search/README.md
@ -0,0 +1,219 @@
+# Search stack PoC infrastructure (Track 1)
+
+Docker Compose stack for the new chat-search architecture in `PLAN.md`
+(worktree `postgres-ferretdb-clickhouse-rag-b6bc87`, "Infrastructure" track).
+Stands up four services, all isolated from the repo's production compose
+files: `ferretdb-postgres`, `ferretdb`, `chat_search_db`, `clickhouse`.
+
+This is infra only - no migrations, no app code. Table DDL
+(`chat_search.documents/embeddings/outbox/watermark`) is track 4's job; this
+stack only provisions the roles, schema, and default grants those migrations
+will run against.
+
+## Start
+
+```bash
+cp search/.env.example search/.env
+# fill in every REPLACE_ME_* value - see .env.example for a one-liner using
+# `openssl rand` to generate them
+cd search
+docker compose up -d
+./healthcheck.sh        # waits for all 4 services healthy, verifies roles exist
+```
+
+Tear down (including volumes - this wipes all PoC data):
+
+```bash
+docker compose -f search/compose.yml down -v
+```
+
+## What was actually verified (2026-08-07, live)
+
+Docker Desktop's WSL integration came online partway through this track. The
+full stack was brought up for real and torn down again afterward (nothing is
+left running):
+
+- All four containers reached Docker `healthy` status from a fresh volume.
+- `chat-search-roles.sh` ran cleanly on `chat_search_db` init: `chat_search`
+  schema created, all three roles created, zero errors in container logs.
+- Role attributes confirmed via `pg_roles`: `chat_search_owner`,
+  `chat_search_writer`, `chat_search_reader` are all `rolsuper=f`,
+  `rolbypassrls=f`, `rolcreaterole=f`, `rolcreatedb=f`. Only the bootstrap
+  admin (`chat_search_admin`, never used by the app) is a superuser.
+- **Default-privilege behavior confirmed against a real table**, not just
+  read from `pg_default_acl`: created `chat_search.smoke_test(id, v
+  vector(3))` as `chat_search_owner`, then connected directly as
+  `chat_search_writer` and did an `INSERT` + a `<=>` cosine-distance
+  `SELECT` - both succeeded via the default-privilege grant, no per-table
+  `GRANT` needed. Connected directly as `chat_search_reader` and ran
+  `SELECT * FROM smoke_test` - got `ERROR: permission denied for table
+  smoke_test`, confirming deny-by-default (this is what makes "no grants on
+  outbox or watermark" hold without the init script needing to know those
+  tables exist yet).
+- `pg_isready` + `SHOW wal_level` on `ferretdb-postgres` returned `logical`.
+- ClickHouse `GET /ping` returned `Ok.`.
+- FerretDB: full Mongo-wire round trip over `mongodb://$FERRETDB_PG_USER:$FERRETDB_PG_PASSWORD@
+  localhost:27021/?authMechanism=SCRAM-SHA-256` using the repo's own
+  `mongodb` driver (`node_modules/mongodb` at the repo root) -
+  `admin.ping()` returned `{ok:1}`, then `insertOne` / `findOne` /
+  `dropDatabase` all round-tripped correctly.
+
+One real bug surfaced only at runtime and is now fixed in both files:
+`ALTER DEFAULT PRIVILEGES ... :'password'` inside a dollar-quoted `DO $$
+... $$` block silently fails (`psql` does not interpolate `:'var'` inside
+`$$`-quoted text - it passes the literal `:'owner_password'` through to the
+server, which errors on `:`). Rewritten using `\gset` + `\if/\else/\endif`
+client-side metacommands instead, which interpolate correctly and were
+re-verified end to end. Also: granting `USAGE` on a schema is **not**
+sufficient for unqualified type names like `vector(1024)` to resolve -
+`search_path` has to include `chat_search` on all three roles, or every
+migration has to schema-qualify the type. Added `ALTER ROLE ... SET
+search_path = chat_search, public` for all three roles rather than push
+qualification requirements onto track 4.
+
+## Port map
+
+Chosen to conflict with none of the ports already used by the repo's other
+compose files. Existing ports (unchanged by this stack):
+
+| Port | Service | Where |
+|---|---|---|
+| 3080 | LibreChat API | `docker-compose.yml`, `deploy-compose.yml` |
+| 80, 443 | nginx client | `deploy-compose.yml`, `utils/docker/test-compose.yml` |
+| 3000 | admin-panel | `docker-compose.yml`, `deploy-compose.yml` |
+| 27018 | mongodb (optional host expose) | `docker-compose.override.yml`, `utils/docker/test-compose.yml` |
+| 7700 | meilisearch | `docker-compose.override.yml`, `utils/docker/test-compose.yml` |
+| 5432 | `vectordb` (pgvector, pg15) | `docker-compose.override.yml` |
+| 5433 | `vectordb` (pgvector, pg15) | `rag.yml` |
+| 8000 | `rag_api` (`RAG_PORT` default) | `rag.yml`, `utils/docker/test-compose.yml` |
+| 27020 | FerretDB differential-test harness (mongo protocol) | `packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml` |
+
+New ports, this stack (`search/compose.yml`, all overridable in `search/.env`):
+
+| Port | Service | Purpose |
+|---|---|---|
+| 27021 | `ferretdb` | Mongo wire protocol - the port LibreChat's Mongo driver would point at |
+| 8089 | `ferretdb` | FerretDB debug/metrics HTTP (`FERRETDB_DEBUG_ADDR`, container port 8088) |
+| 5434 | `ferretdb-postgres` | Direct SQL access to the DocumentDB backing store (Spike A/B poking, not needed by the app) |
+| 5435 | `chat_search_db` | PostgreSQL 17 + pgvector, the new dedicated search store |
+| 8123 | `clickhouse` | HTTP interface |
+| 9000 | `clickhouse` | Native TCP protocol |
+
+Note: this stack's FerretDB (27021) is a **separate instance** from the
+existing differential-test harness's FerretDB (27020,
+`packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml`). Both can
+run at the same time without conflict; they serve different purposes (this
+one is the PoC's live Mongo bridge, that one is Track 2's Jest harness
+target).
+
+## Credentials
+
+Nothing here uses a default credential. `search/.env.example` documents
+every variable; copy it to `search/.env` (already covered by the repo's
+`.env*` gitignore rule) and replace the `REPLACE_ME_*` placeholders before
+starting.
+
+| Variable | Used by | Notes |
+|---|---|---|
+| `FERRETDB_PG_USER` / `FERRETDB_PG_PASSWORD` | `ferretdb-postgres` bootstrap, `ferretdb`'s `FERRETDB_POSTGRESQL_URL` | **Required, no default** - compose refuses to start without them. FerretDB 2.x forwards these same credentials to Mongo-wire clients, so this pair is the password for all projected chat content - see "How FerretDB auth works" below. |
+| `CHAT_SEARCH_BOOTSTRAP_USER` / `CHAT_SEARCH_BOOTSTRAP_PASSWORD` | `chat_search_db` container bootstrap only | Non-default (PLAN.md decision 3). Superuser, but never used by the app - interactive/`docker exec` debugging only. |
+| `CHAT_SEARCH_OWNER_PASSWORD` | `chat_search_owner` role | Migration owner. Track 4's DDL runs as this role. Not superuser, owns the `chat_search` schema. |
+| `CHAT_SEARCH_WRITER_PASSWORD` | `chat_search_writer` role | Projection writer. This is `CHAT_SEARCH_WRITER_URL` in the app's feature-flag list - the projector/outbox consumer/sweep, never a request pod. |
+| `CHAT_SEARCH_READER_PASSWORD` | `chat_search_reader` role | Forced-RLS request reader. This is `CHAT_SEARCH_DATABASE_URL` - the only chat_search_db role a request pod ever holds. No grants on outbox/watermark; RLS policies land with track 4's table DDL. |
+| `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD` | `clickhouse` | `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=0` keeps this account from being a de facto admin; track 6's outbox consumer should provision its own scoped user once ingestion lands. |
+
+None of the four `chat_search_db` roles are superuser, table owner (except
+`chat_search_owner`, which legitimately owns the schema it migrates), or
+`BYPASSRLS` - verified live, see above.
+
+## How FerretDB auth works
+
+FerretDB 2.x does not store credentials itself - it forwards whatever
+credentials the Mongo client presents straight to PostgreSQL for validation.
+`FERRETDB_POSTGRESQL_URL`'s embedded credentials
+are also the credentials Mongo clients authenticate with:
+`mongodb://$FERRETDB_PG_USER:$FERRETDB_PG_PASSWORD@localhost:27021/?authMechanism=SCRAM-SHA-256`.
+Additional least-privilege Mongo-facing users/roles for the app itself
+(rather than this shared `ferretdb` bootstrap credential) are a track 2/4
+concern, not this track's.
+
+## `chat_search_db` roles and grants
+
+`search/init/chat-search-roles.sh` runs once via
+`docker-entrypoint-initdb.d` on a fresh volume (or safely re-run by hand -
+every statement is idempotent). It creates:
+
+1. The three Security roles from `PLAN.md` ("PostgreSQL search schema" >
+   "Security roles"): `chat_search_owner` (migration owner),
+   `chat_search_writer` (projection writer), `chat_search_reader`
+   (forced-RLS request reader).
+2. The `chat_search` schema, owned by `chat_search_owner`.
+3. `search_path = chat_search, public` on all three roles, so unqualified
+   references (`vector(1024)`, bare table names) resolve without every
+   migration having to schema-qualify - `GRANT USAGE ON SCHEMA` alone does
+   not make that happen, confirmed the hard way above.
+4. The `vector` and `pg_trgm` extensions, installed into `chat_search`
+   (pgvector for the embeddings column, pg_trgm for the trigram search arm
+   `PLAN.md` describes under `chat_search.documents`).
+5. `ALTER DEFAULT PRIVILEGES ... FOR ROLE chat_search_owner IN SCHEMA
+   chat_search`: every future table `chat_search_owner` creates
+   automatically grants `chat_search_writer` full DML plus sequence usage.
+   No default privilege is granted to `chat_search_reader` - Postgres denies
+   by default, which is exactly "reader gets no grants on outbox or
+   watermark" without the init script needing to know those two tables
+   exist yet.
+
+What track 4's migrations still have to do, per table, when they create
+`chat_search.documents` and `chat_search.embeddings` (not `outbox` or
+`watermark`):
+
+```sql
+GRANT SELECT ON chat_search.documents TO chat_search_reader;
+ALTER TABLE chat_search.documents ENABLE ROW LEVEL SECURITY;
+ALTER TABLE chat_search.documents FORCE ROW LEVEL SECURITY;
+CREATE POLICY ... ON chat_search.documents ...  -- tenant_id/user_id predicate
+```
+
+(same for `embeddings`). Forced RLS can only be applied to a table that
+exists, so this script can't do it - but the reader role, schema, and
+extensions it needs are already in place.
+
+## What depends on this stack
+
+- **Track 2 (FerretDB compatibility)** differentially tests against a
+  *separate* FerretDB instance
+  (`packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml`, port
+  27020) - not this one. This stack's `ferretdb` (27021) is the PoC's live
+  application-facing Mongo bridge.
+- **Track 3 (`rag_api`)** is out of scope for this compose file - `rag.yml`
+  and the root compose files already provision `rag_api` + `vectordb`
+  separately, and the plan's deliverable list for this track does not
+  include standing up `rag_api`. `chat_search_db`'s credentials
+  (`CHAT_SEARCH_DATABASE_URL` / `CHAT_SEARCH_WRITER_URL`) are what track 3's
+  embed-blend `fast-v1` reads chat candidate vectors through once track 4
+  wires the tables up.
+- **Track 4 (PostgreSQL search / migrations, projector, `ChatSearch`)** is
+  the primary consumer: its migrations run as `chat_search_owner` against
+  `chat_search_db` (port 5435), creating `documents`, `embeddings`,
+  `outbox`, `watermark`; its projector/reconciler runs as
+  `chat_search_writer`; the request path runs as `chat_search_reader`. Its
+  differential specs and the projector's safety poll read from `ferretdb`
+  (port 27021).
+- **Track 6 (ClickHouse historical search)** consumes this stack's
+  `clickhouse` service (ports 8123/9000) for its versioned
+  `ReplacingMergeTree` table and outbox consumer.
+- **A later CDC spike (Spike B, see `PLAN.md` "ClickPipes disposition")**
+  needs `wal_level=logical` on `ferretdb-postgres`, which is already set
+  here (`postgres -c wal_level=logical`) even though nothing in this track
+  consumes it yet.
+
+## Known follow-ups (explicitly out of scope for this track)
+
+- ClickHouse `system.query_log` and PostgreSQL statement logging are not
+  configured here (`PLAN.md` "Observability and logging", finding R27) -
+  that's track 6/7 scope, once real queries exist to worry about leaking.
+- No TLS between the app and any of these services - fine for a local PoC,
+  not for the staging shadow window (track 7).
+- `rag_api`'s connection to `vectordb` still uses the bootstrap superuser
+  (finding R1) - out of scope here since this track does not touch
+  `vectordb` at all, by design (`PLAN.md` decision 3).
--- a/search/compose.yml
+++ b/search/compose.yml
@ -0,0 +1,151 @@
+# Search stack PoC infrastructure (Track 1).
+#
+# Stands up the four services the new chat-search architecture needs, all
+# isolated from the production compose files at the repo root:
+#   - ferretdb-postgres  the DocumentDB-flavored PostgreSQL backing FerretDB
+#   - ferretdb           the Mongo wire-protocol bridge LibreChat talks to
+#   - chat_search_db     a NEW, dedicated PostgreSQL 17 + pgvector service
+#                         (NOT the existing `vectordb` used by rag_api file
+#                         search - see PLAN.md decision 3 / finding R1/R2)
+#   - clickhouse         the additive historical-search tier
+#
+# Image tags verified against their registries on 2026-08-07 (see
+# search/README.md "Image tags verified" for the curl evidence):
+#   ghcr.io/ferretdb/postgres-documentdb:17-0.107.0-ferretdb-2.7.0
+#   ghcr.io/ferretdb/ferretdb:2.7.0
+#   pgvector/pgvector:0.8.6-pg17-trixie
+#   clickhouse/clickhouse-server:26.3.17.110 (26.3 is the current ClickHouse
+#     LTS line per endoflife.date; 26.3.17.110 is its latest published patch)
+#
+# Host ports are chosen to avoid every port used by docker-compose.yml,
+# deploy-compose.yml, rag.yml, utils/docker/test-compose.yml,
+# docker-compose.override.yml(.example), and
+# packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml - see the
+# port map in search/README.md. All are overridable via search/.env.
+#
+# Usage: cp search/.env.example search/.env, fill in real credentials, then
+# from the search/ directory run `docker compose up -d`.
+
+name: librechat-search-poc
+
+services:
+  ferretdb-postgres:
+    image: ghcr.io/ferretdb/postgres-documentdb:17-0.107.0-ferretdb-2.7.0
+    container_name: search-ferretdb-postgres
+    restart: on-failure
+    # wal_level=logical is required by the Spike B ClickPipes-over-DocumentDB
+    # CDC go/no-go (see PLAN.md "ClickPipes disposition"); it is not consumed
+    # by anything in this track, only reserved for that later spike.
+    command:
+      - postgres
+      - -c
+      - wal_level=logical
+    environment:
+      POSTGRES_USER: ${FERRETDB_PG_USER:?set in search/.env, see .env.example}
+      POSTGRES_PASSWORD: ${FERRETDB_PG_PASSWORD:?set in search/.env, see .env.example}
+      POSTGRES_DB: ${FERRETDB_PG_DB:-postgres}
+    ports:
+      - "${FERRETDB_PG_HOST_PORT:-5434}:5432"
+    volumes:
+      - search_ferretdb_pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}",
+        ]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+      start_period: 10s
+
+  # FerretDB 2.x delegates all authentication to PostgreSQL: the credentials
+  # embedded in FERRETDB_POSTGRESQL_URL are also the Mongo-wire credentials
+  # clients authenticate with. No default is supplied for either - unlike the
+  # differential-test harness at
+  # packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml, this
+  # stack holds real projected chat content, so the compose file must not be
+  # runnable without operator-supplied credentials. FERRETDB_AUTH defaults to
+  # true in 2.x; set explicitly here for clarity.
+  ferretdb:
+    image: ghcr.io/ferretdb/ferretdb:2.7.0
+    container_name: search-ferretdb
+    restart: on-failure
+    depends_on:
+      ferretdb-postgres:
+        condition: service_healthy
+    ports:
+      - "${FERRETDB_HOST_PORT:-27021}:27017"
+      - "${FERRETDB_DEBUG_HOST_PORT:-8089}:8088"
+    environment:
+      FERRETDB_POSTGRESQL_URL: postgres://${FERRETDB_PG_USER:?set in search/.env, see .env.example}:${FERRETDB_PG_PASSWORD:?set in search/.env, see .env.example}@ferretdb-postgres:5432/${FERRETDB_PG_DB:-postgres}
+      FERRETDB_AUTH: "true"
+    # The FerretDB image ships its own HEALTHCHECK (`ferretdb ping`); no
+    # override needed here.
+
+  # New, dedicated PostgreSQL 17 + pgvector service. Deliberately NOT the
+  # `vectordb` service used by rag_api file search (pinned pg15, superuser
+  # bootstrap credential, live pgdata2 volume) - see PLAN.md decision 3.
+  chat_search_db:
+    image: pgvector/pgvector:0.8.6-pg17-trixie
+    container_name: search-chat-search-db
+    restart: on-failure
+    environment:
+      POSTGRES_DB: ${CHAT_SEARCH_DB:-chat_search}
+      POSTGRES_USER: ${CHAT_SEARCH_BOOTSTRAP_USER:?set in search/.env, see .env.example}
+      POSTGRES_PASSWORD: ${CHAT_SEARCH_BOOTSTRAP_PASSWORD:?set in search/.env, see .env.example}
+      CHAT_SEARCH_OWNER_PASSWORD: ${CHAT_SEARCH_OWNER_PASSWORD:?set in search/.env, see .env.example}
+      CHAT_SEARCH_WRITER_PASSWORD: ${CHAT_SEARCH_WRITER_PASSWORD:?set in search/.env, see .env.example}
+      CHAT_SEARCH_READER_PASSWORD: ${CHAT_SEARCH_READER_PASSWORD:?set in search/.env, see .env.example}
+    ports:
+      - "${CHAT_SEARCH_DB_HOST_PORT:-5435}:5432"
+    volumes:
+      - search_chat_search_pgdata:/var/lib/postgresql/data
+      - ./init/chat-search-roles.sh:/docker-entrypoint-initdb.d/01-chat-search-roles.sh:ro
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}",
+        ]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+      start_period: 10s
+
+  clickhouse:
+    image: clickhouse/clickhouse-server:26.3.17.110
+    container_name: search-clickhouse
+    restart: on-failure
+    environment:
+      CLICKHOUSE_DB: ${CLICKHOUSE_DB:-chat_search}
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-chat_search}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set in search/.env, see .env.example}
+      # The default account otherwise gets full admin rights; the outbox
+      # consumer (track 6) should provision a scoped user instead of relying
+      # on this one once real ingestion lands.
+      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 0
+    ports:
+      - "${CLICKHOUSE_HTTP_HOST_PORT:-8123}:8123"
+      - "${CLICKHOUSE_NATIVE_HOST_PORT:-9000}:9000"
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+    volumes:
+      - search_clickhouse_data:/var/lib/clickhouse
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1",
+        ]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+      start_period: 15s
+
+volumes:
+  search_ferretdb_pgdata:
+  search_chat_search_pgdata:
+  search_clickhouse_data:
--- a/search/healthcheck.sh
+++ b/search/healthcheck.sh
@ -0,0 +1,216 @@
+#!/usr/bin/env bash
+# Waits for the search PoC stack (search/compose.yml) to come up healthy and
+# verifies the chat_search_db roles exist. Safe to run repeatedly.
+#
+# Everything runs through `docker compose exec`, not host-installed
+# psql/mongosh/clickhouse-client, so this only assumes a working docker CLI.
+#
+# Usage: search/healthcheck.sh [timeout_seconds]
+#
+# Does NOT use pgrep/pkill (hangs in some sandboxes) or any destructive
+# command - read-only checks only.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+COMPOSE_FILE="$SCRIPT_DIR/compose.yml"
+TIMEOUT_SECONDS="${1:-180}"
+POLL_INTERVAL=3
+
+if [ -f "$SCRIPT_DIR/.env" ]; then
+  set -a
+  # shellcheck disable=SC1091
+  source "$SCRIPT_DIR/.env"
+  set +a
+fi
+
+if command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then
+  COMPOSE=(docker compose -f "$COMPOSE_FILE")
+elif command -v docker-compose >/dev/null 2>&1; then
+  COMPOSE=(docker-compose -f "$COMPOSE_FILE")
+else
+  echo "FAIL: neither 'docker compose' nor 'docker-compose' is available." >&2
+  echo "This is expected while Docker Desktop's WSL integration is off - rerun once it's enabled." >&2
+  exit 1
+fi
+
+SERVICES=(ferretdb-postgres ferretdb chat_search_db clickhouse)
+FAILURES=0
+
+log() { printf '%s\n' "$*"; }
+
+container_id_for() {
+  "${COMPOSE[@]}" ps -q "$1" 2>/dev/null
+}
+
+wait_for_health() {
+  local service="$1" elapsed=0 cid status
+  log "-- waiting for '$service' to report healthy (timeout ${TIMEOUT_SECONDS}s)"
+  while [ "$elapsed" -lt "$TIMEOUT_SECONDS" ]; do
+    cid="$(container_id_for "$service")"
+    if [ -z "$cid" ]; then
+      log "   [$elapsed s] container not created yet"
+    else
+      status="$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' "$cid" 2>/dev/null || echo "unknown")"
+      log "   [$elapsed s] $status"
+      if [ "$status" = "healthy" ] || [ "$status" = "running" ]; then
+        # `running` covers ferretdb-postgres's healthcheck racing container
+        # creation on the very first poll; the loop below re-checks health
+        # explicitly for images that define one.
+        if [ "$status" = "healthy" ]; then
+          return 0
+        fi
+      fi
+      if [ "$status" = "unhealthy" ]; then
+        log "   FAIL: $service reported unhealthy"
+        docker logs --tail 30 "$cid" 2>&1 | sed 's/^/     /'
+        return 1
+      fi
+    fi
+    sleep "$POLL_INTERVAL"
+    elapsed=$((elapsed + POLL_INTERVAL))
+  done
+  log "   FAIL: $service did not become healthy within ${TIMEOUT_SECONDS}s"
+  return 1
+}
+
+for svc in "${SERVICES[@]}"; do
+  if ! wait_for_health "$svc"; then
+    FAILURES=$((FAILURES + 1))
+  fi
+done
+
+if [ "$FAILURES" -gt 0 ]; then
+  log ""
+  log "FAIL: $FAILURES service(s) never became healthy; skipping application-level checks."
+  exit 1
+fi
+
+log ""
+log "== all containers healthy; running application-level checks =="
+
+# --- ferretdb-postgres: wal_level=logical (needed by the later CDC spike) ---
+log "-- ferretdb-postgres: wal_level"
+WAL_LEVEL="$("${COMPOSE[@]}" exec -T ferretdb-postgres \
+  psql -v ON_ERROR_STOP=1 -U "${FERRETDB_PG_USER:?set in search/.env}" -d "${FERRETDB_PG_DB:-postgres}" \
+  -tAc "SHOW wal_level;" 2>/dev/null | tr -d '[:space:]')"
+if [ "$WAL_LEVEL" = "logical" ]; then
+  log "   OK: wal_level=logical"
+else
+  log "   FAIL: wal_level='$WAL_LEVEL' (expected 'logical')"
+  FAILURES=$((FAILURES + 1))
+fi
+
+# --- chat_search_db: schema + three roles exist ---
+log "-- chat_search_db: schema and roles"
+ROLE_QUERY="SELECT string_agg(rolname, ',' ORDER BY rolname) FROM pg_roles WHERE rolname IN ('chat_search_owner','chat_search_writer','chat_search_reader');"
+ROLES="$("${COMPOSE[@]}" exec -T chat_search_db \
+  psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \
+  -tAc "$ROLE_QUERY" 2>/dev/null | tr -d '[:space:]')"
+EXPECTED="chat_search_owner,chat_search_reader,chat_search_writer"
+if [ "$ROLES" = "$EXPECTED" ]; then
+  log "   OK: chat_search_owner, chat_search_writer, chat_search_reader all exist"
+else
+  log "   FAIL: expected roles '$EXPECTED', found '$ROLES'"
+  FAILURES=$((FAILURES + 1))
+fi
+
+log "-- chat_search_db: no request-path role is superuser/owner/BYPASSRLS"
+LEAK_QUERY="SELECT string_agg(rolname, ',') FROM pg_roles WHERE rolname = 'chat_search_reader' AND (rolsuper OR rolbypassrls);"
+LEAKY="$("${COMPOSE[@]}" exec -T chat_search_db \
+  psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \
+  -tAc "$LEAK_QUERY" 2>/dev/null | tr -d '[:space:]')"
+if [ -z "$LEAKY" ]; then
+  log "   OK: chat_search_reader is neither superuser nor BYPASSRLS"
+else
+  log "   FAIL: chat_search_reader has an unsafe attribute"
+  FAILURES=$((FAILURES + 1))
+fi
+
+log "-- chat_search_db: chat_search schema + pgvector extension"
+SCHEMA_OK="$("${COMPOSE[@]}" exec -T chat_search_db \
+  psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \
+  -tAc "SELECT 1 FROM information_schema.schemata WHERE schema_name = 'chat_search';" 2>/dev/null | tr -d '[:space:]')"
+VECTOR_OK="$("${COMPOSE[@]}" exec -T chat_search_db \
+  psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \
+  -tAc "SELECT 1 FROM pg_extension WHERE extname = 'vector';" 2>/dev/null | tr -d '[:space:]')"
+if [ "$SCHEMA_OK" = "1" ] && [ "$VECTOR_OK" = "1" ]; then
+  log "   OK: chat_search schema exists, pgvector extension installed"
+else
+  log "   FAIL: schema present=$SCHEMA_OK vector extension present=$VECTOR_OK"
+  FAILURES=$((FAILURES + 1))
+fi
+
+# --- clickhouse: HTTP ping ---
+log "-- clickhouse: HTTP ping"
+PING="$("${COMPOSE[@]}" exec -T clickhouse \
+  wget -q -O - http://localhost:8123/ping 2>/dev/null || true)"
+if [ "$PING" = "Ok." ]; then
+  log "   OK: clickhouse HTTP ping"
+else
+  log "   FAIL: clickhouse ping returned '$PING'"
+  FAILURES=$((FAILURES + 1))
+fi
+
+# --- ferretdb: mongo-wire reachability ---
+# Three tiers, most-authoritative first: mongosh, then the repo's own
+# `mongodb` driver (already a dependency at the repo root - does a real
+# SCRAM-SHA-256 authenticated ping + insert/find/drop round trip, verified
+# working during Track 1 development), then a bare TCP check as last resort.
+log "-- ferretdb: mongo wire protocol reachability"
+FERRETDB_HOST_PORT="${FERRETDB_HOST_PORT:-27021}"
+MONGO_URI="mongodb://${FERRETDB_PG_USER:?set in search/.env}:${FERRETDB_PG_PASSWORD:?set in search/.env}@localhost:${FERRETDB_HOST_PORT}/?authMechanism=SCRAM-SHA-256"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+if command -v mongosh >/dev/null 2>&1; then
+  if mongosh --quiet --eval "db.adminCommand({ping:1})" "$MONGO_URI" >/dev/null 2>&1; then
+    log "   OK: mongosh ping succeeded on port $FERRETDB_HOST_PORT"
+  else
+    log "   FAIL: mongosh ping failed on port $FERRETDB_HOST_PORT"
+    FAILURES=$((FAILURES + 1))
+  fi
+elif command -v node >/dev/null 2>&1 && [ -d "$REPO_ROOT/node_modules/mongodb" ]; then
+  if (cd "$REPO_ROOT" && MONGO_URI="$MONGO_URI" node -e '
+      const { MongoClient } = require("mongodb");
+      (async () => {
+        const client = new MongoClient(process.env.MONGO_URI, { serverSelectionTimeoutMS: 8000 });
+        try {
+          await client.connect();
+          const ping = await client.db("admin").admin().ping();
+          if (ping.ok !== 1) throw new Error("ping.ok !== 1");
+          const col = client.db("search_healthcheck").collection("probe");
+          const { insertedId } = await col.insertOne({ probe: true, ts: new Date() });
+          const found = await col.findOne({ _id: insertedId });
+          if (!found) throw new Error("insert/find round trip failed");
+          await client.db("search_healthcheck").dropDatabase();
+        } finally {
+          await client.close();
+        }
+      })().catch((err) => { console.error(err.message); process.exit(1); });
+    ' >/dev/null 2>/tmp/ferretdb-healthcheck-node.err); then
+    log "   OK: node mongodb driver ping + insert/find/drop round trip succeeded on port $FERRETDB_HOST_PORT"
+  else
+    log "   FAIL: node mongodb driver check failed on port $FERRETDB_HOST_PORT: $(cat /tmp/ferretdb-healthcheck-node.err 2>/dev/null)"
+    FAILURES=$((FAILURES + 1))
+  fi
+else
+  # No mongosh and no local mongodb driver: fall back to a bare TCP
+  # reachability check. The container's own baked-in HEALTHCHECK
+  # (`ferretdb ping`) already verified the Mongo protocol end-to-end above,
+  # so this is a secondary signal only.
+  if (exec 3<>"/dev/tcp/localhost/${FERRETDB_HOST_PORT}") 2>/dev/null; then
+    exec 3<&- 3>&-
+    log "   OK: TCP port $FERRETDB_HOST_PORT is accepting connections (install mongosh, or run from the repo root, for a real ping)"
+  else
+    log "   FAIL: TCP port $FERRETDB_HOST_PORT is not reachable"
+    FAILURES=$((FAILURES + 1))
+  fi
+fi
+
+log ""
+if [ "$FAILURES" -eq 0 ]; then
+  log "PASS: all services healthy, roles present, no obvious leak gate violations."
+  exit 0
+else
+  log "FAIL: $FAILURES check(s) failed."
+  exit 1
+fi
--- a/search/init/chat-search-roles.sh
+++ b/search/init/chat-search-roles.sh
@ -0,0 +1,125 @@
+#!/usr/bin/env bash
+# Bootstraps chat_search_db: creates the chat_search schema, the three
+# Security roles from PLAN.md ("Security roles" under "PostgreSQL search
+# schema"), and default grants for objects the migration owner creates later.
+#
+# Runs once via docker-entrypoint-initdb.d against a fresh
+# search_chat_search_pgdata volume, connected as the bootstrap superuser
+# ($POSTGRES_USER / $POSTGRES_DB, set in search/compose.yml). Nothing in
+# chat_search_db is ever reached by the app as that bootstrap superuser -
+# see search/README.md "Credentials".
+#
+# What this script deliberately does NOT do (track 4's job - the migrations
+# in packages/api or packages/data-schemas that create
+# chat_search.{documents,embeddings,outbox,watermark}):
+#   - create any table (documents/embeddings/outbox/watermark)
+#   - GRANT chat_search_reader SELECT on documents/embeddings (do this per
+#     table, right after CREATE TABLE, running as chat_search_owner)
+#   - ALTER TABLE ... ENABLE/FORCE ROW LEVEL SECURITY + CREATE POLICY
+#     (forced RLS can only be applied to tables that exist)
+#   - grant chat_search_reader anything on outbox/watermark - the deny-by-
+#     default posture below already satisfies "reader gets no grants on
+#     outbox or watermark" as long as track 4 never adds a GRANT for it.
+set -euo pipefail
+
+: "${CHAT_SEARCH_OWNER_PASSWORD:?CHAT_SEARCH_OWNER_PASSWORD must be set (see search/.env.example)}"
+: "${CHAT_SEARCH_WRITER_PASSWORD:?CHAT_SEARCH_WRITER_PASSWORD must be set (see search/.env.example)}"
+: "${CHAT_SEARCH_READER_PASSWORD:?CHAT_SEARCH_READER_PASSWORD must be set (see search/.env.example)}"
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-'PSQL'
+-- Passwords come from the container environment via psql's backtick shell
+-- exec (runs in the postgres image's own shell, not bash string
+-- interpolation), then :'var' asks psql to SQL-quote the literal safely.
+--
+-- NOTE: :'var' substitution does not happen inside dollar-quoted DO $$ ... $$
+-- blocks (psql's lexer treats them as opaque), so idempotency below uses
+-- \gset + \if/\else/\endif client-side metacommands instead of a DO block,
+-- keeping every password-bearing CREATE/ALTER ROLE at the top level.
+\set owner_password `echo "$CHAT_SEARCH_OWNER_PASSWORD"`
+\set writer_password `echo "$CHAT_SEARCH_WRITER_PASSWORD"`
+\set reader_password `echo "$CHAT_SEARCH_READER_PASSWORD"`
+
+-- Migration owner: schema changes only. Owns the schema and every object in
+-- it, but is not superuser and cannot create roles/databases.
+SELECT COUNT(*) = 0 AS need_owner FROM pg_roles WHERE rolname = 'chat_search_owner' \gset
+\if :need_owner
+CREATE ROLE chat_search_owner LOGIN PASSWORD :'owner_password'
+  NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION;
+\else
+ALTER ROLE chat_search_owner PASSWORD :'owner_password';
+\endif
+COMMENT ON ROLE chat_search_owner IS
+  'chat_search migration owner (track 4 DDL only) - interactive/CI use, never a request-path DSN.';
+
+-- Projection writer: documents/embeddings/outbox/watermark DML. Used only
+-- by the lease-held projector/reconciler/outbox consumer, never by request
+-- pods (CHAT_SEARCH_WRITER_URL, not CHAT_SEARCH_DATABASE_URL).
+SELECT COUNT(*) = 0 AS need_writer FROM pg_roles WHERE rolname = 'chat_search_writer' \gset
+\if :need_writer
+CREATE ROLE chat_search_writer LOGIN PASSWORD :'writer_password'
+  NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION;
+\else
+ALTER ROLE chat_search_writer PASSWORD :'writer_password';
+\endif
+COMMENT ON ROLE chat_search_writer IS
+  'chat_search projection writer (projector/outbox consumer/sweep) - CHAT_SEARCH_WRITER_URL.';
+
+-- Request reader: forced RLS, request-path DSN (CHAT_SEARCH_DATABASE_URL).
+-- Not superuser, not the table owner, not BYPASSRLS - the weekend leak gate
+-- in PLAN.md asserts exactly this. No grants on outbox/watermark, ever.
+SELECT COUNT(*) = 0 AS need_reader FROM pg_roles WHERE rolname = 'chat_search_reader' \gset
+\if :need_reader
+CREATE ROLE chat_search_reader LOGIN PASSWORD :'reader_password'
+  NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION;
+\else
+ALTER ROLE chat_search_reader PASSWORD :'reader_password';
+\endif
+ALTER ROLE chat_search_reader SET row_security = on;
+COMMENT ON ROLE chat_search_reader IS
+  'chat_search forced-RLS request reader - CHAT_SEARCH_DATABASE_URL. No outbox/watermark grants.';
+
+-- Schema, owned by the migration owner.
+CREATE SCHEMA IF NOT EXISTS chat_search AUTHORIZATION chat_search_owner;
+
+-- Deny-by-default: revoke whatever PUBLIC would otherwise inherit, then
+-- grant back only what each role needs. No role here gets anything on the
+-- `public` schema either.
+REVOKE ALL ON SCHEMA chat_search FROM PUBLIC;
+REVOKE CREATE ON SCHEMA public FROM PUBLIC;
+
+GRANT USAGE ON SCHEMA chat_search TO chat_search_writer;
+GRANT USAGE ON SCHEMA chat_search TO chat_search_reader;
+
+-- All three roles resolve unqualified names (`vector(1024)`, bare table
+-- names in migrations) against chat_search first - verified empirically:
+-- USAGE on a schema is not enough for bare `vector(...)` type references,
+-- PostgreSQL only consults search_path. `public` stays second (not dropped)
+-- so built-in types/functions there remain reachable unqualified.
+ALTER ROLE chat_search_owner SET search_path = chat_search, public;
+ALTER ROLE chat_search_writer SET search_path = chat_search, public;
+ALTER ROLE chat_search_reader SET search_path = chat_search, public;
+
+-- pgvector, scoped to the chat_search schema per the search_path above.
+CREATE EXTENSION IF NOT EXISTS vector SCHEMA chat_search;
+-- pg_trgm backs the trigram search arm in PLAN.md's PostgreSQL search
+-- schema; installing it now saves track 4 a superuser round trip.
+CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA chat_search;
+
+-- Default grants for whatever chat_search_owner creates from here on
+-- (documents, embeddings, outbox, watermark - track 4's migrations): the
+-- writer gets full DML plus sequence usage automatically, so track 4 does
+-- not need to hand-grant the writer role per table.
+ALTER DEFAULT PRIVILEGES FOR ROLE chat_search_owner IN SCHEMA chat_search
+  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO chat_search_writer;
+ALTER DEFAULT PRIVILEGES FOR ROLE chat_search_owner IN SCHEMA chat_search
+  GRANT USAGE, SELECT ON SEQUENCES TO chat_search_writer;
+
+-- Deliberately no default privilege grant for chat_search_reader: Postgres
+-- denies by default, which is exactly "no grants on outbox or watermark".
+-- Track 4 must explicitly GRANT SELECT to chat_search_reader on
+-- chat_search.documents and chat_search.embeddings only, immediately after
+-- creating each table, in the same migration that applies FORCE ROW LEVEL
+-- SECURITY and the tenant/user RLS policy.
+PSQL
+
+echo "chat-search-roles: chat_search schema + roles (owner/writer/reader) ready."