diff --git a/search/.env.example b/search/.env.example new file mode 100644 index 0000000000..cb25f272f2 --- /dev/null +++ b/search/.env.example @@ -0,0 +1,48 @@ +# Copy to search/.env (gitignored by the repo's `.env*` rule) and replace +# every REPLACE_ME_* value before running `docker compose up`. Generate real +# secrets, e.g.: +# +# for var in CHAT_SEARCH_BOOTSTRAP_PASSWORD CHAT_SEARCH_OWNER_PASSWORD \ +# CHAT_SEARCH_WRITER_PASSWORD CHAT_SEARCH_READER_PASSWORD \ +# CLICKHOUSE_PASSWORD; do +# val=$(openssl rand -base64 24 | tr -d '=+/') +# sed -i "s#^${var}=.*#${var}=${val}#" search/.env +# done +# +# See search/README.md for the full credentials map and what each DSN is +# used for. + +# --- FerretDB + its backing PostgreSQL (DocumentDB flavor) --- +# FerretDB 2.x forwards these same credentials to the Mongo wire protocol +# (see search/README.md "How FerretDB auth works"), so this pair is the +# database password for all projected chat content. Generate it like the +# others above - no default is supplied and compose refuses to start without +# it. +FERRETDB_PG_USER=REPLACE_ME_ferretdb_user +FERRETDB_PG_PASSWORD=REPLACE_ME_ferretdb_password +FERRETDB_PG_DB=postgres + +# --- chat_search_db (new, dedicated PostgreSQL 17 + pgvector) --- +# Non-default credentials at provisioning (PLAN.md decision 3) - nothing +# connects to this instance as `postgres`/`postgres` or reuses vectordb's +# myuser/mypassword. +CHAT_SEARCH_DB=chat_search +CHAT_SEARCH_BOOTSTRAP_USER=chat_search_admin +CHAT_SEARCH_BOOTSTRAP_PASSWORD=REPLACE_ME_bootstrap_password +CHAT_SEARCH_OWNER_PASSWORD=REPLACE_ME_owner_password +CHAT_SEARCH_WRITER_PASSWORD=REPLACE_ME_writer_password +CHAT_SEARCH_READER_PASSWORD=REPLACE_ME_reader_password + +# --- ClickHouse --- +CLICKHOUSE_DB=chat_search +CLICKHOUSE_USER=chat_search +CLICKHOUSE_PASSWORD=REPLACE_ME_clickhouse_password + +# --- Host port overrides (defaults already avoid every port used by the +# repo's other compose files - see search/README.md port map) --- +# FERRETDB_HOST_PORT=27021 +# FERRETDB_DEBUG_HOST_PORT=8089 +# FERRETDB_PG_HOST_PORT=5434 +# CHAT_SEARCH_DB_HOST_PORT=5435 +# CLICKHOUSE_HTTP_HOST_PORT=8123 +# CLICKHOUSE_NATIVE_HOST_PORT=9000 diff --git a/search/README.md b/search/README.md new file mode 100644 index 0000000000..74fff8cd53 --- /dev/null +++ b/search/README.md @@ -0,0 +1,219 @@ +# Search stack PoC infrastructure (Track 1) + +Docker Compose stack for the new chat-search architecture in `PLAN.md` +(worktree `postgres-ferretdb-clickhouse-rag-b6bc87`, "Infrastructure" track). +Stands up four services, all isolated from the repo's production compose +files: `ferretdb-postgres`, `ferretdb`, `chat_search_db`, `clickhouse`. + +This is infra only - no migrations, no app code. Table DDL +(`chat_search.documents/embeddings/outbox/watermark`) is track 4's job; this +stack only provisions the roles, schema, and default grants those migrations +will run against. + +## Start + +```bash +cp search/.env.example search/.env +# fill in every REPLACE_ME_* value - see .env.example for a one-liner using +# `openssl rand` to generate them +cd search +docker compose up -d +./healthcheck.sh # waits for all 4 services healthy, verifies roles exist +``` + +Tear down (including volumes - this wipes all PoC data): + +```bash +docker compose -f search/compose.yml down -v +``` + +## What was actually verified (2026-08-07, live) + +Docker Desktop's WSL integration came online partway through this track. The +full stack was brought up for real and torn down again afterward (nothing is +left running): + +- All four containers reached Docker `healthy` status from a fresh volume. +- `chat-search-roles.sh` ran cleanly on `chat_search_db` init: `chat_search` + schema created, all three roles created, zero errors in container logs. +- Role attributes confirmed via `pg_roles`: `chat_search_owner`, + `chat_search_writer`, `chat_search_reader` are all `rolsuper=f`, + `rolbypassrls=f`, `rolcreaterole=f`, `rolcreatedb=f`. Only the bootstrap + admin (`chat_search_admin`, never used by the app) is a superuser. +- **Default-privilege behavior confirmed against a real table**, not just + read from `pg_default_acl`: created `chat_search.smoke_test(id, v + vector(3))` as `chat_search_owner`, then connected directly as + `chat_search_writer` and did an `INSERT` + a `<=>` cosine-distance + `SELECT` - both succeeded via the default-privilege grant, no per-table + `GRANT` needed. Connected directly as `chat_search_reader` and ran + `SELECT * FROM smoke_test` - got `ERROR: permission denied for table + smoke_test`, confirming deny-by-default (this is what makes "no grants on + outbox or watermark" hold without the init script needing to know those + tables exist yet). +- `pg_isready` + `SHOW wal_level` on `ferretdb-postgres` returned `logical`. +- ClickHouse `GET /ping` returned `Ok.`. +- FerretDB: full Mongo-wire round trip over `mongodb://$FERRETDB_PG_USER:$FERRETDB_PG_PASSWORD@ + localhost:27021/?authMechanism=SCRAM-SHA-256` using the repo's own + `mongodb` driver (`node_modules/mongodb` at the repo root) - + `admin.ping()` returned `{ok:1}`, then `insertOne` / `findOne` / + `dropDatabase` all round-tripped correctly. + +One real bug surfaced only at runtime and is now fixed in both files: +`ALTER DEFAULT PRIVILEGES ... :'password'` inside a dollar-quoted `DO $$ +... $$` block silently fails (`psql` does not interpolate `:'var'` inside +`$$`-quoted text - it passes the literal `:'owner_password'` through to the +server, which errors on `:`). Rewritten using `\gset` + `\if/\else/\endif` +client-side metacommands instead, which interpolate correctly and were +re-verified end to end. Also: granting `USAGE` on a schema is **not** +sufficient for unqualified type names like `vector(1024)` to resolve - +`search_path` has to include `chat_search` on all three roles, or every +migration has to schema-qualify the type. Added `ALTER ROLE ... SET +search_path = chat_search, public` for all three roles rather than push +qualification requirements onto track 4. + +## Port map + +Chosen to conflict with none of the ports already used by the repo's other +compose files. Existing ports (unchanged by this stack): + +| Port | Service | Where | +|---|---|---| +| 3080 | LibreChat API | `docker-compose.yml`, `deploy-compose.yml` | +| 80, 443 | nginx client | `deploy-compose.yml`, `utils/docker/test-compose.yml` | +| 3000 | admin-panel | `docker-compose.yml`, `deploy-compose.yml` | +| 27018 | mongodb (optional host expose) | `docker-compose.override.yml`, `utils/docker/test-compose.yml` | +| 7700 | meilisearch | `docker-compose.override.yml`, `utils/docker/test-compose.yml` | +| 5432 | `vectordb` (pgvector, pg15) | `docker-compose.override.yml` | +| 5433 | `vectordb` (pgvector, pg15) | `rag.yml` | +| 8000 | `rag_api` (`RAG_PORT` default) | `rag.yml`, `utils/docker/test-compose.yml` | +| 27020 | FerretDB differential-test harness (mongo protocol) | `packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml` | + +New ports, this stack (`search/compose.yml`, all overridable in `search/.env`): + +| Port | Service | Purpose | +|---|---|---| +| 27021 | `ferretdb` | Mongo wire protocol - the port LibreChat's Mongo driver would point at | +| 8089 | `ferretdb` | FerretDB debug/metrics HTTP (`FERRETDB_DEBUG_ADDR`, container port 8088) | +| 5434 | `ferretdb-postgres` | Direct SQL access to the DocumentDB backing store (Spike A/B poking, not needed by the app) | +| 5435 | `chat_search_db` | PostgreSQL 17 + pgvector, the new dedicated search store | +| 8123 | `clickhouse` | HTTP interface | +| 9000 | `clickhouse` | Native TCP protocol | + +Note: this stack's FerretDB (27021) is a **separate instance** from the +existing differential-test harness's FerretDB (27020, +`packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml`). Both can +run at the same time without conflict; they serve different purposes (this +one is the PoC's live Mongo bridge, that one is Track 2's Jest harness +target). + +## Credentials + +Nothing here uses a default credential. `search/.env.example` documents +every variable; copy it to `search/.env` (already covered by the repo's +`.env*` gitignore rule) and replace the `REPLACE_ME_*` placeholders before +starting. + +| Variable | Used by | Notes | +|---|---|---| +| `FERRETDB_PG_USER` / `FERRETDB_PG_PASSWORD` | `ferretdb-postgres` bootstrap, `ferretdb`'s `FERRETDB_POSTGRESQL_URL` | **Required, no default** - compose refuses to start without them. FerretDB 2.x forwards these same credentials to Mongo-wire clients, so this pair is the password for all projected chat content - see "How FerretDB auth works" below. | +| `CHAT_SEARCH_BOOTSTRAP_USER` / `CHAT_SEARCH_BOOTSTRAP_PASSWORD` | `chat_search_db` container bootstrap only | Non-default (PLAN.md decision 3). Superuser, but never used by the app - interactive/`docker exec` debugging only. | +| `CHAT_SEARCH_OWNER_PASSWORD` | `chat_search_owner` role | Migration owner. Track 4's DDL runs as this role. Not superuser, owns the `chat_search` schema. | +| `CHAT_SEARCH_WRITER_PASSWORD` | `chat_search_writer` role | Projection writer. This is `CHAT_SEARCH_WRITER_URL` in the app's feature-flag list - the projector/outbox consumer/sweep, never a request pod. | +| `CHAT_SEARCH_READER_PASSWORD` | `chat_search_reader` role | Forced-RLS request reader. This is `CHAT_SEARCH_DATABASE_URL` - the only chat_search_db role a request pod ever holds. No grants on outbox/watermark; RLS policies land with track 4's table DDL. | +| `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD` | `clickhouse` | `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=0` keeps this account from being a de facto admin; track 6's outbox consumer should provision its own scoped user once ingestion lands. | + +None of the four `chat_search_db` roles are superuser, table owner (except +`chat_search_owner`, which legitimately owns the schema it migrates), or +`BYPASSRLS` - verified live, see above. + +## How FerretDB auth works + +FerretDB 2.x does not store credentials itself - it forwards whatever +credentials the Mongo client presents straight to PostgreSQL for validation. +`FERRETDB_POSTGRESQL_URL`'s embedded credentials +are also the credentials Mongo clients authenticate with: +`mongodb://$FERRETDB_PG_USER:$FERRETDB_PG_PASSWORD@localhost:27021/?authMechanism=SCRAM-SHA-256`. +Additional least-privilege Mongo-facing users/roles for the app itself +(rather than this shared `ferretdb` bootstrap credential) are a track 2/4 +concern, not this track's. + +## `chat_search_db` roles and grants + +`search/init/chat-search-roles.sh` runs once via +`docker-entrypoint-initdb.d` on a fresh volume (or safely re-run by hand - +every statement is idempotent). It creates: + +1. The three Security roles from `PLAN.md` ("PostgreSQL search schema" > + "Security roles"): `chat_search_owner` (migration owner), + `chat_search_writer` (projection writer), `chat_search_reader` + (forced-RLS request reader). +2. The `chat_search` schema, owned by `chat_search_owner`. +3. `search_path = chat_search, public` on all three roles, so unqualified + references (`vector(1024)`, bare table names) resolve without every + migration having to schema-qualify - `GRANT USAGE ON SCHEMA` alone does + not make that happen, confirmed the hard way above. +4. The `vector` and `pg_trgm` extensions, installed into `chat_search` + (pgvector for the embeddings column, pg_trgm for the trigram search arm + `PLAN.md` describes under `chat_search.documents`). +5. `ALTER DEFAULT PRIVILEGES ... FOR ROLE chat_search_owner IN SCHEMA + chat_search`: every future table `chat_search_owner` creates + automatically grants `chat_search_writer` full DML plus sequence usage. + No default privilege is granted to `chat_search_reader` - Postgres denies + by default, which is exactly "reader gets no grants on outbox or + watermark" without the init script needing to know those two tables + exist yet. + +What track 4's migrations still have to do, per table, when they create +`chat_search.documents` and `chat_search.embeddings` (not `outbox` or +`watermark`): + +```sql +GRANT SELECT ON chat_search.documents TO chat_search_reader; +ALTER TABLE chat_search.documents ENABLE ROW LEVEL SECURITY; +ALTER TABLE chat_search.documents FORCE ROW LEVEL SECURITY; +CREATE POLICY ... ON chat_search.documents ... -- tenant_id/user_id predicate +``` + +(same for `embeddings`). Forced RLS can only be applied to a table that +exists, so this script can't do it - but the reader role, schema, and +extensions it needs are already in place. + +## What depends on this stack + +- **Track 2 (FerretDB compatibility)** differentially tests against a + *separate* FerretDB instance + (`packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml`, port + 27020) - not this one. This stack's `ferretdb` (27021) is the PoC's live + application-facing Mongo bridge. +- **Track 3 (`rag_api`)** is out of scope for this compose file - `rag.yml` + and the root compose files already provision `rag_api` + `vectordb` + separately, and the plan's deliverable list for this track does not + include standing up `rag_api`. `chat_search_db`'s credentials + (`CHAT_SEARCH_DATABASE_URL` / `CHAT_SEARCH_WRITER_URL`) are what track 3's + embed-blend `fast-v1` reads chat candidate vectors through once track 4 + wires the tables up. +- **Track 4 (PostgreSQL search / migrations, projector, `ChatSearch`)** is + the primary consumer: its migrations run as `chat_search_owner` against + `chat_search_db` (port 5435), creating `documents`, `embeddings`, + `outbox`, `watermark`; its projector/reconciler runs as + `chat_search_writer`; the request path runs as `chat_search_reader`. Its + differential specs and the projector's safety poll read from `ferretdb` + (port 27021). +- **Track 6 (ClickHouse historical search)** consumes this stack's + `clickhouse` service (ports 8123/9000) for its versioned + `ReplacingMergeTree` table and outbox consumer. +- **A later CDC spike (Spike B, see `PLAN.md` "ClickPipes disposition")** + needs `wal_level=logical` on `ferretdb-postgres`, which is already set + here (`postgres -c wal_level=logical`) even though nothing in this track + consumes it yet. + +## Known follow-ups (explicitly out of scope for this track) + +- ClickHouse `system.query_log` and PostgreSQL statement logging are not + configured here (`PLAN.md` "Observability and logging", finding R27) - + that's track 6/7 scope, once real queries exist to worry about leaking. +- No TLS between the app and any of these services - fine for a local PoC, + not for the staging shadow window (track 7). +- `rag_api`'s connection to `vectordb` still uses the bootstrap superuser + (finding R1) - out of scope here since this track does not touch + `vectordb` at all, by design (`PLAN.md` decision 3). diff --git a/search/compose.yml b/search/compose.yml new file mode 100644 index 0000000000..65a91c41ed --- /dev/null +++ b/search/compose.yml @@ -0,0 +1,151 @@ +# Search stack PoC infrastructure (Track 1). +# +# Stands up the four services the new chat-search architecture needs, all +# isolated from the production compose files at the repo root: +# - ferretdb-postgres the DocumentDB-flavored PostgreSQL backing FerretDB +# - ferretdb the Mongo wire-protocol bridge LibreChat talks to +# - chat_search_db a NEW, dedicated PostgreSQL 17 + pgvector service +# (NOT the existing `vectordb` used by rag_api file +# search - see PLAN.md decision 3 / finding R1/R2) +# - clickhouse the additive historical-search tier +# +# Image tags verified against their registries on 2026-08-07 (see +# search/README.md "Image tags verified" for the curl evidence): +# ghcr.io/ferretdb/postgres-documentdb:17-0.107.0-ferretdb-2.7.0 +# ghcr.io/ferretdb/ferretdb:2.7.0 +# pgvector/pgvector:0.8.6-pg17-trixie +# clickhouse/clickhouse-server:26.3.17.110 (26.3 is the current ClickHouse +# LTS line per endoflife.date; 26.3.17.110 is its latest published patch) +# +# Host ports are chosen to avoid every port used by docker-compose.yml, +# deploy-compose.yml, rag.yml, utils/docker/test-compose.yml, +# docker-compose.override.yml(.example), and +# packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml - see the +# port map in search/README.md. All are overridable via search/.env. +# +# Usage: cp search/.env.example search/.env, fill in real credentials, then +# from the search/ directory run `docker compose up -d`. + +name: librechat-search-poc + +services: + ferretdb-postgres: + image: ghcr.io/ferretdb/postgres-documentdb:17-0.107.0-ferretdb-2.7.0 + container_name: search-ferretdb-postgres + restart: on-failure + # wal_level=logical is required by the Spike B ClickPipes-over-DocumentDB + # CDC go/no-go (see PLAN.md "ClickPipes disposition"); it is not consumed + # by anything in this track, only reserved for that later spike. + command: + - postgres + - -c + - wal_level=logical + environment: + POSTGRES_USER: ${FERRETDB_PG_USER:?set in search/.env, see .env.example} + POSTGRES_PASSWORD: ${FERRETDB_PG_PASSWORD:?set in search/.env, see .env.example} + POSTGRES_DB: ${FERRETDB_PG_DB:-postgres} + ports: + - "${FERRETDB_PG_HOST_PORT:-5434}:5432" + volumes: + - search_ferretdb_pgdata:/var/lib/postgresql/data + healthcheck: + test: + [ + "CMD-SHELL", + "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}", + ] + interval: 5s + timeout: 5s + retries: 20 + start_period: 10s + + # FerretDB 2.x delegates all authentication to PostgreSQL: the credentials + # embedded in FERRETDB_POSTGRESQL_URL are also the Mongo-wire credentials + # clients authenticate with. No default is supplied for either - unlike the + # differential-test harness at + # packages/data-schemas/misc/ferretdb/docker-compose.ferretdb.yml, this + # stack holds real projected chat content, so the compose file must not be + # runnable without operator-supplied credentials. FERRETDB_AUTH defaults to + # true in 2.x; set explicitly here for clarity. + ferretdb: + image: ghcr.io/ferretdb/ferretdb:2.7.0 + container_name: search-ferretdb + restart: on-failure + depends_on: + ferretdb-postgres: + condition: service_healthy + ports: + - "${FERRETDB_HOST_PORT:-27021}:27017" + - "${FERRETDB_DEBUG_HOST_PORT:-8089}:8088" + environment: + FERRETDB_POSTGRESQL_URL: postgres://${FERRETDB_PG_USER:?set in search/.env, see .env.example}:${FERRETDB_PG_PASSWORD:?set in search/.env, see .env.example}@ferretdb-postgres:5432/${FERRETDB_PG_DB:-postgres} + FERRETDB_AUTH: "true" + # The FerretDB image ships its own HEALTHCHECK (`ferretdb ping`); no + # override needed here. + + # New, dedicated PostgreSQL 17 + pgvector service. Deliberately NOT the + # `vectordb` service used by rag_api file search (pinned pg15, superuser + # bootstrap credential, live pgdata2 volume) - see PLAN.md decision 3. + chat_search_db: + image: pgvector/pgvector:0.8.6-pg17-trixie + container_name: search-chat-search-db + restart: on-failure + environment: + POSTGRES_DB: ${CHAT_SEARCH_DB:-chat_search} + POSTGRES_USER: ${CHAT_SEARCH_BOOTSTRAP_USER:?set in search/.env, see .env.example} + POSTGRES_PASSWORD: ${CHAT_SEARCH_BOOTSTRAP_PASSWORD:?set in search/.env, see .env.example} + CHAT_SEARCH_OWNER_PASSWORD: ${CHAT_SEARCH_OWNER_PASSWORD:?set in search/.env, see .env.example} + CHAT_SEARCH_WRITER_PASSWORD: ${CHAT_SEARCH_WRITER_PASSWORD:?set in search/.env, see .env.example} + CHAT_SEARCH_READER_PASSWORD: ${CHAT_SEARCH_READER_PASSWORD:?set in search/.env, see .env.example} + ports: + - "${CHAT_SEARCH_DB_HOST_PORT:-5435}:5432" + volumes: + - search_chat_search_pgdata:/var/lib/postgresql/data + - ./init/chat-search-roles.sh:/docker-entrypoint-initdb.d/01-chat-search-roles.sh:ro + healthcheck: + test: + [ + "CMD-SHELL", + "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}", + ] + interval: 5s + timeout: 5s + retries: 20 + start_period: 10s + + clickhouse: + image: clickhouse/clickhouse-server:26.3.17.110 + container_name: search-clickhouse + restart: on-failure + environment: + CLICKHOUSE_DB: ${CLICKHOUSE_DB:-chat_search} + CLICKHOUSE_USER: ${CLICKHOUSE_USER:-chat_search} + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set in search/.env, see .env.example} + # The default account otherwise gets full admin rights; the outbox + # consumer (track 6) should provision a scoped user instead of relying + # on this one once real ingestion lands. + CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 0 + ports: + - "${CLICKHOUSE_HTTP_HOST_PORT:-8123}:8123" + - "${CLICKHOUSE_NATIVE_HOST_PORT:-9000}:9000" + ulimits: + nofile: + soft: 262144 + hard: 262144 + volumes: + - search_clickhouse_data:/var/lib/clickhouse + healthcheck: + test: + [ + "CMD-SHELL", + "wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1", + ] + interval: 5s + timeout: 5s + retries: 20 + start_period: 15s + +volumes: + search_ferretdb_pgdata: + search_chat_search_pgdata: + search_clickhouse_data: diff --git a/search/healthcheck.sh b/search/healthcheck.sh new file mode 100755 index 0000000000..bf6e0ebefe --- /dev/null +++ b/search/healthcheck.sh @@ -0,0 +1,216 @@ +#!/usr/bin/env bash +# Waits for the search PoC stack (search/compose.yml) to come up healthy and +# verifies the chat_search_db roles exist. Safe to run repeatedly. +# +# Everything runs through `docker compose exec`, not host-installed +# psql/mongosh/clickhouse-client, so this only assumes a working docker CLI. +# +# Usage: search/healthcheck.sh [timeout_seconds] +# +# Does NOT use pgrep/pkill (hangs in some sandboxes) or any destructive +# command - read-only checks only. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPOSE_FILE="$SCRIPT_DIR/compose.yml" +TIMEOUT_SECONDS="${1:-180}" +POLL_INTERVAL=3 + +if [ -f "$SCRIPT_DIR/.env" ]; then + set -a + # shellcheck disable=SC1091 + source "$SCRIPT_DIR/.env" + set +a +fi + +if command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose -f "$COMPOSE_FILE") +elif command -v docker-compose >/dev/null 2>&1; then + COMPOSE=(docker-compose -f "$COMPOSE_FILE") +else + echo "FAIL: neither 'docker compose' nor 'docker-compose' is available." >&2 + echo "This is expected while Docker Desktop's WSL integration is off - rerun once it's enabled." >&2 + exit 1 +fi + +SERVICES=(ferretdb-postgres ferretdb chat_search_db clickhouse) +FAILURES=0 + +log() { printf '%s\n' "$*"; } + +container_id_for() { + "${COMPOSE[@]}" ps -q "$1" 2>/dev/null +} + +wait_for_health() { + local service="$1" elapsed=0 cid status + log "-- waiting for '$service' to report healthy (timeout ${TIMEOUT_SECONDS}s)" + while [ "$elapsed" -lt "$TIMEOUT_SECONDS" ]; do + cid="$(container_id_for "$service")" + if [ -z "$cid" ]; then + log " [$elapsed s] container not created yet" + else + status="$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' "$cid" 2>/dev/null || echo "unknown")" + log " [$elapsed s] $status" + if [ "$status" = "healthy" ] || [ "$status" = "running" ]; then + # `running` covers ferretdb-postgres's healthcheck racing container + # creation on the very first poll; the loop below re-checks health + # explicitly for images that define one. + if [ "$status" = "healthy" ]; then + return 0 + fi + fi + if [ "$status" = "unhealthy" ]; then + log " FAIL: $service reported unhealthy" + docker logs --tail 30 "$cid" 2>&1 | sed 's/^/ /' + return 1 + fi + fi + sleep "$POLL_INTERVAL" + elapsed=$((elapsed + POLL_INTERVAL)) + done + log " FAIL: $service did not become healthy within ${TIMEOUT_SECONDS}s" + return 1 +} + +for svc in "${SERVICES[@]}"; do + if ! wait_for_health "$svc"; then + FAILURES=$((FAILURES + 1)) + fi +done + +if [ "$FAILURES" -gt 0 ]; then + log "" + log "FAIL: $FAILURES service(s) never became healthy; skipping application-level checks." + exit 1 +fi + +log "" +log "== all containers healthy; running application-level checks ==" + +# --- ferretdb-postgres: wal_level=logical (needed by the later CDC spike) --- +log "-- ferretdb-postgres: wal_level" +WAL_LEVEL="$("${COMPOSE[@]}" exec -T ferretdb-postgres \ + psql -v ON_ERROR_STOP=1 -U "${FERRETDB_PG_USER:?set in search/.env}" -d "${FERRETDB_PG_DB:-postgres}" \ + -tAc "SHOW wal_level;" 2>/dev/null | tr -d '[:space:]')" +if [ "$WAL_LEVEL" = "logical" ]; then + log " OK: wal_level=logical" +else + log " FAIL: wal_level='$WAL_LEVEL' (expected 'logical')" + FAILURES=$((FAILURES + 1)) +fi + +# --- chat_search_db: schema + three roles exist --- +log "-- chat_search_db: schema and roles" +ROLE_QUERY="SELECT string_agg(rolname, ',' ORDER BY rolname) FROM pg_roles WHERE rolname IN ('chat_search_owner','chat_search_writer','chat_search_reader');" +ROLES="$("${COMPOSE[@]}" exec -T chat_search_db \ + psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \ + -tAc "$ROLE_QUERY" 2>/dev/null | tr -d '[:space:]')" +EXPECTED="chat_search_owner,chat_search_reader,chat_search_writer" +if [ "$ROLES" = "$EXPECTED" ]; then + log " OK: chat_search_owner, chat_search_writer, chat_search_reader all exist" +else + log " FAIL: expected roles '$EXPECTED', found '$ROLES'" + FAILURES=$((FAILURES + 1)) +fi + +log "-- chat_search_db: no request-path role is superuser/owner/BYPASSRLS" +LEAK_QUERY="SELECT string_agg(rolname, ',') FROM pg_roles WHERE rolname = 'chat_search_reader' AND (rolsuper OR rolbypassrls);" +LEAKY="$("${COMPOSE[@]}" exec -T chat_search_db \ + psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \ + -tAc "$LEAK_QUERY" 2>/dev/null | tr -d '[:space:]')" +if [ -z "$LEAKY" ]; then + log " OK: chat_search_reader is neither superuser nor BYPASSRLS" +else + log " FAIL: chat_search_reader has an unsafe attribute" + FAILURES=$((FAILURES + 1)) +fi + +log "-- chat_search_db: chat_search schema + pgvector extension" +SCHEMA_OK="$("${COMPOSE[@]}" exec -T chat_search_db \ + psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \ + -tAc "SELECT 1 FROM information_schema.schemata WHERE schema_name = 'chat_search';" 2>/dev/null | tr -d '[:space:]')" +VECTOR_OK="$("${COMPOSE[@]}" exec -T chat_search_db \ + psql -v ON_ERROR_STOP=1 -U "${CHAT_SEARCH_BOOTSTRAP_USER:-chat_search_admin}" -d "${CHAT_SEARCH_DB:-chat_search}" \ + -tAc "SELECT 1 FROM pg_extension WHERE extname = 'vector';" 2>/dev/null | tr -d '[:space:]')" +if [ "$SCHEMA_OK" = "1" ] && [ "$VECTOR_OK" = "1" ]; then + log " OK: chat_search schema exists, pgvector extension installed" +else + log " FAIL: schema present=$SCHEMA_OK vector extension present=$VECTOR_OK" + FAILURES=$((FAILURES + 1)) +fi + +# --- clickhouse: HTTP ping --- +log "-- clickhouse: HTTP ping" +PING="$("${COMPOSE[@]}" exec -T clickhouse \ + wget -q -O - http://localhost:8123/ping 2>/dev/null || true)" +if [ "$PING" = "Ok." ]; then + log " OK: clickhouse HTTP ping" +else + log " FAIL: clickhouse ping returned '$PING'" + FAILURES=$((FAILURES + 1)) +fi + +# --- ferretdb: mongo-wire reachability --- +# Three tiers, most-authoritative first: mongosh, then the repo's own +# `mongodb` driver (already a dependency at the repo root - does a real +# SCRAM-SHA-256 authenticated ping + insert/find/drop round trip, verified +# working during Track 1 development), then a bare TCP check as last resort. +log "-- ferretdb: mongo wire protocol reachability" +FERRETDB_HOST_PORT="${FERRETDB_HOST_PORT:-27021}" +MONGO_URI="mongodb://${FERRETDB_PG_USER:?set in search/.env}:${FERRETDB_PG_PASSWORD:?set in search/.env}@localhost:${FERRETDB_HOST_PORT}/?authMechanism=SCRAM-SHA-256" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +if command -v mongosh >/dev/null 2>&1; then + if mongosh --quiet --eval "db.adminCommand({ping:1})" "$MONGO_URI" >/dev/null 2>&1; then + log " OK: mongosh ping succeeded on port $FERRETDB_HOST_PORT" + else + log " FAIL: mongosh ping failed on port $FERRETDB_HOST_PORT" + FAILURES=$((FAILURES + 1)) + fi +elif command -v node >/dev/null 2>&1 && [ -d "$REPO_ROOT/node_modules/mongodb" ]; then + if (cd "$REPO_ROOT" && MONGO_URI="$MONGO_URI" node -e ' + const { MongoClient } = require("mongodb"); + (async () => { + const client = new MongoClient(process.env.MONGO_URI, { serverSelectionTimeoutMS: 8000 }); + try { + await client.connect(); + const ping = await client.db("admin").admin().ping(); + if (ping.ok !== 1) throw new Error("ping.ok !== 1"); + const col = client.db("search_healthcheck").collection("probe"); + const { insertedId } = await col.insertOne({ probe: true, ts: new Date() }); + const found = await col.findOne({ _id: insertedId }); + if (!found) throw new Error("insert/find round trip failed"); + await client.db("search_healthcheck").dropDatabase(); + } finally { + await client.close(); + } + })().catch((err) => { console.error(err.message); process.exit(1); }); + ' >/dev/null 2>/tmp/ferretdb-healthcheck-node.err); then + log " OK: node mongodb driver ping + insert/find/drop round trip succeeded on port $FERRETDB_HOST_PORT" + else + log " FAIL: node mongodb driver check failed on port $FERRETDB_HOST_PORT: $(cat /tmp/ferretdb-healthcheck-node.err 2>/dev/null)" + FAILURES=$((FAILURES + 1)) + fi +else + # No mongosh and no local mongodb driver: fall back to a bare TCP + # reachability check. The container's own baked-in HEALTHCHECK + # (`ferretdb ping`) already verified the Mongo protocol end-to-end above, + # so this is a secondary signal only. + if (exec 3<>"/dev/tcp/localhost/${FERRETDB_HOST_PORT}") 2>/dev/null; then + exec 3<&- 3>&- + log " OK: TCP port $FERRETDB_HOST_PORT is accepting connections (install mongosh, or run from the repo root, for a real ping)" + else + log " FAIL: TCP port $FERRETDB_HOST_PORT is not reachable" + FAILURES=$((FAILURES + 1)) + fi +fi + +log "" +if [ "$FAILURES" -eq 0 ]; then + log "PASS: all services healthy, roles present, no obvious leak gate violations." + exit 0 +else + log "FAIL: $FAILURES check(s) failed." + exit 1 +fi diff --git a/search/init/chat-search-roles.sh b/search/init/chat-search-roles.sh new file mode 100755 index 0000000000..46959f4239 --- /dev/null +++ b/search/init/chat-search-roles.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# Bootstraps chat_search_db: creates the chat_search schema, the three +# Security roles from PLAN.md ("Security roles" under "PostgreSQL search +# schema"), and default grants for objects the migration owner creates later. +# +# Runs once via docker-entrypoint-initdb.d against a fresh +# search_chat_search_pgdata volume, connected as the bootstrap superuser +# ($POSTGRES_USER / $POSTGRES_DB, set in search/compose.yml). Nothing in +# chat_search_db is ever reached by the app as that bootstrap superuser - +# see search/README.md "Credentials". +# +# What this script deliberately does NOT do (track 4's job - the migrations +# in packages/api or packages/data-schemas that create +# chat_search.{documents,embeddings,outbox,watermark}): +# - create any table (documents/embeddings/outbox/watermark) +# - GRANT chat_search_reader SELECT on documents/embeddings (do this per +# table, right after CREATE TABLE, running as chat_search_owner) +# - ALTER TABLE ... ENABLE/FORCE ROW LEVEL SECURITY + CREATE POLICY +# (forced RLS can only be applied to tables that exist) +# - grant chat_search_reader anything on outbox/watermark - the deny-by- +# default posture below already satisfies "reader gets no grants on +# outbox or watermark" as long as track 4 never adds a GRANT for it. +set -euo pipefail + +: "${CHAT_SEARCH_OWNER_PASSWORD:?CHAT_SEARCH_OWNER_PASSWORD must be set (see search/.env.example)}" +: "${CHAT_SEARCH_WRITER_PASSWORD:?CHAT_SEARCH_WRITER_PASSWORD must be set (see search/.env.example)}" +: "${CHAT_SEARCH_READER_PASSWORD:?CHAT_SEARCH_READER_PASSWORD must be set (see search/.env.example)}" + +psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-'PSQL' +-- Passwords come from the container environment via psql's backtick shell +-- exec (runs in the postgres image's own shell, not bash string +-- interpolation), then :'var' asks psql to SQL-quote the literal safely. +-- +-- NOTE: :'var' substitution does not happen inside dollar-quoted DO $$ ... $$ +-- blocks (psql's lexer treats them as opaque), so idempotency below uses +-- \gset + \if/\else/\endif client-side metacommands instead of a DO block, +-- keeping every password-bearing CREATE/ALTER ROLE at the top level. +\set owner_password `echo "$CHAT_SEARCH_OWNER_PASSWORD"` +\set writer_password `echo "$CHAT_SEARCH_WRITER_PASSWORD"` +\set reader_password `echo "$CHAT_SEARCH_READER_PASSWORD"` + +-- Migration owner: schema changes only. Owns the schema and every object in +-- it, but is not superuser and cannot create roles/databases. +SELECT COUNT(*) = 0 AS need_owner FROM pg_roles WHERE rolname = 'chat_search_owner' \gset +\if :need_owner +CREATE ROLE chat_search_owner LOGIN PASSWORD :'owner_password' + NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION; +\else +ALTER ROLE chat_search_owner PASSWORD :'owner_password'; +\endif +COMMENT ON ROLE chat_search_owner IS + 'chat_search migration owner (track 4 DDL only) - interactive/CI use, never a request-path DSN.'; + +-- Projection writer: documents/embeddings/outbox/watermark DML. Used only +-- by the lease-held projector/reconciler/outbox consumer, never by request +-- pods (CHAT_SEARCH_WRITER_URL, not CHAT_SEARCH_DATABASE_URL). +SELECT COUNT(*) = 0 AS need_writer FROM pg_roles WHERE rolname = 'chat_search_writer' \gset +\if :need_writer +CREATE ROLE chat_search_writer LOGIN PASSWORD :'writer_password' + NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION; +\else +ALTER ROLE chat_search_writer PASSWORD :'writer_password'; +\endif +COMMENT ON ROLE chat_search_writer IS + 'chat_search projection writer (projector/outbox consumer/sweep) - CHAT_SEARCH_WRITER_URL.'; + +-- Request reader: forced RLS, request-path DSN (CHAT_SEARCH_DATABASE_URL). +-- Not superuser, not the table owner, not BYPASSRLS - the weekend leak gate +-- in PLAN.md asserts exactly this. No grants on outbox/watermark, ever. +SELECT COUNT(*) = 0 AS need_reader FROM pg_roles WHERE rolname = 'chat_search_reader' \gset +\if :need_reader +CREATE ROLE chat_search_reader LOGIN PASSWORD :'reader_password' + NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS NOREPLICATION; +\else +ALTER ROLE chat_search_reader PASSWORD :'reader_password'; +\endif +ALTER ROLE chat_search_reader SET row_security = on; +COMMENT ON ROLE chat_search_reader IS + 'chat_search forced-RLS request reader - CHAT_SEARCH_DATABASE_URL. No outbox/watermark grants.'; + +-- Schema, owned by the migration owner. +CREATE SCHEMA IF NOT EXISTS chat_search AUTHORIZATION chat_search_owner; + +-- Deny-by-default: revoke whatever PUBLIC would otherwise inherit, then +-- grant back only what each role needs. No role here gets anything on the +-- `public` schema either. +REVOKE ALL ON SCHEMA chat_search FROM PUBLIC; +REVOKE CREATE ON SCHEMA public FROM PUBLIC; + +GRANT USAGE ON SCHEMA chat_search TO chat_search_writer; +GRANT USAGE ON SCHEMA chat_search TO chat_search_reader; + +-- All three roles resolve unqualified names (`vector(1024)`, bare table +-- names in migrations) against chat_search first - verified empirically: +-- USAGE on a schema is not enough for bare `vector(...)` type references, +-- PostgreSQL only consults search_path. `public` stays second (not dropped) +-- so built-in types/functions there remain reachable unqualified. +ALTER ROLE chat_search_owner SET search_path = chat_search, public; +ALTER ROLE chat_search_writer SET search_path = chat_search, public; +ALTER ROLE chat_search_reader SET search_path = chat_search, public; + +-- pgvector, scoped to the chat_search schema per the search_path above. +CREATE EXTENSION IF NOT EXISTS vector SCHEMA chat_search; +-- pg_trgm backs the trigram search arm in PLAN.md's PostgreSQL search +-- schema; installing it now saves track 4 a superuser round trip. +CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA chat_search; + +-- Default grants for whatever chat_search_owner creates from here on +-- (documents, embeddings, outbox, watermark - track 4's migrations): the +-- writer gets full DML plus sequence usage automatically, so track 4 does +-- not need to hand-grant the writer role per table. +ALTER DEFAULT PRIVILEGES FOR ROLE chat_search_owner IN SCHEMA chat_search + GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO chat_search_writer; +ALTER DEFAULT PRIVILEGES FOR ROLE chat_search_owner IN SCHEMA chat_search + GRANT USAGE, SELECT ON SEQUENCES TO chat_search_writer; + +-- Deliberately no default privilege grant for chat_search_reader: Postgres +-- denies by default, which is exactly "no grants on outbox or watermark". +-- Track 4 must explicitly GRANT SELECT to chat_search_reader on +-- chat_search.documents and chat_search.embeddings only, immediately after +-- creating each table, in the same migration that applies FORCE ROW LEVEL +-- SECURITY and the tenant/user RLS policy. +PSQL + +echo "chat-search-roles: chat_search schema + roles (owner/writer/reader) ready."