Commit graph

20 commits

Author SHA1 Message Date
Marco Beretta
e36598f019
⏩ ci: Record Playwright Shard Timings Automatically and Run Fifteen Shards (#18115)
* ci: Record Playwright Shard Timings From Green Dev Runs
* ci: Run the Playwright Mock Suite on Fifteen Memory Shards
* ci: Run the Playwright Workflow Script Tests Once per Run
* test: Make the Due Schedule E2E Wait for One Engine Tick Only
* test: Cover the Shard Timings Hand-off With Acceptance Scenarios
2026-11-14 01:25:48 +01:00
Danny Avila
bd9108a87a
🧯 ci: Stop Diagnostic Artifact Uploads From Failing or Stalling Test Jobs (#18068)
On 2026-11-11 the artifact service answered every request with an HTML
page. In Playwright run 43092661470 eight memory shards failed in
"Upload Playwright HTML report" after retrying for 9 to 10.5 minutes;
in six of them every test had passed.

Uploads that no job downloads (Playwright reports, traces and
screenshots, the Lighthouse report, the advisory quality reports) are
now `continue-on-error` with a three minute step timeout, so a failed
upload stays out of the job result and a stalled one cannot run the job
into its own timeout. The Jest result uploads were already
`continue-on-error` and gain the same bound. Build outputs and mutation
shards that a later job downloads stay fatal.

merge-gate.test.mjs holds the gated workflows to that split, and
workflow.test.mts holds Change Quality to it.
2026-11-11 19:17:15 -05:00
Danny Avila
9f0646c00d
🚦 ci: Make Dev Workflows Ready for a Merge Queue (#18054)
* 🚦 ci: Make Dev Workflows Ready for a Merge Queue

* ♻️ ci: Evaluate Merge Gate Expressions With the Shared Workflow Evaluator

* 🧺 ci: Share the Installed Tree in Merge Gate Tests the Way ci-setup Requires
2026-11-11 12:06:31 -05:00
Danny Avila
97f7fcdbac
🥅 ci: Run Playwright After Merges to Dev and Guard Specs Under Test Selection (#18049)
* 🥅 ci: Run Playwright After Every Merge to Dev

The mock e2e suite ran on pull requests, nightly on main and on demand, so
when a spec broke on dev no run on dev could go red: the vote workflow that
runs the suite after merges cannot fail by design.

Add a push trigger for dev with the pull request path filter. A push runs
the pull request profile with no lane narrowed (10 memory shards, 2 Redis
transport shards, both MCP lanes and MCP Apps), and the gate job reports it.

Pushes take their own concurrency group that never cancels: the run in
progress finishes, and GitHub keeps one pending run per group, so merges
that land meanwhile collapse into one run of the newest commit. Pull
request, nightly and manual runs keep their per-ref cancelling group.

The matrix test now evaluates the workflow's own expressions per event
rather than matching their text.

* 🔭 ci: Always Run Source-Scanning Guard Specs Under Test Selection

Codegraph test selection narrows a pull request run to the test files an
import graph reaches. A spec that reads source with fs has no import edge
to what it reads: toolCapabilityGates.spec.js scans four source trees, was
not selected when #17864 changed one of the files it counts, and failed on
dev afterwards.

List every such spec, with the paths it reads, in .github/guard-specs.json.
Both selectors now add a workspace's guards to a FILES decision and turn a
NONE into a guards-only run when the change touches a path one of them
reads, renames a file, or edits the list. A list that cannot be read runs
the workspace FULL. FULL decisions and every earlier fail-open path are
unchanged.

The rule keys off the decision, not the event. A selection with no file
left in the checkout gets no guards, so the test job still runs FULL on a
stale selection instead of running only the guards.

guard-specs.test.mjs fails when a spec in a gated workspace reads the
repository through fs without being listed, when a listed path is missing,
and when a workflow would not run for a path one of its guards reads. That
last check adds the five backend and one frontend paths the triggers
lacked. The test also runs both real select steps against canned codegraph
responses.
2026-11-11 10:30:09 -05:00
Danny Avila
bb3cac9d95
🧰 ci: Provision Pinned Redis Binaries Without a Distro Mirror (#18055)
* 🧰 ci: Provision Pinned Redis Binaries Without a Distro Mirror

The MCP list_changed jobs and Cache Integration Tests installed redis-server and redis-tools with apt-get at test time. When the runners could not resolve the Ubuntu mirrors, apt-get update spent 3m20s giving up, the install stalled into the step timeout, and the next step found no redis-cli. The version was whatever the mirror served that day.

A composite action now restores redis-server and redis-cli 8.0.5 from the Actions cache and, on a miss, builds them once from the pinned upstream commit with bounded retries. The step fails with one clear error when Redis cannot be provisioned.

* 🔦 ci: Keep Lighthouse Running When the Ubuntu Mirror Is Unreachable

The Lighthouse job installed its pinned Chromium with `playwright install --with-deps`, which runs apt-get first. With the Ubuntu mirrors unresolvable, apt exited 100 and the job failed before any audit ran.

On the runner every library Chromium links is already present with the preinstalled Chrome; the only packages apt added were nine optional fonts. The browser download stays fatal and keeps the lockfile-pinned build, while the system packages go through the bounded best-effort script the Playwright jobs already use.

* ⏱️ ci: Give the Cold Redis Build Room on a Slow Runner

The first hosted runs built Redis in 49, 58 and 103 seconds. A 180 second cap left the slowest of those less than twice its own time, so a slower runner would have failed a job for being slow. The build may now take 300 seconds and the calling steps eight minutes; an unreachable host still fails within 98 seconds. The ready line reports the fetch and the build separately.
2026-11-11 10:29:25 -05:00
Dustin Healy
fdf75e0174
🔧 fix: Fence Channel Startup and Transaction Capability (#17831)
* 🔧 fix: Fence Channel Startup and Transaction Capability

* 🔧 fix: Gate Clustered Workers Until Startup Completes
2026-11-10 21:04:10 -08:00
Dustin Healy
1f27f5848a
📦 feat: Package Shared Slack Workspace Deployments (#17815)
* feat: Package Shared Slack Workspace Deployments

* fix: Require Explicit Shared Slack Deployment Prerequisites

* fix: Validate Public Slack Deployment Origins

* fix: Keep Shared Slack App Setup in Organization Administration

* 🔧 fix: Validate Shared Gateway Deployment Prerequisites
2026-11-10 21:04:10 -08:00
Dustin Healy
d269d35a2c
🐳 feat: Publish Release-Matched Channel Gateway Images (#17784)
* feat: publish release-matched channel gateway images

* test: isolate compose configuration from service env files

* test: render channel compose in a complete isolated fixture

* fix: publish stable images from their release commit

* fix: omit gateway builds from releases predating channels
2026-11-10 21:04:08 -08:00
Marco Beretta
558e0144fe
⚖️ ci: Refresh Playwright Shard Timings From Recent Runs (#17765) 2026-11-04 11:21:59 +01:00
Danny Avila
38c96a5e3a
🚦 ci: Throttle Codegraph E2E Votes to One Run per Window (#17380)
* 🚦 ci: Throttle Codegraph E2E Votes to One Run per Window

A vote runs the whole mock suite on ten runners, about 154 job-minutes, and
every merge to dev started one: 2,227 job-minutes on 2026-10-22, queued beside
PR CI for the same runners.

With repo variable CODEGRAPH_E2E_VOTE_MIN_HOURS set to N > 0, the votes
workflow skips its push runs (in a concurrency group of their own) and the new
Codegraph E2E Vote Gate decides instead: a merge starts a vote on dev
(workflow_dispatch) once the newest started vote is N hours old, and an hourly
catch-up votes the newest merge no vote covers once the window opens. A merge
votes as before while the catch-up has not run for three hours or the history
cannot be read. Unset, empty or 0 keeps today's push-triggered votes and skips
the gate; CODEGRAPH_E2E_VOTES=off still stops both.

The decision is a pure function in .github/scripts/vote-gate.mjs, tested in
vote-gate.test.mjs and run beside the shard tests in playwright-mock.yml.

* 🚦 fix: Keep an Unfinished Handoff Run from Covering Its Merge

With the throttle on, a merge creates the votes workflow's push run, whose
vote job will be skipped, and the gate's push run at about the same time.
Until that handoff run completed, the gate read it as a queued trial: it
matched the merge on head_sha, so the merge started no vote and waited for a
later catch-up tick, and it also held the window as a started vote.

While throttled, a push run that has not completed is now read as a handoff:
it neither covers a merge nor holds the window. A finished push run that
voted still counts as before.
2026-10-24 08:50:20 +02:00
Danny Avila
27e8930d52
⏱️ ci: Balance Playwright Shards by Recorded Duration (#17019)
Plan the complete mock suite's shards from recorded per-file durations instead of Playwright's equal-count split, widen the memory lanes from six to ten, and give the nightly full suite ten shards per store so it fits its 30-minute job timeout again.
2026-10-16 07:00:43 -04:00
Lia
9dc8db0ea8 🛶 fix: Preserve Canary Sync Resolutions (#16361)
Retain the tracked-file integration outcomes recorded by original canary sync PR #16361.

No new implementation is introduced. Shared and ambiguous corrections are carried by the final sync-resolution commit so the resulting tracked tree equals frozen canary f66ad88e1c.

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16361
Resolution-Sources: 6d7ba72b89, 241c692b63, 6113242b7b, 3c8a5703e7, a45be88578, 3789cbf590, 81e038972f, c505db2256, 420760d8b2
2026-10-01 03:46:19 +00:00
lia-by-librechat[bot]
c60dac37d0
🏰 ci: Gate Main Promotions on Approved Dev Commits (#16570)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Main Promotion Policy Tests / Main promotion denial and fast-forward tests (push) Has been cancelled
Co-authored-by: Lia <lia@librechat.ai>
2026-09-30 15:01:32 -04:00
lia-by-librechat[bot]
24570b68cf
🐙 ci: Spread Playwright Coverage Across Isolated Shards (#16358)
* 🐙 ci: Spread Playwright Coverage Across Isolated Shards

* ci: Allow Manual Validation of the PR Shard Profile

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-25 09:54:03 -04:00
lia-by-librechat[bot]
3a3fbb7e62
🏝️ ci: Keep Canary Pull Requests on Canary (#16285)
Co-authored-by: Lia <lia@librechat.ai>
2026-09-24 06:03:34 -04:00
Danny Avila
f20b978075
🎯 ci: Retarget Pull Requests Opened Against main to dev (#15416)
* 🎯 ci: Retarget Pull Requests Opened Against `main` to `dev`

Contributions land on `dev`; `main` only moves at release time. With `main` as
the repository's default branch, pull requests default to it, and the mismatch
has been resolved by hand at merge time.

Adds a `pull_request_target` hook that switches a newly opened or reopened pull
request's base from `main` to `dev` and posts a one-time explanation, plus a
manual `workflow_dispatch` sweep (dry-run by default) for the existing backlog.

Release-bound upstream branches keep `main` as their base: `dev`, `release/*`,
`hotfix/*`, `backport/*` and `*-main`. The `target: main` label exempts any
pull request. The exemptions apply only to branches on this repository, so a
fork branch that happens to match one of those names is still retargeted.

The job checks out the trusted base commit rather than the pull request head,
so the write-scoped token never runs contributor code.

`pull_request_target` reads its workflow definition from the base branch, so
this must exist on `main` to take effect.

* 📝 docs: Record the `dev`-First Branch Policy for Agents

`CLAUDE.md` and `AGENTS.md` never stated which branch work targets, and the
default branch is `main` — so `gh pr create` and the GitHub UI point at `main`
unless told otherwise, and an agent following the defaults gets it wrong.

States the policy, the reason a backport to `main` is never needed (`main` is
kept as a strict ancestor of `dev`), and the exemptions the retarget workflow
honors.

* 📐 docs: Record Typecheck, Formatting and Issue-Closing Gotchas for Agents

Four repeat cycle-wasters that neither file stated, each verified against the
current tree:

- `tsdown` emits without typechecking, so `npm run build` proves nothing about
  types in `packages/api`, `packages/client` or `packages/data-schemas`; only
  `packages/data-provider` runs `tsc` in its build.
- `packages/client/tsconfig.json` excludes `*.spec.ts(x)` and `*.test.ts(x)`,
  so type errors in its specs surface only at run time.
- `npm run sort-imports` with no arguments rewrites all six source roots.
- `Fixes #N` never closes an issue here, because closing keywords are honored
  only on merges into the default branch and everything merges to `dev`.

Also notes that worktrees share one stash stack, and points at
`npm run static-checks -- --against origin/dev` as the local mirror of CI.

* 🔁 fix: Address Codex Review on the Retarget Workflow

- **Backport branches are no longer exempt.** Exempting `backport/*` and
  `*-main` contradicted the invariant the policy rests on: `main` is a
  fast-forward of `dev`, so a backport merged straight to `main` is precisely
  what breaks it. `target: main` remains the deliberate, explicit escape hatch;
  `dev`, `release/*` and `hotfix/*` stay exempt as release-bound branches.
- **`npm run update` does not sync `dev`.** `config/update.js` hard-codes
  `git checkout main` and `git pull origin main` — it is the self-host
  deployment updater, not a branch-sync tool. Telling contributors to refresh
  `dev` with it would have branched them off `main` and recreated the very
  flow this change exists to prevent. Replaced with the explicit git commands.
- **A dry run no longer reports work it did not do.** The summary said
  "Retargeted N" on a dispatch that wrote nothing, which could read as a
  completed sweep.
- **A failed explanation comment is recorded, not swallowed.** The base edit
  has already landed at that point, so a later run would skip the pull request
  and the contributor would never receive the note. It now counts, fails the
  run, and names the retry: `EXPLAIN_MISSING=true`, which posts a missing
  explanation onto an already-retargeted pull request.
- **The sweep paginates.** `gh pr list --limit 500` is a cap, not a page size,
  and would have silently ignored the remainder while reporting success.

Verified against the stubbed `gh` boundary: `*-main` now retargets, the dry run
performs zero writes and says so, a failing comment yields a non-zero exit, and
`EXPLAIN_MISSING` posts onto a pull request already based on `dev`. The
paginated query returns all 258 open pull requests based on `main`.

* 🧷 fix: Address the Second Codex Round on the Retarget Workflow

- **`already_explained` now separates "lookup failed" from "no marker".** A
  transient comments-API failure returned the same status as a clean miss, so
  the run would post a second explanation and report success — defeating the
  deduplication it exists for. A failed read now declines to post and is
  recorded.
- **`CONTRIBUTING.md` syncs from the canonical remote.** The previous fix
  pulled `dev` from `origin`, which in the fork-based flow described two
  sections later is the contributor's own fork — so it would fail outright or
  silently sync a stale branch, landing them on the wrong base again.
- **The hook retries on `synchronize`.** If the `opened` run failed before
  `gh pr edit` landed, nothing retried and the pull request stayed on `main`.
  A retargeted pull request no longer matches `branches: [main]`, so this adds
  no redundant runs — and the existing backlog now drains on contributor
  activity rather than only by sweep.
- **Sweeps serialize against each other** via a shared concurrency group, while
  the hook stays keyed per pull request. Deliberately not one global group:
  GitHub keeps a single pending job per group, so a burst of openings would
  cancel queued retargets — trading a rare duplicate comment for dropped
  retargets is the wrong way round.
- **`main` is an ancestor of `dev`, not a strict one.** Right after a sync the
  two refs are equal, which the previous wording described as a violated
  invariant.

Verified: a failing comments lookup edits the base but posts nothing and exits
non-zero.
2026-08-31 11:29:34 -04:00
Danny Avila
061e4b02a3
🎞️ ci: Fix Playwright ffmpeg Install Hang and Cache the Download (#15065)
Every Playwright job spent a flat 90s on `npx playwright install ffmpeg`,
and none of them ended up with a usable ffmpeg.

The 2.3MB download finishes in under a second; extraction then hangs until
`timeout -k 10 90` reaps it (exit 124, masked by `continue-on-error`). That
is a Node 24.16.0 readable-stream change (nodejs/node#62557) colliding with
yauzl/fd-slicer never firing `close` after EOF, which hangs extract-zip.
It leaves a truncated `ffmpeg-linux` — 5,055,201 bytes against the zip's
declared 5,101,056, segfaulting on exec — and no INSTALLATION_COMPLETE
marker, so Playwright treated ffmpeg as uninstalled. `video: 'on-first-retry'`
has therefore never worked in CI, and every first retry of a flaky test died
in browserContext.newPage: exactly the failure the step existed to prevent.

Upstream fixed it in Playwright 1.60.0 (microsoft/playwright#40747) and Node
reverted it in 24.18.0 (nodejs/node#63834). Node 24.16.0 is pinned in 17
places including the Dockerfiles, so bump Playwright instead — it is a dev
dependency, and `^1.56.1` already permitted 1.62.1; only the lockfile pinned
it. Staying at or above 1.62.1 also avoids the tsconfig-resolution
regressions in 1.62.0.

Caching alone could not have fixed this: a cold cache still hangs, and what
would have been cached is the corrupt binary. So the ffmpeg download is now
restored from cache keyed on the resolved playwright-core version, the
install is skipped outright on a hit, and the cache is only saved once the
binary is verified to actually execute — a partial extraction can never be
promoted into a cache that every later job restores.

Per job: 90s to ~0s on a hit, ~2s on a miss.
2026-08-21 01:36:09 -04:00
Danny Avila
6f05f2427b
👷 ci: Stop Optional Playwright Fonts From Failing E2E (#14852)
`npx playwright install-deps chrome` is the third-most-common e2e failure:
three of the last twenty-five Playwright runs died on it, taking the whole
aggregate gate with them. The step is not installing anything CI needs.

The runner's Chrome is an apt package, so apt has already satisfied every
library Playwright lists — the log shows each one "already the newest version".
All `install-deps` adds are decorative CJK/Thai/Cyrillic font packages, ~21MB
pulled from azure.archive.ubuntu.com by seven jobs on every PR. No CI assertion
depends on them: the only spec that screenshots gates its comparison behind
`E2E_VISUAL_SNAPSHOTS`, which no workflow sets, and no baselines are committed.

Keep the install, but demote it. `google-chrome --version` becomes its own
fatal step so a genuinely missing browser still fails loudly and immediately,
while the font install retries with a per-attempt cap and degrades to a warning.
The Redis install in the list_changed job stays fatal — that one is required.
2026-08-14 20:35:37 -04:00
Danny Avila
67398e0a69
🏷️ ci: Fix Helm Tag Sync Git Auth (#13460) 2026-06-01 18:32:38 -04:00
Danny Avila
d5a2d3b26b
🏷️ ci: Sync Helm Chart Tags (#13446)
* ci: Sync Helm chart tags

* ci: Address Helm tag sync review
2026-06-01 08:43:32 -04:00