From bb3cac9d95aa08a9762ce76fa1701d4126d41178 Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Wed, 11 Nov 2026 10:29:25 -0500 Subject: [PATCH] =?UTF-8?q?=F0=9F=A7=B0=20ci:=20Provision=20Pinned=20Redis?= =?UTF-8?q?=20Binaries=20Without=20a=20Distro=20Mirror=20(#18055)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 🧰 ci: Provision Pinned Redis Binaries Without a Distro Mirror The MCP list_changed jobs and Cache Integration Tests installed redis-server and redis-tools with apt-get at test time. When the runners could not resolve the Ubuntu mirrors, apt-get update spent 3m20s giving up, the install stalled into the step timeout, and the next step found no redis-cli. The version was whatever the mirror served that day. A composite action now restores redis-server and redis-cli 8.0.5 from the Actions cache and, on a miss, builds them once from the pinned upstream commit with bounded retries. The step fails with one clear error when Redis cannot be provisioned. * 🔦 ci: Keep Lighthouse Running When the Ubuntu Mirror Is Unreachable The Lighthouse job installed its pinned Chromium with `playwright install --with-deps`, which runs apt-get first. With the Ubuntu mirrors unresolvable, apt exited 100 and the job failed before any audit ran. On the runner every library Chromium links is already present with the preinstalled Chrome; the only packages apt added were nine optional fonts. The browser download stays fatal and keeps the lockfile-pinned build, while the system packages go through the bounded best-effort script the Playwright jobs already use. * ⏱️ ci: Give the Cold Redis Build Room on a Slow Runner The first hosted runs built Redis in 49, 58 and 103 seconds. A 180 second cap left the slowest of those less than twice its own time, so a slower runner would have failed a job for being slow. The build may now take 300 seconds and the calling steps eight minutes; an unreachable host still fails within 98 seconds. The ready line reports the fetch and the build separately. --- .github/CODEOWNERS | 1 + .github/actions/redis-binaries/action.yml | 39 +++ .github/actions/redis-binaries/pin.json | 6 + .github/actions/redis-binaries/provision.mjs | 316 ++++++++++++++++++ .../actions/redis-binaries/provision.test.mjs | 301 +++++++++++++++++ .github/scripts/install-playwright-fonts.sh | 11 +- .github/workflows/cache-integration-tests.yml | 33 +- .github/workflows/lighthouse.yml | 11 +- .github/workflows/playwright-mock.yml | 21 +- 9 files changed, 703 insertions(+), 36 deletions(-) create mode 100644 .github/actions/redis-binaries/action.yml create mode 100644 .github/actions/redis-binaries/pin.json create mode 100644 .github/actions/redis-binaries/provision.mjs create mode 100644 .github/actions/redis-binaries/provision.test.mjs diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a39dace850..36afbcba04 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -2,4 +2,5 @@ /.github/CODEOWNERS @danny-avila /.github/workflows/ @danny-avila /.github/scripts/ @danny-avila +/.github/actions/ @danny-avila /.github/MAIN_PROMOTION.md @danny-avila diff --git a/.github/actions/redis-binaries/action.yml b/.github/actions/redis-binaries/action.yml new file mode 100644 index 0000000000..70a0ab635d --- /dev/null +++ b/.github/actions/redis-binaries/action.yml @@ -0,0 +1,39 @@ +name: Redis binaries +description: >- + Puts redis-server and redis-cli at the Redis version pinned in pin.json on PATH without a + package manager: restores them from the Actions cache and, when the cache has none for this + pin and runner OS release, builds them once from the pinned upstream commit. Use after Node.js + is set up and before anything starts Redis or calls redis-cli. + +runs: + using: composite + steps: + - name: Resolve the pinned Redis build + id: pin + shell: bash + run: node "$GITHUB_ACTION_PATH/provision.mjs" key + + # No restore-keys: binaries of another pin or another OS release are never a starting point. + - name: Restore Redis binaries + id: cache + uses: actions/cache/restore@v5 + with: + path: ~/.cache/redis-binaries + key: ${{ steps.pin.outputs.key }} + + - name: Provision redis-server and redis-cli + shell: bash + env: + REDIS_BINARIES_CACHE_HIT: ${{ steps.cache.outputs.cache-hit }} + run: node "$GITHUB_ACTION_PATH/provision.mjs" + + # Saved here rather than in a post-job step: only binaries that were just proven to run are + # promoted into the cache every later job restores, and a failing test run still leaves + # them behind for the next one. + - name: Save Redis binaries + if: steps.cache.outputs.cache-hit != 'true' + continue-on-error: true + uses: actions/cache/save@v5 + with: + path: ~/.cache/redis-binaries + key: ${{ steps.cache.outputs.cache-primary-key }} diff --git a/.github/actions/redis-binaries/pin.json b/.github/actions/redis-binaries/pin.json new file mode 100644 index 0000000000..e6e4eb200e --- /dev/null +++ b/.github/actions/redis-binaries/pin.json @@ -0,0 +1,6 @@ +{ + "version": "8.0.5", + "commit": "214a4905c3edb6b69ac9a9c5d769b7d3fb81ed54", + "repository": "https://github.com/redis/redis.git", + "flags": ["BUILD_TLS=no", "USE_SYSTEMD=no"] +} diff --git a/.github/actions/redis-binaries/provision.mjs b/.github/actions/redis-binaries/provision.mjs new file mode 100644 index 0000000000..2507c3efa5 --- /dev/null +++ b/.github/actions/redis-binaries/provision.mjs @@ -0,0 +1,316 @@ +#!/usr/bin/env node +/** + * Puts `redis-server` and `redis-cli` at the pinned Redis version into + * `~/.cache/redis-binaries/bin`, proves they run, and adds that directory to + * PATH for the steps that follow. + * + * Nothing here talks to a package manager. A job that installed Redis with + * apt needed the distro mirror to resolve at test time, and got whichever + * version the mirror served that day. The binaries are instead restored from + * the Actions cache; when the cache has none that match the pin, they are + * built once from the pinned upstream commit. That fetch goes to the host the + * job's own checkout already depended on, and git only accepts an object + * whose content hashes to the commit asked for, so the pin is the checksum. + * + * `pin.json` holds everything that decides which binaries come out: version, + * commit, repository and make flags. `key` prints the cache key derived from + * it, from this file and from the runner's OS release, because a binary + * linked against one release's libc is not trusted on another. + * + * Exits non-zero, with one `::error` line, when no runnable pinned binaries + * could be put in place: a later step never meets a missing `redis-cli`. + */ +import os from 'node:os'; +import path from 'node:path'; +import { createHash } from 'node:crypto'; +import { fileURLToPath } from 'node:url'; +import { spawnSync } from 'node:child_process'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { + appendFileSync, + chmodSync, + closeSync, + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + openSync, + readFileSync, + rmSync, +} from 'node:fs'; + +const SCRIPT = fileURLToPath(import.meta.url); +const PIN_FILE = path.join(path.dirname(SCRIPT), 'pin.json'); +const BINARIES = ['redis-server', 'redis-cli']; +const HOME_CACHE = path.join(os.homedir(), '.cache', 'redis-binaries'); +const BIN = path.join(HOME_CACHE, 'bin'); + +/** + * An unreachable host fails within 30 + 2 + 30 + 6 + 30 s. Hosted runners + * needed 49 to 103 s for the whole cold path, so the build may take about + * three times the slowest of those; fetch and build together stay inside an + * eight-minute step. + */ +const FETCH_TIMEOUT_MS = 30_000; +const FETCH_BACKOFF_MS = [2_000, 6_000]; +const FETCH_ATTEMPTS = FETCH_BACKOFF_MS.length + 1; +const BUILD_TIMEOUT_MS = 300_000; +const VERSION_TIMEOUT_MS = 10_000; +const BUILD_LOG_TAIL = 40; + +/** A workflow command ends at the first newline, so a multi-line message is folded into one. */ +const headline = (error) => + String(error?.message ?? error) + .split(/\s*\n\s*/) + .filter(Boolean) + .join(' '); + +function readPin() { + const pin = JSON.parse(readFileSync(PIN_FILE, 'utf8')); + const valid = + typeof pin.version === 'string' && + /^[0-9a-f]{40}$/.test(pin.commit) && + typeof pin.repository === 'string' && + Array.isArray(pin.flags); + if (!valid) { + throw new Error(`${PIN_FILE} must name a version, a full commit SHA, a repository and flags`); + } + return pin; +} + +function fail(error) { + console.error(error); + console.error(`::error title=Redis binaries::${headline(error)}`); + process.exit(1); +} + +function loadPin() { + try { + return readPin(); + } catch (error) { + return fail(error); + } +} + +const PIN = loadPin(); +const SHORT_COMMIT = PIN.commit.slice(0, 8); + +/** Best effort: a compiler orphaned by a timeout may still be writing into the directory. */ +function discard(directory) { + try { + rmSync(directory, { recursive: true, force: true }); + } catch { + return; + } +} + +/** Runs a command to completion; throws with the reason when it fails, is missing or times out. */ +function run(command, args, options = {}) { + const result = spawnSync(command, args, { + encoding: 'utf8', + killSignal: 'SIGKILL', + ...options, + }); + if (result.status === 0) { + return result.stdout ?? ''; + } + const timedOut = result.error?.code === 'ETIMEDOUT'; + const reason = timedOut + ? `timed out after ${options.timeout / 1000}s` + : result.error?.message || + result.stderr?.trim() || + `exit ${result.status ?? result.signal ?? 'unknown'}`; + throw new Error(`${command} ${args[0]}: ${reason}`); +} + +function osRelease() { + if (process.platform !== 'linux') { + return `${process.platform}-${os.release().split('.')[0]}`; + } + try { + const release = readFileSync('/etc/os-release', 'utf8'); + const field = (name) => release.match(new RegExp(`^${name}="?([^"\\n]+)"?$`, 'm'))?.[1]; + return `${field('ID') ?? 'linux'}-${field('VERSION_ID') ?? os.release()}`; + } catch { + return `linux-${os.release()}`; + } +} + +function cacheKey() { + const recipe = createHash('sha256') + .update(readFileSync(SCRIPT)) + .update(readFileSync(PIN_FILE)) + .digest('hex') + .slice(0, 16); + return `redis-binaries-${osRelease()}-${process.arch}-${PIN.version}-${recipe}`; +} + +/** The version line of one cached binary; throws unless it runs and is the pinned build. */ +function identify(name) { + const binary = path.join(BIN, name); + const line = run(binary, ['--version'], { timeout: VERSION_TIMEOUT_MS }).split('\n', 1)[0].trim(); + const match = + name === 'redis-server' + ? line.match(/^Redis server v=(\S+) sha=([0-9a-f]{8})/) + : line.match(/^redis-cli (\S+) \(git:([0-9a-f]{8})\)/); + if (match?.[1] !== PIN.version || match?.[2] !== SHORT_COMMIT) { + throw new Error( + `${binary} is not Redis ${PIN.version} built from ${SHORT_COMMIT}: ${line || 'no version line'}`, + ); + } + return line; +} + +function git(args, directory, timeout) { + return run('git', args, { + cwd: directory, + timeout, + env: { ...process.env, GIT_TERMINAL_PROMPT: '0' }, + }); +} + +/** Each attempt gets its own repository, so a fetch killed by its timeout leaves no lock behind. */ +function fetchCommit(root, attempt) { + const directory = path.join(root, `attempt-${attempt}`); + mkdirSync(directory); + git(['init', '--quiet'], directory); + git( + ['fetch', '--quiet', '--depth', '1', '--no-tags', PIN.repository, PIN.commit], + directory, + FETCH_TIMEOUT_MS, + ); + git( + ['-c', 'advice.detachedHead=false', 'checkout', '--quiet', '--detach', 'FETCH_HEAD'], + directory, + ); + const head = git(['rev-parse', 'HEAD'], directory).trim(); + if (head !== PIN.commit) { + throw new Error(`${PIN.repository} answered with ${head}, not the pinned ${PIN.commit}`); + } + return directory; +} + +async function checkout(root, attempt = 1) { + try { + return fetchCommit(root, attempt); + } catch (error) { + if (attempt === FETCH_ATTEMPTS) { + throw new Error( + `Could not fetch Redis ${PIN.version} (${PIN.commit}) from ${PIN.repository} after ${FETCH_ATTEMPTS} attempts: ${headline(error)}`, + ); + } + const delay = FETCH_BACKOFF_MS[attempt - 1]; + console.log( + `::warning::${headline(error)}. Retrying in ${delay / 1000}s (attempt ${attempt + 1} of ${FETCH_ATTEMPTS}).`, + ); + await sleep(delay); + } + return checkout(root, attempt + 1); +} + +/** Compiler output goes to a file; only its tail is shown, and only when the build fails. */ +function build(directory) { + const log = path.join(directory, 'redis-binaries-build.log'); + const output = openSync(log, 'w'); + const jobs = String(os.availableParallelism()); + try { + run('make', ['-C', 'src', '-j', jobs, ...PIN.flags, ...BINARIES], { + cwd: directory, + timeout: BUILD_TIMEOUT_MS, + stdio: ['ignore', output, output], + }); + } catch (error) { + const tail = readFileSync(log, 'utf8').trimEnd().split('\n').slice(-BUILD_LOG_TAIL); + console.log(`::group::Last ${tail.length} lines of the Redis build`); + console.log(tail.join('\n')); + console.log('::endgroup::'); + throw new Error(`Building Redis ${PIN.version} failed: ${headline(error)}`); + } finally { + closeSync(output); + } +} + +function install(directory) { + discard(BIN); + mkdirSync(BIN, { recursive: true }); + for (const name of BINARIES) { + copyFileSync(path.join(directory, 'src', name), path.join(BIN, name)); + chmodSync(path.join(BIN, name), 0o755); + } +} + +/** Nothing unusable stays behind: a failed build leaves the cache directory without binaries. */ +async function buildFromSource() { + const started = Date.now(); + const root = mkdtempSync(path.join(process.env.RUNNER_TEMP || os.tmpdir(), 'redis-source-')); + try { + const directory = await checkout(root); + const fetched = Date.now(); + build(directory); + install(directory); + const lines = BINARIES.map(identify); + const seconds = (from, to) => Math.round((to - from) / 1000); + const took = `fetch ${seconds(started, fetched)}s, build ${seconds(fetched, Date.now())}s`; + return { lines, source: `built from ${PIN.repository} at ${SHORT_COMMIT}: ${took}` }; + } catch (error) { + discard(BIN); + throw error; + } finally { + discard(root); + } +} + +async function provision() { + if (!BINARIES.some((name) => existsSync(path.join(BIN, name)))) { + console.log(`No Redis binaries in ${BIN}; building ${PIN.version} from ${SHORT_COMMIT}.`); + return buildFromSource(); + } + try { + return { lines: BINARIES.map(identify), source: 'already present' }; + } catch (error) { + console.log(`Rebuilding Redis ${PIN.version}: ${headline(error)}`); + return buildFromSource(); + } +} + +function exportPath() { + if (!process.env.GITHUB_PATH) { + console.log(` GITHUB_PATH is not set; add ${BIN} to PATH yourself`); + return; + } + appendFileSync(process.env.GITHUB_PATH, `${BIN}\n`); +} + +async function main() { + const result = await provision(); + console.log(`Redis ${PIN.version} ready in ${BIN} (${result.source})`); + for (const line of result.lines) { + console.log(` ${line}`); + } + exportPath(); + if (result.source !== 'already present' && process.env.REDIS_BINARIES_CACHE_HIT === 'true') { + console.log( + '::warning::The restored Redis binary cache had no runnable pinned build, and an exact cache hit is never re-saved: every job rebuilds Redis until the cache entry is deleted or the pin changes.', + ); + } +} + +function printKey() { + const key = cacheKey(); + console.log(key); + if (process.env.GITHUB_OUTPUT) { + appendFileSync(process.env.GITHUB_OUTPUT, `key=${key}\n`); + } +} + +const command = process.argv[2] ?? 'provision'; +const commands = { key: printKey, provision: main }; + +Promise.resolve() + .then(() => { + if (!Object.hasOwn(commands, command)) { + throw new Error(`Unknown command "${command}"; expected "provision" or "key"`); + } + return commands[command](); + }) + .catch(fail); diff --git a/.github/actions/redis-binaries/provision.test.mjs b/.github/actions/redis-binaries/provision.test.mjs new file mode 100644 index 0000000000..189df1d19b --- /dev/null +++ b/.github/actions/redis-binaries/provision.test.mjs @@ -0,0 +1,301 @@ +import os from 'node:os'; +import path from 'node:path'; +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { fileURLToPath } from 'node:url'; +import { spawnSync } from 'node:child_process'; +import { + chmodSync, + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, + writeFileSync, +} from 'node:fs'; + +/** + * Runs the real script, with real git and real make, against a stand-in for + * the Redis repository: a local repository whose `src/Makefile` "builds" + * `redis-server` and `redis-cli` as shell scripts that print the version lines + * the real binaries print, embedding the commit they were built from the way + * the Redis build does. The script reads the pin from the file beside it, so + * each test runs a copy of it next to a pin for that stand-in. Only the + * network is replaced: a `git` first on PATH counts fetches and can fail the + * first few before handing over to the real one. + */ +const SCRIPT = fileURLToPath(new URL('./provision.mjs', import.meta.url)); +const VERSION = '9.9.9'; +const posixOnly = { + skip: process.platform === 'win32' && 'the stand-in binaries are shell scripts', +}; + +const MAKEFILE = ` +SHA := $(shell git rev-parse --short=8 HEAD) +VERSION := __VERSION__ +redis-server: +\t@printf '#!/bin/sh\\necho "Redis server v=%s sha=%s:0 malloc=libc bits=64 build=tls-%s-systemd-%s"\\n' '$(VERSION)' '$(SHA)' '$(BUILD_TLS)' '$(USE_SYSTEMD)' > $@ +\t@chmod +x $@ +redis-cli: +\t@printf '#!/bin/sh\\necho "redis-cli %s (git:%s)"\\n' '$(VERSION)' '$(SHA)' > $@ +\t@chmod +x $@ +`; + +const BROKEN_MAKEFILE = ` +redis-server redis-cli: +\t@echo "cc: fatal error: no input files" >&2 +\t@exit 2 +`; + +const GIT_SHIM = `#!/bin/sh +if [ "$1" = fetch ]; then + echo fetch >> "$FAKE_FETCH_LOG" + if [ "$(wc -l < "$FAKE_FETCH_LOG")" -le "\${FAKE_FETCH_FAILURES:-0}" ]; then + echo "fatal: unable to access 'https://example.invalid/': Could not resolve host" >&2 + exit 128 + fi +fi +exec "$REAL_GIT" "$@" +`; + +const REAL_GIT = spawnSync('sh', ['-c', 'command -v git'], { encoding: 'utf8' }).stdout.trim(); + +/** The developer's own git configuration (signing, hooks, templates) stays out of the stand-in. */ +const HERMETIC_GIT = { ...process.env, GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1' }; + +function sh(command, args, cwd) { + const result = spawnSync(command, args, { cwd, encoding: 'utf8', env: HERMETIC_GIT }); + assert.equal(result.status, 0, `${command} ${args.join(' ')}: ${result.stderr}`); + return result.stdout.trim(); +} + +function fixture(t, { makefile = MAKEFILE.replace('__VERSION__', VERSION) } = {}) { + const root = mkdtempSync(path.join(os.tmpdir(), 'redis-binaries-')); + t.after(() => rmSync(root, { recursive: true, force: true })); + const home = path.join(root, 'home'); + const action = path.join(root, 'action'); + const upstream = path.join(root, 'upstream'); + const shim = path.join(root, 'shim'); + const bin = path.join(home, '.cache', 'redis-binaries', 'bin'); + const fetchLog = path.join(root, 'fetches.log'); + const githubPath = path.join(root, 'github-path'); + const githubOutput = path.join(root, 'github-output'); + for (const directory of [home, action, path.join(upstream, 'src'), shim]) { + mkdirSync(directory, { recursive: true }); + } + + writeFileSync(path.join(upstream, 'src', 'Makefile'), makefile); + sh('git', ['init', '--quiet', '--initial-branch', 'main'], upstream); + sh('git', ['add', '.'], upstream); + sh( + 'git', + ['-c', 'user.name=fixture', '-c', 'user.email=fixture@example.invalid', 'commit', '-qm', 'x'], + upstream, + ); + const commit = sh('git', ['rev-parse', 'HEAD'], upstream); + + writeFileSync(path.join(shim, 'git'), GIT_SHIM); + chmodSync(path.join(shim, 'git'), 0o755); + copyFileSync(SCRIPT, path.join(action, 'provision.mjs')); + const pin = (overrides = {}) => + writeFileSync( + path.join(action, 'pin.json'), + JSON.stringify({ + version: VERSION, + commit, + repository: upstream, + flags: ['BUILD_TLS=no', 'USE_SYSTEMD=no'], + ...overrides, + }), + ); + pin(); + + const run = (args = [], env = {}) => { + const result = spawnSync(process.execPath, [path.join(action, 'provision.mjs'), ...args], { + cwd: root, + encoding: 'utf8', + env: { + ...process.env, + HOME: home, + PATH: `${shim}${path.delimiter}${process.env.PATH}`, + RUNNER_TEMP: root, + REAL_GIT, + FAKE_FETCH_LOG: fetchLog, + GITHUB_PATH: githubPath, + GITHUB_OUTPUT: githubOutput, + REDIS_BINARIES_CACHE_HIT: '', + ...env, + }, + }); + return { status: result.status, output: `${result.stdout}${result.stderr}` }; + }; + const place = (version, sha) => { + mkdirSync(bin, { recursive: true }); + const lines = { + 'redis-server': `Redis server v=${version} sha=${sha}:0 malloc=libc bits=64 build=old`, + 'redis-cli': `redis-cli ${version} (git:${sha})`, + }; + for (const [name, line] of Object.entries(lines)) { + writeFileSync(path.join(bin, name), `#!/bin/sh\necho "${line}"\n`); + chmodSync(path.join(bin, name), 0o755); + } + }; + const read = (file) => (existsSync(file) ? readFileSync(file, 'utf8') : ''); + const fetches = () => read(fetchLog).split('\n').filter(Boolean).length; + const cached = () => (existsSync(bin) ? readdirSync(bin).sort() : []); + const leftovers = () => readdirSync(root).filter((name) => name.startsWith('redis-source-')); + const exported = () => read(githubPath); + const output = () => read(githubOutput); + return { bin, commit, pin, run, place, fetches, cached, leftovers, exported, output }; +} + +test('a cold cache is filled by one fetch of the pinned commit and one build', posixOnly, (t) => { + const { bin, commit, run, fetches, cached, leftovers, exported } = fixture(t); + const { status, output } = run(); + assert.equal(status, 0, output); + assert.match( + output, + new RegExp( + `Redis 9\\.9\\.9 ready in ${bin} \\(built from .* at ${commit.slice(0, 8)}: fetch \\d+s, build \\d+s\\)`, + ), + ); + assert.match(output, new RegExp(`Redis server v=9\\.9\\.9 sha=${commit.slice(0, 8)}:0`)); + assert.match(output, /build=tls-no-systemd-no/, 'the pinned flags reach make'); + assert.match(output, new RegExp(`redis-cli 9\\.9\\.9 \\(git:${commit.slice(0, 8)}\\)`)); + assert.match(output, new RegExp(`No Redis binaries in ${bin}; building 9\\.9\\.9 from`)); + assert.doesNotMatch(output, /::warning::|::error/); + assert.equal(fetches(), 1); + assert.deepEqual(cached(), ['redis-cli', 'redis-server']); + assert.equal(exported(), `${bin}\n`); + assert.deepEqual(leftovers(), [], 'the source tree is removed once the binaries are in place'); +}); + +test('binaries already in the cache are verified and nothing is fetched', posixOnly, (t) => { + const { bin, run, fetches, exported } = fixture(t); + assert.equal(run().status, 0); + const { status, output } = run([], { + FAKE_FETCH_FAILURES: '99', + REDIS_BINARIES_CACHE_HIT: 'true', + }); + assert.equal(status, 0, output); + assert.match(output, /Redis 9\.9\.9 ready in .* \(already present\)/); + assert.doesNotMatch(output, /::warning::/); + assert.equal(fetches(), 1, 'the second run needs no network'); + assert.equal(exported(), `${bin}\n${bin}\n`); +}); + +test('cached binaries of another version or commit are rebuilt, with a warning', posixOnly, (t) => { + const { commit, run, place, fetches } = fixture(t); + for (const [version, sha] of [ + ['9.9.8', commit.slice(0, 8)], + [VERSION, '0badc0de'], + ]) { + place(version, sha); + const { status, output } = run([], { REDIS_BINARIES_CACHE_HIT: 'true' }); + assert.equal(status, 0, output); + assert.match(output, /Rebuilding Redis 9\.9\.9: .* is not Redis 9\.9\.9 built from/); + assert.match(output, /\(built from /); + assert.match(output, /::warning::The restored Redis binary cache had no runnable pinned build/); + } + assert.equal(fetches(), 2); +}); + +test('a fetch that fails once is retried and the build goes ahead', posixOnly, (t) => { + const { run, fetches, cached } = fixture(t); + const { status, output } = run([], { FAKE_FETCH_FAILURES: '1' }); + assert.equal(status, 0, output); + assert.match( + output, + /::warning::git fetch: fatal: unable to access .* Retrying in 2s \(attempt 2 of 3\)/, + ); + assert.equal(fetches(), 2); + assert.deepEqual(cached(), ['redis-cli', 'redis-server']); +}); + +test( + 'the job fails after three attempts when the pinned commit cannot be fetched', + posixOnly, + (t) => { + const { pin, run, fetches, cached, leftovers, exported } = fixture(t); + const missing = 'f'.repeat(40); + pin({ commit: missing }); + const { status, output } = run(); + assert.equal(status, 1, output); + assert.match( + output, + new RegExp( + `^::error title=Redis binaries::Could not fetch Redis 9\\.9\\.9 \\(${missing}\\) from .* after 3 attempts: git fetch: `, + 'm', + ), + ); + assert.equal(fetches(), 3, 'bounded, not a loop'); + assert.deepEqual(cached(), []); + assert.deepEqual(leftovers(), []); + assert.equal(exported(), '', 'no later step is pointed at binaries that do not exist'); + }, +); + +test('the job fails with the end of the build log when Redis does not compile', posixOnly, (t) => { + const { run, cached, leftovers, exported } = fixture(t, { makefile: BROKEN_MAKEFILE }); + const { status, output } = run(); + assert.equal(status, 1, output); + assert.match(output, /::group::Last \d+ lines of the Redis build/); + assert.match(output, /cc: fatal error: no input files/); + assert.match(output, /^::error title=Redis binaries::Building Redis 9\.9\.9 failed: make -C: /m); + assert.deepEqual(cached(), []); + assert.deepEqual(leftovers(), []); + assert.equal(exported(), ''); +}); + +test( + 'a build that is not the pinned version is rejected and not left in the cache', + posixOnly, + (t) => { + const { run, cached, exported } = fixture(t, { + makefile: MAKEFILE.replace('__VERSION__', '9.9.8'), + }); + const { status, output } = run(); + assert.equal(status, 1, output); + assert.match( + output, + /^::error title=Redis binaries::.*redis-server is not Redis 9\.9\.9 built from/m, + ); + assert.deepEqual(cached(), []); + assert.equal(exported(), ''); + }, +); + +test('the cache key follows the pin and the script, and is stable otherwise', posixOnly, (t) => { + const { commit, pin, run, output } = fixture(t); + const key = () => { + const result = run(['key']); + assert.equal(result.status, 0, result.output); + return result.output.trim(); + }; + const first = key(); + assert.match(first, new RegExp(`^redis-binaries-.+-${process.arch}-9\\.9\\.9-[0-9a-f]{16}$`)); + assert.equal(key(), first); + assert.equal(output(), `key=${first}\nkey=${first}\n`); + pin({ flags: ['BUILD_TLS=yes', 'USE_SYSTEMD=no'] }); + const flagged = key(); + assert.notEqual(flagged, first); + pin({ commit: commit.replace(/.$/, (last) => (last === '0' ? '1' : '0')) }); + assert.notEqual(key(), first); + assert.notEqual(key(), flagged); +}); + +test('a malformed pin or an unknown command fails with one error line', posixOnly, (t) => { + const { pin, run } = fixture(t); + const unknown = run(['install']); + assert.equal(unknown.status, 1, unknown.output); + assert.match(unknown.output, /^::error title=Redis binaries::Unknown command "install"/m); + pin({ commit: 'main' }); + const malformed = run(); + assert.equal(malformed.status, 1, malformed.output); + assert.match( + malformed.output, + /^::error title=Redis binaries::.*pin\.json must name a version, a full commit SHA/m, + ); +}); diff --git a/.github/scripts/install-playwright-fonts.sh b/.github/scripts/install-playwright-fonts.sh index ada4fa5537..0c200a42a7 100755 --- a/.github/scripts/install-playwright-fonts.sh +++ b/.github/scripts/install-playwright-fonts.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash # -# Installs Playwright's optional font packages for the `chrome` channel. +# Installs Playwright's optional font packages for the `chrome` channel, or for +# the browser named as the first argument. # # The GitHub runner ships Chrome as an apt package, so every library Playwright # lists is already satisfied by apt itself. The only packages `install-deps` adds @@ -8,6 +9,11 @@ # the one spec that takes screenshots gates the comparison behind # `E2E_VISUAL_SNAPSHOTS`, which CI never sets. # +# Lighthouse passes `chromium`: its lockfile-pinned build links the same +# libraries the runner's Chrome already brought in, so on the runner the same +# fonts are all that is missing, and a launch that did lack a library fails in +# the workflow's own browser check rather than here. +# # Ubuntu's mirrors stall often enough that a hard failure here has repeatedly # taken down whole e2e runs, so each attempt is capped and a final failure is # only a warning. The workflow keeps `continue-on-error: true` as a backstop for @@ -15,11 +21,12 @@ set -uo pipefail +readonly BROWSER="${1:-chrome}" readonly ATTEMPT_TIMEOUT_SECONDS=70 readonly MAX_ATTEMPTS=3 for attempt in $(seq 1 "${MAX_ATTEMPTS}"); do - if timeout "${ATTEMPT_TIMEOUT_SECONDS}" npx playwright install-deps chrome; then + if timeout "${ATTEMPT_TIMEOUT_SECONDS}" npx playwright install-deps "${BROWSER}"; then exit 0 fi echo "::warning::playwright install-deps attempt ${attempt}/${MAX_ATTEMPTS} failed or timed out" diff --git a/.github/workflows/cache-integration-tests.yml b/.github/workflows/cache-integration-tests.yml index 260638f213..334c26cf03 100644 --- a/.github/workflows/cache-integration-tests.yml +++ b/.github/workflows/cache-integration-tests.yml @@ -20,6 +20,7 @@ on: - 'packages/data-schemas/package.json' - 'redis-config/**' - '.github/workflows/cache-integration-tests.yml' + - '.github/actions/redis-binaries/**' - '!**.md' permissions: @@ -44,30 +45,16 @@ jobs: with: node-version: '24.22.1' - - name: Install Redis tools - timeout-minutes: 10 - run: | - # Same runner apt contention that broke the MCP job in - # playwright-mock.yml: apt-daily/unattended-upgrades hold - # /var/lib/apt/lists/lock at boot. Without a step timeout this hung - # until the job-level one fired, taking the whole leg with it. - sudo systemctl stop apt-daily.service apt-daily-upgrade.service \ - unattended-upgrades.service 2>/dev/null || true - sudo systemctl kill --kill-who=all apt-daily.service \ - apt-daily-upgrade.service 2>/dev/null || true + - name: Test Redis binary provisioning + run: node --test .github/actions/redis-binaries/provision.test.mjs - apt_with_lock_wait() { - for attempt in $(seq 1 30); do - if sudo apt-get -o DPkg::Lock::Timeout=60 "$@"; then - return 0 - fi - echo "apt-get $1 could not take the lock (attempt ${attempt}/30), retrying" - sleep 10 - done - return 1 - } - apt_with_lock_wait update - apt_with_lock_wait install -y redis-server redis-tools + # The pinned redis-server and redis-cli come from the Actions cache, or + # are built from the pinned commit on a miss. apt is not involved: its + # lock contention and an unreachable Ubuntu mirror have each taken this + # job down before a single test ran. + - name: Install Redis tools + timeout-minutes: 8 + uses: ./.github/actions/redis-binaries - name: Start Single Redis Instance run: | diff --git a/.github/workflows/lighthouse.yml b/.github/workflows/lighthouse.yml index 9f8518aed9..f4f8032477 100644 --- a/.github/workflows/lighthouse.yml +++ b/.github/workflows/lighthouse.yml @@ -77,7 +77,16 @@ jobs: api/node_modules key: ${{ steps.cache-node-modules.outputs.cache-primary-key }} - name: Install lockfile-pinned Chromium - run: npx playwright install --with-deps --no-shell chromium + run: npx playwright install --no-shell chromium + # The audit keeps the lockfile-pinned Chromium build, so its download above stays fatal. + # Its system packages do not: the runner's own Chrome already brings every library + # Chromium links, and what `--with-deps` installed on top were nine optional font packages + # (22MB from the Ubuntu mirror) that no budget reads. With the mirror unreachable, apt + # failed the combined command before any audit ran. + - name: Install optional Chromium system packages (best effort) + timeout-minutes: 4 + continue-on-error: true + run: .github/scripts/install-playwright-fonts.sh chromium - name: Select Lighthouse browser run: | chrome=$(node -p "require('@playwright/test').chromium.executablePath()") diff --git a/.github/workflows/playwright-mock.yml b/.github/workflows/playwright-mock.yml index 274b4b6f9a..d071052ec6 100644 --- a/.github/workflows/playwright-mock.yml +++ b/.github/workflows/playwright-mock.yml @@ -518,22 +518,23 @@ jobs: path: ~/.cache/ms-playwright key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }} - # This job deliberately skips the optional font install: its bounded - # Playwright apt process can outlive the wrapper on a slow mirror and - # retain the package-manager lock needed by the required Redis install. - # The MCP suite does not enable visual snapshot assertions. + # This job deliberately skips the optional font install: the MCP suite + # does not enable visual snapshot assertions, so nothing here needs apt. - # Redis is a hard requirement for this job, so this step stays fatal. + # Redis is a hard requirement for this job, so this step stays fatal. The + # action puts the pinned redis-server and redis-cli on PATH from the + # Actions cache, or builds them from the pinned commit on a miss; no + # distro mirror has to resolve for this job to get Redis. - name: Install Redis runtime dependencies - timeout-minutes: 5 - run: | - sudo apt-get -o DPkg::Lock::Timeout=300 update - sudo apt-get -o DPkg::Lock::Timeout=300 install -y redis-server redis-tools + timeout-minutes: 8 + uses: ./.github/actions/redis-binaries + # A daemonized redis-server returns before it opens its port, so the + # first ping waits for it instead of racing it. - name: Start standalone Redis and Redis Cluster run: | redis-server --daemonize yes --port 6379 - redis-cli -p 6379 ping + timeout 30 bash -c 'until redis-cli -p 6379 ping; do sleep 0.5; done' chmod +x redis-config/start-cluster.sh redis-config/stop-cluster.sh ./redis-config/start-cluster.sh redis-cli -p 7001 cluster info