From 919a46312b1732865fc11f81cd0a1a06366d749f Mon Sep 17 00:00:00 2001 From: Danny Avila Date: Wed, 10 Jun 2026 22:44:54 -0400 Subject: [PATCH] =?UTF-8?q?=F0=9F=A7=B9=20ci:=20Relieve=20disk=20pressure?= =?UTF-8?q?=20on=20GitNexus=20deploy=20(#13666)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gitnexus droplet is ~8.7GB usable, not the 60GB the disk-cleanup comment assumed. With /usr (~2.8GB), the in-use docker images (~2.1GB), and the growing /opt/gitnexus/indexes (~1.2GB), deploys were aborting at the `AVAIL_MB < 2048` guard ("Disk critically low"). Two fixes: - Reclaim the previous gitnexus image after force-recreate. The pre-pull `docker system prune -af` cannot remove it while the old container is still running, so a stale ~700MB generation accumulated every deploy. A post-recreate `docker image prune -f` makes the box self-cleaning. - Lower the abort threshold 2048 -> 1536MB. The image is ~700MB and shares most layers with the running one, so an incremental pull needs well under 1GB; the old guard was sized for the 60GB assumption. Also corrects the stale 60GB comment to reflect the actual disk. --- .github/workflows/gitnexus-deploy.yml | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/.github/workflows/gitnexus-deploy.yml b/.github/workflows/gitnexus-deploy.yml index 492918339c..d5f33e676e 100644 --- a/.github/workflows/gitnexus-deploy.yml +++ b/.github/workflows/gitnexus-deploy.yml @@ -460,10 +460,11 @@ jobs: # ── Disk cleanup ────────────────────────────────────── # Docker accumulates old image layers, dangling images, and - # build cache across deploys. On a 60GB droplet with a 700MB+ - # gitnexus image, this fills the disk after ~40 deploys. - # Prune everything not used by currently-running containers - # BEFORE pulling the new image so the extract has room. + # build cache across deploys. This droplet is only ~8.7GB + # usable with a 700MB+ gitnexus image, so disk pressure is + # constant. Prune everything not used by currently-running + # containers BEFORE pulling the new image so the extract has + # room; the post-recreate prune below reclaims the old image. echo "Disk before cleanup:" df -h / | tail -1 # Omit --volumes: Caddy's caddy-data and caddy-config volumes @@ -475,9 +476,13 @@ jobs: echo "Disk after cleanup:" df -h / | tail -1 - # Fail fast if disk is critically low even after prune + # Fail fast if disk is critically low even after prune. The + # gitnexus image is ~700MB and shares most layers with the + # running one, so an incremental pull needs well under 1GB. + # 1536MB leaves headroom on this small droplet without the + # over-conservative 2GB guard aborting on a healthy box. AVAIL_MB=$(df --output=avail -m / | tail -1 | tr -d ' ') - if [ "$AVAIL_MB" -lt 2048 ]; then + if [ "$AVAIL_MB" -lt 1536 ]; then echo "::error::Disk critically low (${AVAIL_MB}MB free). Aborting deploy." exit 1 fi @@ -485,6 +490,13 @@ jobs: docker compose pull gitnexus docker compose up -d --force-recreate gitnexus + # The previous gitnexus image is now dangling (the running + # container was recreated onto the freshly pulled image). The + # pre-pull prune above couldn't touch it because it was still + # in use at that point. Reclaim it now so the old generation + # doesn't accumulate — critical on this 10GB droplet. + docker image prune -f 2>/dev/null || true + # Reload Caddy in-place so a changed Caddyfile takes effect # without losing TLS certs or restarting connections. If caddy # isn't running yet (first-time bootstrap), bring it up.