fix(cleaner-agent): force-prune unused volumes when disk/inode threshold is hit - #164
Draft
openswe-shared-services[bot] wants to merge 1 commit into
Draft
openswe-shared-services[bot] wants to merge 1 commit into
openswe-shared-services[bot] wants to merge 1 commit into
Conversation
…old is hit The mid-lifetime watchdog (cleaner-agent.sh) trips at 90% disk/inode usage but its clean_volumes()/clean_images() steps skip anything touched within VOLUMES_RETAIN_PERIOD/IMAGE_RETAIN_PERIOD, and its own last-resort step only prunes images. A dind pod reused across several builds can stay pinned above the inode threshold for the rest of its life, so the next build to land on it starts already near capacity (CCS-45). Add an unconditional 'docker volume prune' as a final emergency step - it only removes volumes not attached to a running container, so it never touches an in-progress build. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
|
/e2e |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a final emergency step to
cleaner-agent.shthat runsdocker volume prune --forcewhen disk/inode usage is still aboveSTART_DISK_USAGE_THRESHOLD/START_INODES_USAGE_THRESHOLD(0.9) after the existing container/volume/image cleanup and image-purge steps.Why
Triage of CCS-45 (
justpark-inc, SaaS Classic,system/linux_paying_plantier) found that hosted builds intermittently fail with "out of disk space" while starting at 95%+ inode utilization, even though byte usage is well under the volume size.Root cause hypothesis: a dind pod is reused for multiple builds (
CLEAN_PERIOD_BUILDS/CLEAN_PERIOD_SECONDS). During that shared lifetime, the only watchdog iscleaner-agent.sh, which trips at 90% disk/inode usage but:clean_volumes()/clean_images()(cleaner/functions.sh) skip anything touched withinVOLUMES_RETAIN_PERIOD/IMAGE_RETAIN_PERIOD(deliberately long on this tier to help build-cache hit rate),docker image prune ... --filter until=3h), never volumes.The only unconditional full reclaim (
docker system prune -a --volumes --force) lives indocker-clean.sh, but that only runs when the pod itself is torn down (run.sh, gated byCLEAN_DOCKER). So a pod that gets a burst of build activity can sit pinned above the inode threshold for the rest of its life, and whichever build lands on it next starts already near capacity.This adds a volume-aware step to the mid-lifetime emergency path:
docker volume pruneonly removes volumes not attached to any container, so it's safe to run unconditionally (never touches an in-progress build) and gives the watchdog a way to actually reclaim inode-heavy dangling volumes without waiting for pod teardown.Also adds a matching
docker_volume_cleaner_agent_actions_volume_purgesPrometheus gauge inmonitor/metrics.sh, following the existing metric pattern, so this new action is observable.Notes
clean_volumes()'s existing retain-period behavior — it only adds a stronger last-resort step, same tier as the existing image purge.service.yamlversion per repo convention.Made by Open SWE · anthropic:claude-sonnet-5 (medium)
Security Report — codefresh/dind
Note
Compared security scans:
Current image:
quay.io/codefresh/dev/dind:open-sweccs-45-inode-exhaustion-volume-purge@sha256:fd94a5cd3030b6b675a945fd7c70f7c95c1b09db29f4e8ea5aa2ac0588a78b91
Baseline:
quay.io/codefresh/dind:master@sha256:b2230c0889370fa297b7e9643ee43e730897f8befdb6d23b007b4b011764c62f
Fixed CVEs: 0
Fixed issues: 0