Skip to content

fix: resolve PIDPressure evictions and crash-safety issues - #86

Closed
Zherphy wants to merge 1 commit into
opensourceways:masterfrom
Zherphy:fix/pid-pressure-and-crash-safety
Closed

fix: resolve PIDPressure evictions and crash-safety issues#86
Zherphy wants to merge 1 commit into
opensourceways:masterfrom
Zherphy:fix/pid-pressure-and-crash-safety

Conversation

@Zherphy

@Zherphy Zherphy commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the root cause of repeated pod restarts and evictions in the openeuler-bigfiles production namespace. All 17 failed pods were killed due to node PID exhaustion (PIDPressure) — the Go application spawns git child processes (via GetLFSMappingpython3 lfsNameQuery.py) but never reaps them, causing zombie [git] <defunct> processes to accumulate until the node PID table is exhausted.

Current state observed in production: 87,747 zombie processes in a single running pod.

Changes

Critical — Zombie Process Reaping (Root Cause Fix)

  • main.go: Add SIGCHLD signal handler with syscall.Wait4(WNOHANG) to reap zombie child processes
  • Dockerfile: Add tini as PID 1 (defense-in-depth — tini also reaps zombies even if the Go handler misses some)

Critical — Crash Safety

  • server/server.go: Replace all panic()/must() calls in HTTP handlers with proper error responses. Previously, a single OBS API hiccup or URL parse error would crash the entire process, killing all in-flight requests.
  • server/server.go: generateDownloadUrl now returns (*url.URL, error) instead of panicking

High — Startup Reliability

  • main.go: initConfig() and initObsClient() errors are now fatal — the server refuses to start with broken DB/auth/OBS dependencies
  • main.go: Fix dead-code err check after server.New() — the if err != nil block was checking a shadowed variable

Medium — Operational Improvements

  • main.go: Add SIGTERM graceful shutdown via http.Server.Shutdown()
  • server/server.go: Add depth limit (10) to recursive checkRepoOidName() to prevent stack overflow
  • server/server.go: Enhance health check to verify DB (ping) and OBS (list buckets) connectivity
  • db/db.go: Move AutoMigrate from every InsertLFSObj call to one-time RunMigration() at startup
  • db/db.go: Replace log.Fatal with proper error return in Init()

Test Updates

  • server/server_test.go: Update tests to match new function signatures

Verification

  • go build ./main.go — compiles successfully
  • go test ./server -skip 'TestAddGithubMetaData_AfterFuncRecover|TestAddMetaData_AfterFuncRecover' — all tests pass

Production Impact

After deploying this fix:

  1. Zombie git processes will be reaped automatically (SIGCHLD handler + tini)
  2. OBS API transient errors will no longer crash the entire server
  3. Failed initialization will prevent a broken server from starting
  4. Health check will correctly report DB/OBS connectivity issues
  5. Graceful shutdown prevents request drops during pod termination

Root cause: zombie [git] child processes accumulate because PID 1 (Go app)
never calls wait() to reap them, exhausting the node PID table and
triggering Kubernetes PIDPressure evictions.

Key changes:
- main.go: add SIGCHLD signal handler to reap zombie git processes
- main.go: add SIGTERM graceful shutdown via http.Server.Shutdown()
- main.go: make initConfig/initObsClient errors fatal (was silently continuing)
- main.go: fix dead-code err check after server.New() (err was shadowed)
- Dockerfile: add tini as PID 1 for defense-in-depth zombie reaping
- server/server.go: replace panic()/must() in HTTP handlers with proper
  error responses (was crashing entire process on OBS API hiccup)
- server/server.go: generateDownloadUrl now returns (*url.URL, error)
  instead of panicking on failure
- server/server.go: enhance health check to verify DB and OBS connectivity
- server/server.go: add depth limit (10) to checkRepoOidName recursion
- db/db.go: move AutoMigrate from every InsertLFSObj call to startup
  RunMigration() (called once in main.go)
- db/db.go: replace log.Fatal with proper error return in Init()
@opensourceways-bot

Copy link
Copy Markdown

Welcome To opensourceways Community

Hey @Zherphy , thanks for your contribution to the community.

Bot Usage Manual

I'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands.

Contact Guide

If you have any questions, please contact the SIG: infratructure ,
and any of the maintainers: @GeorgeCao-hw, @TangJia025, @pkking, @zhongjun2 ,
and any of the committers: @Goalina, @Zherphy, @tfhddd .

@opensourceways-bot

Copy link
Copy Markdown

CLA Signature Pass

Zherphy, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@opensourceways-bot

Copy link
Copy Markdown

Linking Issue Notice

@Zherphy , the pull request must be linked to at least one issue.
If an issue has already been linked, but the needs-issue label remains, you can remove the label by commenting /check-issue .

@Zherphy Zherphy closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants