feat: add health check retry logic and alert cooldown to reduce false…

[inderjeet20]

🚀 Summary

Reduce false-positive backend health alerts by introducing retry logic and cooldown.

🐛 Problem

Health alerts are currently triggered even when the backend is healthy. This appears to be caused by transient failures in the health check.

Additionally, alerting logic is not version-controlled and likely depends on an external VPS script, making it harder to maintain and debug.

✅ Solution

• Added a versioned monitoring script: deployment/health-check.sh
• Implemented retry threshold (e.g., alert only after 3 consecutive failures)
• Added cooldown mechanism (e.g., suppress alerts for a fixed duration after triggering)
• Introduced state persistence to track failures and alert timing

🎯 Benefits

• Reduces false-positive alerts
• Prevents alert spam (observed ~35 min intervals)
• Makes monitoring logic version-controlled and maintainable
• Improves reliability of alerting system

📝 Notes

• This is a proposed improvement and may need alignment with the current monitoring setup
• Open to feedback on approach or integration

🧪 Testing

• Logic reviewed for retry and cooldown behavior
• Health endpoint verified to return healthy during normal operation
• Designed to handle intermittent failures without triggering unnecessary alerts

[github-actions]

Thank you for opening this PR!

Before a maintainer takes a look, it would be really helpful if you could walk through your changes using GitHub's review tools.

Please take a moment to:

• Check the "Files changed" tab
• Leave comments on any lines for functions, comments, etc. that are important, non-obvious, or may need attention
• Clarify decisions you made or areas you might be unsure about and/or any future updates being considered.
• Finally, submit all the comments!

More information on how to conduct a self review:
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/reviewing-proposed-changes-in-a-pull-request

This helps make the review process smoother and gives us a clearer understanding of your thought process.

Once you've added your self-review, we'll continue from our side. Thank you!

[its-me-abhishek]

I found 2 issues worth fixing before merging (AI-Generated, please check the feasibility of these, might be incorrect):

1. Blocking: deployment/health-check.sh treats any HTTP 200 as healthy.

• In check_backend_endpoint(), the code does:
  • curl --silent --show-error --fail --max-time "$CURL_TIMEOUT_SECONDS" "$HEALTH_URL" >/dev/null
• That only checks the status code, not the actual response body.
• A stale page, maintenance response, or a false-positive 200 from an upstream proxy would still be considered healthy, which directly undermines the goal of reducing false-positive alerts.
• Recommendation: validate the returned payload as well, e.g. check for a known healthy marker (healthy, ok, or JSON status field) instead of only checking for HTTP success.

2. Medium: cooldown state is not reset on recovery.

• In main(), when the checks pass, it resets consecutive_failures but does not clear last_alert_at.
• That means a later incident can still be suppressed immediately after a recovered run if the previous incident was recent, even though the issue has already cleared.
• Recommendation: clear last_alert_at when the system becomes healthy again, or otherwise scope the cooldown to the active failure window rather than permanently retaining the old timestamp.

These are both in deployment/health-check.sh; the first one is the more important correctness issue.