fix(core): Keep resolving the hostname after Sentry.close() - #6119
Open
runningcode wants to merge 4 commits into
Open
runningcode wants to merge 4 commits into
runningcode wants to merge 4 commits into
Conversation
MainEventProcessor was Closeable, so Scopes.close() closed it, and it shut down the process-wide HostnameCache singleton. Nothing ever replaced that singleton: INSTANCE is assigned once and never cleared, so a re-init handed the same shut-down cache to the new MainEventProcessor, and to MetricsApi and LoggerApi, which read it directly. The damage was silent and permanent. While the cache was still fresh, getHostname() kept returning the value it already had. On the first expiry after the close, getHostname() flipped updateRunning to true and then submit() threw RejectedExecutionException on the terminated executor. That is a RuntimeException, so it was swallowed into handleCacheUpdateFailure(), but the updateRunning reset lives in the submitted callable's finally block, which never ran. updateRunning stayed true, so the compareAndSet guard failed from then on and no refresh was ever attempted again. server_name froze at its last resolved value for the life of the process, with no exception and no log line. Nothing needs to close this cache. Its executor is a single daemon thread with allowCoreThreadTimeOut(true) and a 30 second keep-alive, so the worker exits on its own once idle and never holds up process exit; the thread exists for about 30 seconds out of every 5 hour refresh interval. Scopes.close() already leaves the timer executor running for exactly this reason. The one test that covered this path, SentryClientTest's `when client is closed, hostname cache is closed`, asserted isClosed() on a processor that had never resolved a hostname, where isClosed() returned true because the cache was still null. It never exercised the behavior it named. Replaced with an assertion that MainEventProcessor is not Closeable, which fails if the wiring comes back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📲 Install BuildsAndroid
|
updateRunning is cleared in exactly one place, the submitted callable's finally block, so it is cleared if and only if the callable runs. Every failure from Future.get() leaves the callable running, so it still clears the flag itself. A failure from submit() does not: the callable was never queued, nothing clears the flag, and the compareAndSet guard in getHostname() then fails forever, so no refresh is ever attempted again. Removing MainEventProcessor's close() took away the only reachable way to make submit() throw, but the invariant was still wrong: a bounded queue, a shutdown added later, or a failure to start a thread would silently resurrect the same permanent freeze. Splitting submit() out of the try means the two cases can be told apart. Clearing the flag on a timeout or an interrupt as well would be wrong, since the callable is still running there and refreshes would pile up behind a slow lookup; MainEventProcessorTest's `sets servername to null if retrieving takes longer time` covers that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It asserted a type relationship rather than behavior, which says nothing about whether the hostname keeps resolving. The behavior that matters is covered by HostnameCacheTest: `worker thread times out while idle instead of staying alive` guards the self-terminating executor that makes closing unnecessary, and `a refresh that cannot be queued does not stop later refreshes` guards the latch that turned a one-off failure into a permanent one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
runningcode
marked this pull request as ready for review
September 15, 2026 16:10
runningcode
requested review from
0xadam-brown,
adinauer,
markushi and
romtsn
as code owners
September 15, 2026 16:10
runningcode
added
the
sanity-check
runningcode
commented
Sep 15, 2026
| // itself; doing it here as well would let refreshes pile up behind a slow lookup. | ||
| try { | ||
| final Future<Void> futureTask = executorService.submit(hostRetriever); | ||
| futureTask.get(GET_HOSTNAME_TIMEOUT, TimeUnit.MILLISECONDS); |
Contributor
Author
There was a problem hiding this comment.
this blocks for 1 second which IMO defeats the whole purpose of using an executor. im leaving this out of scope of this PR though. curious if others have thoughts on why this is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📜 Description
So going deep in the weeds on the
HostnameCachefor the Clocks project made me realize we had a bug here. Yes we also have a bug in that we're using a wallclock to determine the hostnamecache timeout but that's a different PR.The main issue here is that once you call close on the singleton
HostnameCache, we can never look up another hostname again since the executor is shutdown. Since it is a singleton there's no way to restart it or create a new instance making it a permanent failure.Here's a description of how this goes wrong:
getHostname()flipsupdateRunningtotrueviacompareAndSetand callsupdateCache().executorService.submit(...)throwsRejectedExecutionExceptionon the terminated executor.RuntimeException, so it is swallowed intohandleCacheUpdateFailure().updateRunning.set(false)reset lives in the submitted callable'sfinallyblock — and the callable never ran.You see there's a second bug in that
updateRunningis never set to false on aRejectedExecutionExceptionand then we never attempt another refresh.The fix is that nothing needs to close this cache. Its executor is a single daemon thread with
allowCoreThreadTimeOut(true)and a 30-second keep-alive, so the worker exits on its own once idle and never holds up process exit, the thread exists for roughly 30 seconds out of every 5-hour refresh interval.Scopes.close()already leaves the timer executor running for exactly this reason:💡 Motivation and Context
Any
Sentry.close()or re-init permanently stopped hostname refreshes for the whole process. Both paths go throughScopes.close(), which closes everyCloseableevent processor.💚 How did you test it?
There are new unit tests here.
📝 Checklist
sendDefaultPIIis enabled.MainEventProcessoris@ApiStatus.Internal, so droppingCloseableandclose()from it changessentry.apiwithout changing the public contract. The practical effect is thatScopes.close()'s "close anyCloseableevent processor" loop now skips it, which is the fix.🔮 Next steps
None.
🤖 Generated with Claude Code