Skip to content

fix(pd): keep KvClient watches alive after reconnect failures - #3157

Open
contrueCT wants to merge 1 commit into
apache:masterfrom
contrueCT:fix/3152-pd-kv-watch-reconnect
Open

fix(pd): keep KvClient watches alive after reconnect failures#3157
contrueCT wants to merge 1 commit into
apache:masterfrom
contrueCT:fix/3152-pd-kv-watch-reconnect

Conversation

@contrueCT

Copy link
Copy Markdown
Contributor

Purpose of the PR

KvClient previously retried a failed watch only once. If that reconnect also failed, the
watch stopped permanently, and unexpected stream completion did not trigger recovery.

The reconnect path also exposed a lifecycle issue: AbstractClient.resetStub() invoked the
virtual close() method, so transport initialization on a KvClient dispatched to
KvClient.close() and marked the whole client closed.

Main Changes

  • Separate AbstractClient transport cleanup from the overridable client lifecycle close.
  • Track each exact-key or prefix watch as an independent subscription with its current observer.
  • Retry onError, Leader_Changed, and unexpected onCompleted with a fixed delay until the
    watch recovers or the client closes.
  • Deduplicate reconnect scheduling for the same observer and ignore stale observer callbacks.
  • Stop the reconnect executor and invalidate active observers when KvClient.close() is called.
  • Add deterministic regression coverage for repeated failures followed by recovery, leader
    changes, completion, deduplication, stale observers, prefix semantics, and close behavior.

This PR does not add event replay, protocol revisions, PD Server changes, or the Server-side
reconciliation tracked by #3151. Events emitted while a watch is disconnected are still not
replayed.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • mvn -q -o test -pl hugegraph-pd/hg-pd-test -am -Dtest=KvClientTest -DfailIfNoTests=false -Drat.skip=true -Djacoco.skip=true (12 tests, 0 failures/errors)
    • mvn -q -o package -pl hugegraph-pd -am -DskipTests -Dmaven.javadoc.skip=true -Drat.skip=true -Djacoco.skip=true
    • mvn -q -o -f hugegraph-pd/hg-pd-client/pom.xml apache-rat:check
    • mvn -q -o -f hugegraph-pd/hg-pd-test/pom.xml apache-rat:check

The full local PD suite was also attempted. The common suite passed 83/83 and the core suite
passed 89/91 with 2 skipped. The client suite could not complete without a local PD service at
127.0.0.1:8686 (4 failures and 5 errors reported connection refused), which prevented the rest
suite from executing in that Maven run.

Repository-wide RAT is not a valid signal in this checkout because pre-existing ignored
.upgrade-artifacts files produce 68 unrelated unapproved-license entries; RAT passes for both
changed modules.

Does this PR potentially affect the following parts?

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. bug Something isn't working pd PD module tests Add or improve test cases labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working pd PD module size:XL This PR changes 500-999 lines, ignoring generated files. tests Add or improve test cases

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

[Bug] PD KvClient watch can permanently stop after reconnect failure

1 participant