fix(market-adapter): keep Kibana LP fetches alive under proxy connect… - #12
Open
Impulssi wants to merge 1 commit into
Open
fix(market-adapter): keep Kibana LP fetches alive under proxy connect…#12Impulssi wants to merge 1 commit into
Impulssi wants to merge 1 commit into
Conversation
…ion resets The Kibana console proxy at kibana.bitshares.dev now resets connections mid-transfer once a single search page streams enough data (observed with full _source payloads around ~8k documents, and even ~2k documents with _source: true). The LP fetcher queried with size 10000 and _source: true, so every fetch died at the availability probe — and worse, silently: - doKibanaRequest only listened for request-phase errors; a response stream aborted mid-body never emits 'end', so the surrounding promise stayed pending forever and fetch_lp_data hung with no error output. - There was no retry, so a single transient reset killed the whole fetch. ## Fix - kibana_client: handle 'aborted'/'error' on the response stream and reject the promise, so callers fail loudly instead of hanging forever. - kibana_candles: request only the _source branches the field map actually reads (plus the fixed timestamp/sequence fields) instead of full documents. Restricted projections keep each page small and fast (verified live: 2000-doc pages with restricted _source return in ~1.5s where full _source aborts). - kibana_candles: default kibanaPageSize 10000 -> 2000. - kibana_candles: retry transient page failures (aborted / connection reset / timeout) with linear backoff — search_after pagination is stateless on the server, so replaying a failed page is safe. - buildDirectionalDocumentQuery accepts an optional sourceFields argument; omitting it keeps the legacy _source: true behavior for other callers. ## Testing Notes - tests/test_kibana_candles.ts: new coverage for the source projection derivation, the _source wiring in the directional query (plus legacy fallback), and transient-page retry recovery. - npm run typecheck clean. - Verified live against kibana.bitshares.dev with fetch_lp_data.js (BTS/XBTSX.USDT pool 1.19.48, 2-month 1h fetch): the availability probe and both chunk fetches complete where they previously hung indefinitely.
Impulssi
force-pushed
the
fix/kibana-lp-fetch-stability
branch
from
August 26, 2026 06:03
f8aca53 to
f4d779c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ion resets
The Kibana console proxy at kibana.bitshares.dev now resets connections mid-transfer once a single search page streams enough data (observed with full _source payloads around ~8k documents, and even ~2k documents with _source: true). The LP fetcher queried with size 10000 and _source: true, so every fetch died at the availability probe — and worse, silently:
Fix
Testing Notes
Note: I'm new to this codebase and worked through this change with an AI
coding assistant (opencode). The bug was hit on my own live instance and
all measurements / reproduction steps are mine — but please flag anything
in the code that looks off and I'll dig in and follow up.