Skip to content

fix(backtest): bound every log tail to the final 256 KB - #19

Closed
Marinski wants to merge 1 commit into
psyb0t:masterfrom
Marinski:fix/tail-bounded-reads
Closed

Marinski wants to merge 1 commit into
psyb0t:masterfrom
Marinski:fix/tail-bounded-reads

Conversation

@Marinski

Copy link
Copy Markdown
Contributor

What

The live /tail endpoint read entire log files into memory on every poll: _tail_dir_log and get_tail's run.log block called a whole-file reader, then decoded and split the result.

A Strategy Tester log grows to gigabytes while the run being polled writes it, so each poll allocated the file + a decoded copy + a line list — measured 45–65 s per call against a 1.9 GB log, with the GIL held through the decode. Every thread in the process stalled behind it: /ping took 25–50 s, the container healthcheck flapped, clients timed out, and a healthy terminal was indistinguishable from a wedged one. _tail_terminal_log (the failure-path tail) had the same unbounded read.

Fix

  • _read_tail_text: seek to the final 256 KB, 2-byte aligned so UTF-16 code units stay intact; sniff UTF-16 from a BOM or NUL second byte; drop the truncated first line of a mid-file window. Used by both tail helpers and the run.log block.
  • _tail_dir_log inherits the two picker rules _tail_terminal_log already had and it predated: newest by mtime, never metaeditor.log (the alphabetical pick returned a stale compile log — 'm' > '2' — instead of the run being polled).
  • Whole-file reads remain only where the whole file is genuinely needed (INI parse, report parse).

Tests

12 new tests: bounded-window correctness (truncated first line dropped, UTF-16 alignment via a non-ASCII log, small-file passthrough, missing file), bounded _tail_terminal_log, and _tail_dir_log's mtime pick / metaeditor exclusion / bounded read / empty dir. Full suite: 432 passed.

The live /tail endpoint read entire log files into memory on every poll:
_tail_dir_log and get_tail's run.log block called a whole-file reader,
then decoded and split the result. A Strategy Tester log grows to
gigabytes WHILE the run that is being polled writes it, so each poll
allocated the file plus a decoded copy plus a line list — 45-65 s per
call measured against a 1.9 GB log, with the GIL held through the decode.
Every thread in the process stalled behind it: /ping took 25-50 s, the
container healthcheck flapped, clients timed out, and a healthy terminal
was indistinguishable from a wedged one. (_tail_terminal_log, the
failure-path tail, had the same unbounded read.)

_read_tail_text seeks to the final 256 KB, aligned to a 2-byte boundary
so UTF-16 code units stay intact, sniffs UTF-16 from a BOM or a NUL
second byte, and drops the truncated first line of a mid-file window.
Both tail helpers and the run.log block now use it; whole-file reads
remain only where the whole file is genuinely needed (INI parse, report
parse).

_tail_dir_log also inherits the two picker rules _tail_terminal_log
already had and it predated: newest by mtime, never metaeditor.log —
the alphabetical pick returned a stale compile log ('m' > '2') instead
of the run being polled.
@Marinski

Copy link
Copy Markdown
Contributor Author

Folding this into #18 rather than asking you to review it separately, and closing it here.

Both change mt5api/backtest/handler.py, both are terminal-log and symbol-cache robustness, and #18 already has a review round from you on it. Two PRs over one file was one more than you needed.

The commit is now on fix/symbol-suffix-and-operator-scripts as b24399d, unchanged, and the merge was clean. #18's description covers it. The reasoning above stays here for reference.

Nothing here has been reviewed yet, so nothing is being taken away from you — if you would rather it stayed separate, say so and I will reopen it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant