feat(DM01-6260): optimize console log loading with gzip compression and progressive rendering - #643
Merged
Conversation
…t 500 lines Backend: Console.get() now accepts ?tail=N. For archived logs (job.console) the last N lines are sliced in Python; for streaming logs (console table) a subquery ORDER BY date DESC LIMIT N avoids a full table scan. Backward compatible — omitting tail returns the full log unchanged. Frontend: loadConsole() defaults to tail=500 so the initial page load fetches only the last 500 lines. consoleTruncated flag drives a notice bar in Console.vue with a "Load full log" link that calls loadFullConsole(), which resets sections and re-fetches without the limit.
…ilent prefetch Backend: - Archived tail requests now use PostgreSQL array slicing so large logs (e.g. 250 MB) are trimmed on the DB side instead of loading the full text into Python memory before slicing. - Add _console_response() helper that gzip-compresses responses larger than 100 KB when the client sends Accept-Encoding: gzip, reducing a 250 MB transfer to ~15 MB (~15x faster). Frontend: - loadConsole() fetches ?tail=500 first so the page renders immediately without waiting for the full log download. - _prefetchFullConsole() silently fetches the full log in the background and replaces the sections once complete — no user interaction required. - Removed consoleTruncated flag, loadFullConsole(), and the truncation notice from Console.vue; UI is identical to before.
- Filter GC 'deleted' marker: add AND console != 'deleted' to both tail
and non-tail archived log queries so old jobs no longer return the
literal string 'deleted' as console output
- Add project_id scope to console table fallback queries via subquery
to prevent cross-project console data leakage
- Fix chunk-vs-line mismatch: fetch max(tail, 200) chunks then trim to
exact tail lines in Python after concatenation
- Fix gzip threshold: use len(output.encode('utf-8')) instead of
len(output) so multi-byte Unicode content is measured correctly
- Use _consoleFetched flag instead of sections.length as the loadConsole
re-entry guard so the guard holds even after _prefetchFullConsole
resets sections to []
- Only run _prefetchFullConsole for finished jobs to prevent WebSocket
line duplication on running jobs
- Reset _consoleFetched = false on loadConsole network error so the
user can retry
liuwei08
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Console output pages for large jobs (e.g. 250 MB) were taking 3+ minutes to load. This PR reduces initial load time to under 1 second with no UX change, and fixes several security and correctness issues found during review.
Performance changes
?tail=Nparameter onGET .../consoleORDER BY date DESC LIMIT N— 250 MB logs are never fully loaded into Python memory?tail=500first, then silently prefetches full log in backgroundBug fixes (from code review)
project_idto prevent cross-project console data leakageconsole = 'deleted'(set by GC after 30 days) so old jobs no longer display the literal string"deleted"as outputmax(tail, 200)chunks then trim to exacttaillines in Python, fixing the mismatch between DB row count and actual log line countlen(output.encode('utf-8'))instead oflen(output)so multi-byte Unicode content is measured correctlysections.lengthcheck with a dedicated_consoleFetchedflag so the guard holds even after_prefetchFullConsoleresetssections = []_prefetchFullConsoleonly runs for finished jobs, preventing live WebSocket lines from being duplicated when the full log replaces the tail viewCI change
service-aksandservice-gardenerbuild/deploy jobs frominfrabox/generator/deployments.jsonTest plan
GET .../console?tail=500returns exactly the last 500 linesGET .../console(no param) returns the full log unchanged"deleted"as console outputGET /api/v1/projects/<A>/jobs/<B_job_id>/consolereturns empty when job does not belong to project A?tail=abcfalls back to full log with no 500 errorJira: DM01-6260