Parquet v3 read concurrency - #2235
Open
UnamedRus wants to merge 5 commits into
Open
Conversation
Budget memory and threads separately per read stage instead of one shared fraction, and add a ColumnDataPrefetch stage that issues the compressed data-page reads (charged to its own memory budget) while ColumnData only decodes. The old single 0.2 fraction capped the data stage at 0.2 of both memory and threads, so only ~2 row groups were read/decoded ahead and the S3 link sat idle on latency-bound, high-RTT reads. Now compressed reads run deep (cheap per row group) while decoded row groups stay bounded, hiding per-GET latency. Also reconcile the decoded-memory charge up to the actual footprint inside decodePrimitiveColumn, before formOutputColumn moves the column, so the honest cap actually bounds decode-ahead. Squashed extraction of 34816a3 + 114640e + f260506 from the parquet-v3 feature branch onto antalya-26.6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
UnamedRus
force-pushed
the
parquet-v3-read-concurrency
branch
from
August 18, 2026 13:46
ad5f3aa to
79d3658
Compare
Collaborator
|
Could you add Changelog entry please? |
mkmkme
previously approved these changes
Aug 19, 2026
mkmkme
left a comment
mkmkme
left a comment
Collaborator
There was a problem hiding this comment.
Looks good from the brief look
Collaborator
Author
|
Will do (and benchmark justification), but i would ask to start reviewing of it tomorrow, i want to do some small changes and remove some slop. :) |
ianton-ru
reviewed
Aug 19, 2026
| /// Budget memory and threads separately: a single 0.2 fraction capped ColumnData at 0.2 of both, | ||
| /// so only ~2 row groups were read/decoded ahead. Give ColumnData most of the memory and threads | ||
| /// (deep, decode-bound); give the small latency-bound index/bloom reads threads for parallelism. | ||
| using S = ReadStage; |
There was a problem hiding this comment.
Useless alias. Keeps several characters in each line, but adds cognitive complexity.
| set_fractions(S::ColumnDataPrefetch, 0.45, 1); | ||
| /// Decode: bounded memory (decoded row groups are large) but most threads. Caps resident decoded | ||
| /// row groups independently of prefetch depth. | ||
| set_fractions(S::ColumnData, 0.30, 3); |
There was a problem hiding this comment.
Each stage get part of memory/thread budget based on some magic numbers.
I trust that numbers has a ground, but good to add it somewhere.
UnamedRus
force-pushed
the
parquet-v3-read-concurrency
branch
from
August 21, 2026 12:21
ca7a4f6 to
79d3658
Compare
The reader scheduler splits a static memory and thread budget across the read stages (bloom/index reads, compressed prefetch, decode) using fixed relative weights. Those weights are the main lever for latency-bound object-storage reads - the compressed prefetch weight sets the read-ahead depth, the decode thread weight sets decode concurrency - but they were hardcoded, so tuning them required a rebuild. Add `input_format_parquet_read_stage_weights`, an experimental `Map` setting. Keys are `<stage>.<resource>` (resource is `memory` or `threads`); values are relative weights, normalized per resource across stages exactly like the built-in defaults. Only the listed keys override; unspecified stages keep their defaults, so an empty map (the default) reproduces the previous behavior bit for bit. Unknown stage or resource names, negative weights, and a configuration that zeroes out the whole memory or thread budget all raise `BAD_ARGUMENTS` rather than silently falling back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: UnamedRus <dtitmoav@gmail.com>
…atches `ColumnDataPrefetch` tasks do all their work (`determinePagesToPrefetch` + `startPrefetch`) synchronously in `scheduleTask`; their `runTask` body is empty and the reads proceed asynchronously in the Prefetcher's io pool. Their run time is therefore ~0, but `cost_estimate_bytes` was set to the charged compressed bytes, which are large. `scheduleTasksIfNeeded` uses `cost_estimate_bytes` as a proxy for run time to group tiny tasks into batches. Reporting the large compressed-byte charge made the batcher split these zero-work tasks across up to `parsing_threads` batches - one thread-pool dispatch each - for tasks that do nothing on the thread. Report a cost of 0 for `ColumnDataPrefetch` tasks so they collapse into a single batch, removing the spurious per-subgroup dispatch overhead introduced by the prefetch/decode stage split. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
UnamedRus
force-pushed
the
parquet-v3-read-concurrency
branch
from
August 24, 2026 09:34
ccd3c2d to
df7105e
Compare
Condense the oversized comment blocks added by the read-concurrency work (the stage-weight rationale in `ReadManager::init`, the ColumnDataPrefetch/ColumnData scheduleTask and runTask notes, and the cost-estimate exception) to a few lines each. No code changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Number of changes to bring parquet v3 reader perf closer to arrow based
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):