Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25372 +/- ##
==========================================
- Coverage 81.94% 81.94% -0.01%
==========================================
Files 1135 1135
Lines 428278 428403 +125
Branches 428278 428403 +125
==========================================
+ Hits 350947 351049 +102
- Misses 56375 56383 +8
- Partials 20956 20971 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
I think this should be explained in more detail. Other than that, the cursors Vec should keep the memory reservations alive for each input partition, and when the stream is polled, the previous reservation would be dropped and a new reservation created. |
Currently the behaviour is to only keep the current batch reserved per cursor, meaning we don't actually reserve all the rows we are keeping owned.
This leads to a lot of invisible memory, that is held, but not accounted by the pool.
(This scales fairly linearly with the number of partitions, in a regular round-robin case, for example.)
While the mem overhead was greatly reduced by @ariel-miculas 's #23619 and #23802
It is still a very problematic area in terms of memory visibility.
(Round-robin is an extreme case by nature)
This PR does not reduce the mem usage or anything, it just makes the memory visible and keeps everything that needs to be reserved - reserved.
I believe in order to ensure robustness, once this is merged another PR will need to be made to cap the merge reservation to the peak of the ExternalSorter reservation (due to some behaviours I've seen with fair-type pools, which treat spillable and non-spillable operators differently)