Conversation
Pass `offset` in addition to limit/fetch
github-actions
Bot
added
documentation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25404 +/- ##
==========================================
+ Coverage 82.28% 82.30% +0.01%
==========================================
Files 1137 1137
Lines 430211 432078 +1867
Branches 430211 432078 +1867
==========================================
+ Hits 354003 355617 +1614
- Misses 54784 54953 +169
- Partials 21424 21508 +84 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Box TableScan's statistics_requests field because otherwise it becomes bigger than 176 bytes and `test_size_of_logical_plan()` fails
github-actions
Bot
added
substrait
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Currently
TableProviderprovides thelimitargument to thescan()method to request a maximum number of rows. There is no way to tell the implementation to skip some of the rows, for example to fetch the second/third/Nth page of rows (i.e. SQL... LIMIT 20 OFFSET 40).Adding an additional field to ScanArgs (named
offsetorskip) will make it possible for implementations to override thescan_with_args()method and optimize their scan to read and return only the requested rows.What changes are included in this PR?
offsetis added toScanArgs, with a setter and a getter.TableProvidertrait -supports_offset_pushdown() -> bool. By default it returnsfalsebut any implementation that can support skipping of rows could override it to returntrueand combined with a custom implementation ofscan_with_args()to optimise its data scan/read.TableProvider::scan()to use::scan_with_args()where they could support offset push downNote:
datafusion-ffiis not updated because it does not exposescan_with_args()yet.What is the testing strategy for this PR?
New unit tests are added for the implementations which support offset pushdown.
Are there any user-facing changes?
The new functionality is opt-in! All currently existing custom implementations of
TableProvidertrait will continue to compile and run without any modifications.Any custom implementation that wants to make use of the new functionality will need to override
TableProvider::supports_offset_pushdown()to returntrueand make use ofScanArgs::offsetin itsscan_with_args()implementation.