Add disableRowMaterialization option to skip per-cell decode - #465
Open
mani-mathur-arch wants to merge 2 commits into
Open
Add disableRowMaterialization option to skip per-cell decode#465mani-mathur-arch wants to merge 2 commits into
mani-mathur-arch wants to merge 2 commits into
Conversation
New opt-in connection option (default off). When set, the Arrow converter still fetches, decompresses and parses each batch but returns `null` row placeholders instead of running the per-cell type conversion, so only the row count is meaningful. Wired through both the Thrift and kernel backends via ClientConfig, mirroring preserveBigNumericPrecision. Lets throughput benchmarks measure fetch cost without the per-row materialization overhead. Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 16:06 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 16:06 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 16:06 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 16:06 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:24 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:24 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:24 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:24 — with
GitHub Actions
Inactive
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
…zation Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
mani-mathur-arch
force-pushed
the
disable-row-materialization
branch
from
July 27, 2026 19:30
49c74a9 to
04fb09b
Compare
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:30 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:30 — with
GitHub Actions
Inactive
mani-mathur-arch
temporarily deployed
to
azure-prod
July 27, 2026 19:30 — with
GitHub Actions
Inactive
msrathore-db
left a comment
Contributor
There was a problem hiding this comment.
Add support for kernel mode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in connection option,
disableRowMaterialization(defaultfalse), that lets a caller drain a result set for its row count and fetch/parse cost without paying the per-cell row-materialization overhead.When enabled,
ArrowResultConverterstill fetches, decompresses and IPC-parses every Arrow batch, but skipstable.toArray()+getRows()(the recursiveconvertArrowTypes→convertThriftTypespass) and returnsnullrow placeholders instead. Only the row count is meaningful on this path — the returned objects carry no cell values.Motivation
The Node driver materializes every cell into a JS object on every fetch path, which dominates wall-clock on large result sets and makes throughput benchmarks measure decode cost rather than fetch cost. This flag gives benchmarks (and any count-only consumer) a way to isolate fetch/parse from per-row decode. It does not change any default behavior.
Changes
disableRowMaterialization?: booleanadded toConnectionOptions(IDBSQLClient.ts) andClientConfig(IClientContext.ts), defaultedfalseinDBSQLClient.getDefaultConfigand copied fromconnect()options — same plumbing aspreserveBigNumericPrecision.ArrowResultConverter.fetchNextreturnsnew Array(rowCount).fill(null)when the flag is set (filled, not sparse, sofetchAll's.flat()preserves the count).ARROW_BASED_SET+URL_BASED_SET/CloudFetch) and kernel.What it does not do
The flag skips per-cell materialization, not the Arrow IPC batch parse (
RecordBatchReader.from/new Table), which still runs. So it brings fetch to an "Arrow batches parsed, rows not materialized" level, identical on both the Thrift and kernel backends.How is this tested?
tests/unit/result/ArrowResultConverter.test.ts—returns null placeholders with exact counts when disableRowMaterialization is set— mirrors the existing multi-batch "should respect row count in batch" case and asserts each fetch returns the exact row count with every elementnull.eslintclean on changed files,tsc --noEmitadds no new errors (the pre-existingexamples/tokenFederationmodule-resolution errors are present onmaintoo), and 155 unit tests pass across the touched suites (ArrowResultConverter,ResultSlicer,DBSQLClient,DBSQLOperation).