Support wide decimals in `DecimalBytePartsArray` and kernels by mhk197 · Pull Request #9809 · vortex-data/vortex

mhk197 · 2026-09-08T20:36:13Z

DecimalBytePartsArray previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.

The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:

Decimal storage	Children
`i8` / `i16` / `i32` / `i64`	Signed MSP only; shares the original value buffer
`i128`	`i64` MSP + one `u64` lower part
`i256`	`i64` MSP + three `u64` lower parts

Only the MSP carries validity. Every lower part must be a non-nullable u64 array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain ArrayRefs, so their individual encodings are independent of the decimal representation.

execute::<DecimalArray> reassembles a DecimalArray from the MSP and lower parts children.

take with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.

codspeed-hq · 2026-09-08T20:44:01Z

Merging this PR will regress 2 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 2223 untouched benchmarks
🆕 72 new benchmarks
⏩ 204 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`decompress[datetime_for_bp]`	160.3 µs	193.9 µs	-17.32%
❌	WallTime	`filtered_owned_i64_avx512[OneNullInEight]`	22.4 µs	26.3 µs	-15.02%
⚡	Simulation	`decompress[u64, (4000, 1024)]`	86.3 µs	71.5 µs	+20.7%
⚡	Simulation	`allocate_drop_arrow[0]`	456.9 ns	402.7 ns	+13.45%
⚡	Simulation	`chunked_bool_canonical_into[(1000, 10)]`	30.7 µs	27.2 µs	+12.9%
⚡	Simulation	`allocate_drop_bytes[0]`	575.7 ns	521.6 ns	+10.39%
🆕	WallTime	`dbp_assemble_neon[(I128, 1024)]`	N/A	605 ns	N/A
🆕	WallTime	`dbp_assemble_neon[(I128, 8192)]`	N/A	3.7 µs	N/A
🆕	WallTime	`dbp_assemble_neon[(I256, 1024)]`	N/A	2.1 µs	N/A
🆕	WallTime	`dbp_assemble_neon[(I256, 8192)]`	N/A	15 µs	N/A
🆕	WallTime	`dbp_assemble_neon[(I64, 1024)]`	N/A	108 ns	N/A
🆕	WallTime	`dbp_assemble_neon[(I64, 8192)]`	N/A	111 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I128, 1024)]`	N/A	246 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I128, 8192)]`	N/A	251 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I256, 1024)]`	N/A	254 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I256, 8192)]`	N/A	257 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I64, 1024)]`	N/A	271 ns	N/A
🆕	WallTime	`dbp_split_all_null_neon[(I64, 8192)]`	N/A	287 ns	N/A
🆕	WallTime	`dbp_split_all_valid_neon[(I128, 1024)]`	N/A	767 ns	N/A
🆕	WallTime	`dbp_split_all_valid_neon[(I128, 8192)]`	N/A	4.2 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing mk/dbp-array (18b8b99) with mk/dbp-v2-feature (4edb7aa)}

204 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

Represent wide decimals with a signed high part and up to three unsigned low parts. Add validation, execution, kernel support, property tests, and assembly benchmarks while keeping serialization on the frozen format. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Move array helpers onto a crate-private extension trait, preserve decimal precision and scale when replacing the MSP, and group slicing with the other compute operations. Inline canonical execution and select scalar storage directly from the lower-part count. Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

mhk197 · 2026-09-10T18:02:57Z

            .cast(array.msp().dtype().with_nullability(*target_nullability))?;

-        Ok(Some(
-            DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(),


I think this was redundant to begin with because we check above that target dtype is same as current modulo nullability

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs · 2026-09-11T09:15:57Z

    divan::main();
 }

+#[vortex_bench_support::cpu_features]


can you do this over the kernel working with [T] not vortex arrays. This is more noisy so it would be nice to only run if over small loops with allocs

joseph-isaacs · 2026-09-11T09:16:09Z

joseph-isaacs · 2026-09-11T09:16:57Z


    #[test]
    fn test_cast_decimal_byte_parts_nullability() {
        let mut ctx = array_session().create_execution_ctx();
        let decimal_dtype = DecimalDType::new(10, 2);
        let array =
            DecimalByteParts::try_new(buffer![100i32, 200, 300, 400].into_array(), decimal_dtype)
                .unwrap();

        // Cast to nullable decimal
        let casted = array
            .into_array()
            .cast(DType::Decimal(decimal_dtype, Nullability::Nullable))
            .unwrap();
        assert_eq!(
            casted.dtype(),
            &DType::Decimal(decimal_dtype, Nullability::Nullable)
        );

        // Verify the values are preserved
        let decoded = casted.execute::<DecimalArray>(&mut ctx).unwrap();
        assert_eq!(decoded.len(), 4);
    }

    #[test]
    fn test_cast_decimal_byte_parts_nullable_to_non_nullable() {
        let mut ctx = array_session().create_execution_ctx();
        let decimal_dtype = DecimalDType::new(10, 2);
        let array = DecimalByteParts::try_new(
            PrimitiveArray::from_option_iter([Some(100i32), None, Some(300)]).into_array(),
            decimal_dtype,
        )
        .unwrap();

        // Cast to non-nullable should fail due to nulls - force evaluation via execute::<Canonical>
        let result = array
            .into_array()
            .cast(DType::Decimal(decimal_dtype, Nullability::NonNullable))
            .and_then(|a| a.execute::<Canonical>(&mut ctx).map(|c| c.into_array()));
        assert!(result.is_err());
    }

    #[rstest]
    #[case::i32(DecimalByteParts::try_new(
        buffer![100i32, 200, 300, 400, 500].into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
    #[case::i64(DecimalByteParts::try_new(
        buffer![1000i64, 2000, 3000, 4000].into_array(),
        DecimalDType::new(19, 4),
    ).unwrap())]
    #[case::nullable(DecimalByteParts::try_new(
        PrimitiveArray::from_option_iter([Some(100i32), None, Some(300), Some(400), None])
            .into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
    #[case::single(DecimalByteParts::try_new(
        buffer![42i32].into_array(),
        DecimalDType::new(5, 1),
    ).unwrap())]
    #[case::negative(DecimalByteParts::try_new(
        buffer![-100i32, -200, 300, -400, 500].into_array(),
        DecimalDType::new(10, 2),
    ).unwrap())]
+    #[case::one_lower_part(i128_parts(
+        vec![1i128 << 70, -(1i128 << 70), 5, (1i128 << 64) - 1, 0],
+        Validity::NonNullable,
+    ))]
+    #[case::three_lower_parts(i256_parts(
+        vec![i256_of(1, 0), i256_of(-1, 5), i256_of(0, u128::MAX)],
+        Validity::NonNullable,
+    ))]
    fn test_cast_decimal_byte_parts_conformance(#[case] array: DecimalBytePartsArray) {


we don't need this many new tests

joseph-isaacs · 2026-09-11T09:17:22Z

+        // The MSP alone only determines the ordering when it holds the whole value. With
+        // lower parts present, fall back to comparing the canonical decimal.
+        if !lhs.lower_parts().is_empty() {
+            return Ok(None);
+        }


add a todo saying we could be smarter here

joseph-isaacs · 2026-09-11T09:19:43Z

+impl TakeReduce for DecimalByteParts {
+    fn take(array: ArrayView<'_, Self>, indices: &ArrayRef) -> VortexResult<Option<ArrayRef>> {
+        // Taking with nullable indices makes every taken part nullable, but lower parts must
+        // stay non-nullable `u64` — validity belongs to the MSP alone. Fall back to the
+        // canonical path rather than rebuilding parts we would have to strip nullability from.
+        if indices.dtype().is_nullable() && !array.lower_parts().is_empty() {
+            return Ok(None);
+        }
+
+        array
+            .map_parts(|part| part.take(indices.clone()))
+            .map(|a| Some(a.into_array()))
+    }
+}


add a todo saying we could impl this using fill null or smthing else

joseph-isaacs · 2026-09-11T09:22:08Z

+    let mut value = T::from(msp).vortex_expect("MSP fits in the output type");
+    for part in lower {
+        value = (value << LOWER_PART_BITS)
+            | T::from(part).vortex_expect("lower word fits in the output type");


Can we remove this failure it will likely break simd. Can this ever fail. It should be checked outside the loop once and not checked here?

Can add infallible upcast from u64/i64 to i256 instead

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

This was referenced Sep 8, 2026

Compress wide decimals when the writer permits the v2 format #9759

Draft

Add wide decimal byte-part splitting and assembly logic #9808

Merged

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format #9810

Open

mhk197 force-pushed the mk/dbp-array branch 3 times, most recently from f7dfebe to a566a28 Compare September 9, 2026 02:43

mhk197 changed the title ~~Support multi-part decimal arrays and kernels~~ Support wide decimals in DecimalBytePartsArray and kernels Sep 9, 2026

mhk197 marked this pull request as ready for review September 9, 2026 04:17

mhk197 removed this pull request from stack #9811 September 9, 2026 15:09

mhk197 force-pushed the mk/dbp-array branch from f12c08c to 32a80b1 Compare September 9, 2026 15:09

mhk197 added this pull request to stack #9813 September 9, 2026 15:09

mhk197 force-pushed the mk/dbp-array branch from 32a80b1 to 9a179ed Compare September 9, 2026 18:31

mhk197 added changelog/skip Do not list PR in the changelog and removed changelog/skip Do not list PR in the changelog labels Sep 9, 2026

mhk197 force-pushed the mk/dbp-array branch from 9a179ed to 76a0db4 Compare September 9, 2026 20:06

mhk197 requested review from joseph-isaacs and robert3005 September 9, 2026 20:28

Base automatically changed from mk/dbp-parts to mk/dbp-v2-feature September 10, 2026 15:15

mhk197 added 5 commits September 10, 2026 16:15

Share decimal value assembly and validate lower-part dtypes

e30bd99

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Validate required decimal byte-parts slots before typed access

6a7b033

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

fix comment

a20094e

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs force-pushed the mk/dbp-array branch from 7283d4f to a20094e Compare September 10, 2026 15:15

mhk197 commented Sep 10, 2026

View reviewed changes

mhk197 added 2 commits September 10, 2026 16:26

QOL and perf changes for splitting

38b5bc4

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

comment fix

df7cbfc

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

reorg

b2e49fb

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

joseph-isaacs reviewed Sep 11, 2026

View reviewed changes

Comment thread encodings/decimal-byte-parts/benches/dbp_split.rs

joseph-isaacs Sep 11, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

joseph-isaacs reviewed Sep 11, 2026

View reviewed changes

comments

18b8b99

Signed-off-by: Matt Katz <mhkatz97@gmail.com>

Navigation Menu

Uh oh!

Support wide decimals in `DecimalBytePartsArray` and kernels - #9809

Support wide decimals in `DecimalBytePartsArray` and kernels#9809
mhk197 wants to merge 9 commits into
mk/dbp-v2-featurefrom
mk/dbp-array

mhk197 commented Sep 8, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

mhk197 Sep 10, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

joseph-isaacs Sep 11, 2026

Uh oh!

mhk197 Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mhk197 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 2 benchmarks

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mhk197 commented Sep 8, 2026 •

edited

Loading

codspeed-hq Bot commented Sep 8, 2026 •

edited

Loading