[feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support - #66761
Open
jacktengg wants to merge 24 commits into
Open
[feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support#66761jacktengg wants to merge 24 commits into
jacktengg wants to merge 24 commits into
Conversation
jacktengg
requested review from
924060929,
CalvinKirs,
airborne12,
csun5285,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde,
starocean999 and
yiguolei
as code owners
August 14, 2026 05:44
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
jacktengg
force-pushed
the
timestamp_ns-split-commit1-cast-and-functions
branch
from
August 14, 2026 17:03
a75a76c to
f7b433d
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17603 ms |
Contributor
TPC-DS: Total hot run time: 84745 ms |
Contributor
ClickBench: Total hot run time: 14.56 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review status: CONVERGED (three rounds). Requesting changes for two new P1 correctness blockers.
New findings:
- The newly registered TIMESTAMP_NS interval aliases perform unchecked signed negation, absolute value, and quarter multiplication before their range checks, so legal extreme interval operands can wrap or invoke undefined behavior.
- The new TIMESTAMP_NS
add_time/sub_timeruntime branch passes a signed TIME microsecond value into an unsignedTimeIntervalmagnitude without composing the operand sign with the operation sign, so negative TIME operands can disagree with FE folding.
Existing live/raw blocker families covering rolling-upgrade type exposure, mixed temporal coercion, pre-epoch window_funnel serialization, Arrow timezone metadata, fractional partition/statistics behavior, and Cloud on-update metadata remain current and were not duplicated inline.
Critical checkpoint conclusions:
- Goal, proof, scope, and focus: the 627-path cross-stack change implements a fixed-scale signed epoch-nanosecond type across FE, BE, storage, functions, protocols, and tests. The breadth is feature-driven; no extra user focus was supplied. Ordinary values, fractions, endpoints, nested types, protocols, storage/indexes, defaults, and aggregates have broad static test coverage, with the two arithmetic contracts above missing.
- Concurrency, lifecycle, and static state: additions are immutable values, static registrations, or query/expression/aggregate-local state. No new thread, shared mutable owner, lock order, cross-TU lifetime, background task, or cleanup imbalance was found.
- Configuration and compatibility: no dynamic configuration was added. Thrift/protobuf IDs and execution-version paths were checked; the surviving mixed-version concern is already covered by an existing thread.
- Parallel paths, special values, and errors: FE/BE folding, nullable/const dispatch, comparisons/joins/IN, ASOF, arrays/maps/Variant, sequence, floor/ceil, casts, timezone conversion, aggregates, and transport siblings were traced. Surviving concerns are the two inline findings or existing duplicate fences; no ignored status or independent silent-continuation path remained.
- Tests and results: unit/regression queries and generated outputs were inspected for signed endpoints, negative epochs, nanosecond fractions, functions, storage/indexes, protocols, and expected errors. Extreme integer intervals and negative TIME operands are not covered. No build or test was run because the authoritative review bundle forbids both.
- Observability and performance: no new distributed/background workflow requires metrics, no hot-loop logging was added, and conversions remain bounded fixed-width or civil-calendar work. No independent dominant-path regression was substantiated.
- Persistence, transactions, writes, and crash behavior: local schema/default round trips, storage encodings, keys, indexes, Variant, PBlock/binary forms, and sentinel handling were checked. No new EditLog, visibility, MoW bitmap, atomicity, leak, or crash-recovery defect was found; already-commented Cloud/statistics gaps remain.
- FE/BE variables and protocols: statement millisecond/nanosecond snapshots, descriptors, PValues, MySQL, HTTP/Flight, Arrow, JNI/Java UDF, and persisted-schema paths were checked. Endpoint payloads remain symmetric apart from existing timezone/rollout fences.
- Core invariants: committed-data visibility, visible versions, delete bitmaps, and tracked-memory ownership are not changed by this feature.
Static review only; no source changes, builds, or tests were performed.
Contributor
TPC-H: Total hot run time: 16944 ms |
Contributor
TPC-DS: Total hot run time: 81754 ms |
Contributor
ClickBench: Total hot run time: 14.44 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#66761 Problem Summary: String-to-TIMESTAMP_NS casts always used the strict BE parser, so DATETIMEV2-compatible flexible formats produced NULL at runtime while FE constant folding succeeded. Use DATETIMEV2 non-strict parsing for non-strict casts while preserving nanosecond precision, rounding, timezone conversion, range checks, and strict-mode behavior. ### Release note String casts to TIMESTAMP_NS now accept DATETIMEV2-compatible non-strict formats when enable_strict_cast=false. ### Check List (For Author) - Test: - Unit Test: FunctionCastTest.string_to_timestamp_ns* and DataTypeTimeStampNsTest.* - Regression test: datatype_p0/timestamp_ns/test_cast_string_non_strict - Behavior changed: Yes. Non-strict string casts to TIMESTAMP_NS accept DATETIMEV2-compatible flexible formats. - Does this need documentation: No
66 tasks
### What problem does this PR solve? Issue Number: None Related PR: apache#66761 Problem Summary: Existing TIMESTAMP_NS regression coverage did not systematically exercise supported expressions and functions with all-literal arguments under both constant-folding modes, mixed literal and column arguments under both modes, and all-column arguments. Add expression and function matrices using the exact minimum and maximum values, the Unix epoch, and a normal nanosecond value. Respect signatures that require literal intervals or formats, cover aggregate and window paths, and verify boundary overflow errors without nondeterministic ordering. ### Release note None ### Check List (For Author) - Test: Regression test - test_timestamp_ns_expression_argument_matrix - test_timestamp_ns_function_argument_matrix - test_timestamp_ns_expressions - test_timestamp_ns_functions - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66761 Problem Summary: TIMESTAMP_NS string casts duplicated DATETIMEV2 grammar, fractional parsing, rounding, and timezone conversion logic. Template the shared DATETIMEV2 string parser over its result type, retain DATETIMEV2 behavior, parse up to nine fractional digits for TIMESTAMP_NS, and finalize through TimeStampNsValue after timezone conversion and range validation. Route the TIMESTAMP_NS SerDe through the shared cast helpers. ### Release note None ### Check List (For Author) - Test: - Unit Test: targeted TIMESTAMP_NS, DATETIMEV2, and TIMESTAMPTZ cast tests - Regression test: test_cast_string_non_strict and test_timestamp_ns_cast - Build: ./build.sh --be - Style: clang-format, check-format, and build-hygiene checks - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66761 Problem Summary: The nanosecond function only exposed a TIMESTAMP_NS signature, so DATEV2 and DATETIMEV2 arguments failed during function resolution. Add FE signatures and constant folding overloads, and register BE implementations that return zero for DATEV2 and convert DATETIMEV2 microseconds to nanoseconds while preserving exact TIMESTAMP_NS extraction. ### Release note The nanosecond function now supports DATEV2 and DATETIMEV2 arguments. ### Check List (For Author) - Test: - Unit Test: targeted FE and BE nanosecond tests - Regression test: test_nanosecond_datev2_datetimev2 - Build: ./build.sh --fe and ./build.sh --be - Style: clang-format, check-format, checkstyle, and build hygiene - Behavior changed: Yes (nanosecond now accepts DATEV2 and DATETIMEV2) - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17202 ms |
Contributor
TPC-DS: Total hot run time: 82146 ms |
Contributor
ClickBench: Total hot run time: 14.67 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Issue Number: None Related PR: apache#66761 Problem Summary: TIMESTAMP_NS had basic Arrow round-trip coverage but lacked complete time-unit boundary checks, nested FE schema checks, and an end-to-end Arrow Flight SQL regression. Add exact conversion and overflow tests, nested schema assertions, and Flight queries for nanosecond boundary values under multiple session time zones. None - Test: Unit Test and Regression test - `./run-fe-ut.sh --run org.apache.doris.service.arrowflight.FlightSqlSchemaHelperArrowTypeTest` - `./run-be-ut.sh --run --filter=DataTypeSerDeArrowTest.TimeStampNsArrowRoundTrip:DataTypeSerDeArrowTest.TimeStampNsArrowUnitConversion -j 16` - `./run-regression-test.sh --run -d arrow_flight_sql_p0 -s test_select` - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 16931 ms |
Contributor
TPC-DS: Total hot run time: 84247 ms |
Contributor
ClickBench: Total hot run time: 14.94 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Docs: apache/doris-website#4091
Problem Summary:
Doris DATETIME only support at most six fractional digits, it's not sufficient for event times, traces, etc which require higher precision of sub-seconds.
Introduce
TIMESTAMP_NSas a separate fixed-scale SQL type backed by signed Int64 Unix-epoch nanoseconds. It always has scale 9 and supports values from1677-09-21 00:12:43.145224192through2262-04-11 23:47:16.854775807. Wire the type through Thrift and protobuf metadata, the FE catalog and Nereids, BE columns and SerDe, storage and indexes, result protocols, JNI and Java UDF transport, complex and VARIANT types, and schema metadata.Add exact casts, comparisons, type inference, and constant folding across date-like types while preserving nanoseconds and rejecting partial or precision-losing implicit conversions. Add timestamp-aware scalar and calendar functions, aggregation, joins and ASOF joins, scan predicates, runtime filters, partition pruning, query-cache normalization, automatic partitions, and MTMV roll-up. Keep FE folding and BE runtime behavior consistent at signed endpoints, leap years, DST gaps, calendar-clamped boundaries, and fractional formatting and rounding boundaries.
Keep existing DATETIME and DATETIMEV2 scale semantics unchanged. FROM_UNIXTIME retains a DECIMAL64 microsecond path for scale 0 through 6 and uses the DECIMAL128 nanosecond path only for scale 7 through 9. The %f formatter rounds half-up to microseconds with second carry, while %n preserves the nanosecond fraction.
Release note
Add
TIMESTAMP_NS, a fixed-scale nanosecond timestamp type stored as signed Unix-epoch nanoseconds. It supports SQL literals, casts, date/time functions, storage and indexes, partitions, joins, aggregates, complex types, Java UDF/JNI transport, and FE constant folding and type inference. Existing DATETIME and DATETIMEV2 types remain limited to scale 0 through 6.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)