Skip to content

feat(operator): read the whole-number parts out of a timestamp column - #8480

Draft
kz930 wants to merge 2 commits into
apache:mainfrom
kz930:feat/extract-datetime-fields
Draft

feat(operator): read the whole-number parts out of a timestamp column#8480
kz930 wants to merge 2 commits into
apache:mainfrom
kz930:feat/extract-datetime-fields

Conversation

@kz930

@kz930 kz930 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

An operator that reads the year, quarter, month, day, weekday, day of year, ISO week, hour, minute or second out of a timestamp column, each added as a whole-number column of its own.

Nothing here is parsed. The column is already a moment when it arrives, and which text became which moment was settled upstream, the way KNIME, Alteryx and Spark all separate parsing from extraction. The parts are read as ISO-8601 states them, because the two runtimes disagree by default: pandas counts Monday as 0, java.time counts it as 1.

Any related issues, documentation, discussions?

Not part of #8325: that set makes the operators Texera already had exportable. This adds a new one, which implements the trait from the start rather than gaining it. The trait is in #8327, which has merged.

Closes #8478, the task this change is the whole of.

Aggregate groups by a column's values, and a timestamp has a different value in almost every row, so grouping by one puts each row in its own group. Sales by month could not be asked for, though the platform carries a TIMESTAMP type, a Gantt chart and a time-series plot.

How was this PR tested?

Twelve unit tests in the operator's own spec, over the schema it declares, the values the executor reads and the Python it emits. The parity harness runs it on 13 configurations, one per field plus a hostile column name and an empty cell, comparing the engine's answer against the exported script's.

The operator in the editor, reading event_time and asking for the year, the month and the weekday:

config

The three whole-number columns beside the timestamp they came from. 2024-01-15 is a Monday, 2024-07-04 a Thursday and 2025-02-09 a Sunday, reading 1, 4 and 7 as ISO-8601 states them rather than the 0, 3 and 6 pandas would give:

result

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@github-actions github-actions Bot added feature dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI common platform Non-amber Scala service paths labels Sep 10, 2026
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 75c85aa benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 352 0.215 25,336/74,051/74,051 us 🔴 +117.6% / 🔴 +359.0%
🔴 bs=100 sw=10 sl=64 783 0.478 124,131/159,809/159,809 us 🔴 +16.2% / 🔴 +47.0%
bs=1000 sw=10 sl=64 907 0.553 1,098,777/1,152,424/1,152,424 us ⚪ within ±5% / 🔴 +11.1%
Baseline details

Latest main 75c85aa from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 352 tuples/sec 408 tuples/sec 755.28 tuples/sec -13.7% -53.4%
bs=10 sw=10 sl=64 MB/s 0.215 MB/s 0.249 MB/s 0.461 MB/s -13.7% -53.4%
bs=10 sw=10 sl=64 p50 25,336 us 23,679 us 12,957 us +7.0% +95.5%
bs=10 sw=10 sl=64 p95 74,051 us 34,032 us 16,134 us +117.6% +359.0%
bs=10 sw=10 sl=64 p99 74,051 us 34,032 us 20,333 us +117.6% +264.2%
bs=100 sw=10 sl=64 throughput 783 tuples/sec 824 tuples/sec 980.1 tuples/sec -5.0% -20.1%
bs=100 sw=10 sl=64 MB/s 0.478 MB/s 0.503 MB/s 0.598 MB/s -5.0% -20.1%
bs=100 sw=10 sl=64 p50 124,131 us 119,840 us 101,894 us +3.6% +21.8%
bs=100 sw=10 sl=64 p95 159,809 us 137,502 us 108,718 us +16.2% +47.0%
bs=100 sw=10 sl=64 p99 159,809 us 137,502 us 122,482 us +16.2% +30.5%
bs=1000 sw=10 sl=64 throughput 907 tuples/sec 915 tuples/sec 1,011 tuples/sec -0.9% -10.3%
bs=1000 sw=10 sl=64 MB/s 0.553 MB/s 0.559 MB/s 0.617 MB/s -1.1% -10.4%
bs=1000 sw=10 sl=64 p50 1,098,777 us 1,093,457 us 996,422 us +0.5% +10.3%
bs=1000 sw=10 sl=64 p95 1,152,424 us 1,126,982 us 1,037,670 us +2.3% +11.1%
bs=1000 sw=10 sl=64 p99 1,152,424 us 1,126,982 us 1,072,152 us +2.3% +7.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,568.47,200,128000,352,0.215,25336.28,74051.10,74051.10
1,100,10,64,20,2555.13,2000,1280000,783,0.478,124130.63,159809.45,159809.45
2,1000,10,64,20,22061.29,20000,12800000,907,0.553,1098776.74,1152424.07,1152424.07

Aggregate groups by a column's values, and a timestamp has a different value in
almost every row, so grouping by one puts each row in its own group. Sales by
month could not be asked for, though the platform carries a TIMESTAMP type, a
Gantt chart and a time-series plot.

Nothing here is parsed: the column is already a moment by the time it arrives,
and which text became which moment was settled upstream, the way KNIME, Alteryx
and Spark all separate parsing from extraction. The parts are read as ISO-8601
states them, since the two runtimes do not agree by default — pandas counts
Monday as 0 and java.time counts it as 1.

Each part is added under a name derived from the column it came from, so reading
two timestamp columns names four distinct results.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930
kz930 force-pushed the feat/extract-datetime-fields branch from d6f6a34 to 535f06e Compare September 10, 2026 05:15
@github-actions github-actions Bot removed dependencies Pull requests that update a dependency file platform Non-amber Scala service paths labels Sep 10, 2026
@codecov-commenter

codecov-commenter commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.89%. Comparing base (75c85aa) to head (95aadac).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8480      +/-   ##
============================================
+ Coverage     93.62%   93.89%   +0.26%     
+ Complexity     4857     4840      -17     
============================================
  Files          1212     1210       -2     
  Lines         50037    49261     -776     
  Branches       6132     5991     -141     
============================================
- Hits          46847    46253     -594     
+ Misses         1676     1516     -160     
+ Partials       1514     1492      -22     
Flag Coverage Δ *Carryforward flag
access-control-service 80.18% <ø> (ø)
agent-service 99.32% <ø> (ø) Carriedforward from dcb25ef
amber 89.91% <ø> (+0.05%) ⬆️ Carriedforward from dcb25ef
computing-unit-managing-service 77.14% <ø> (ø)
config-service 87.12% <ø> (ø)
file-service 83.65% <ø> (ø)
frontend 96.80% <ø> (+0.64%) ⬆️ Carriedforward from dcb25ef
notebook-migration-service 83.73% <ø> (ø)
pyamber 98.47% <ø> (ø) Carriedforward from dcb25ef
workflow-compiling-service 74.09% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kz930

kz930 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 this adds an operator that reads the year, month, weekday and the rest out of a timestamp column, each as a whole-number column of its own. Would you take a look when you have a moment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common feature frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read the whole-number parts out of a timestamp column

2 participants