feat(workflow-operator): export the first five operators, and serve the script - #8502
Merged
Merged
Conversation
…he script Distinct, Filter, Limit, Projection and Union implement the trait the translator reads, which is what turns it from a mechanism into something that produces a running script. Union is the one with a variadic port, so it names the whole list of upstreams rather than a fixed count. The endpoint the editor calls comes with them, because what it is worth testing on is a script that runs: it takes a plan, compiles it so a scan source can read the schema off the file it points at, hands the translator the output schemas that gives, and returns the result. A failed compile is logged and translation goes on without the schemas, so a workflow whose file is not chosen yet still exports. pyStringLiteral comes with them too: a generator has to write a column name into the source it emits, and writing the quotes by hand lets any quote, backslash or newline in the name close the literal early. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 445 | 0.271 | 21,116/32,087/32,087 us | 🟢 -6.7% / 🔴 +98.9% |
| 🔴 | bs=100 sw=10 sl=64 | 930 | 0.568 | 104,789/142,438/142,438 us | 🔴 +10.8% / 🔴 +31.0% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,060 | 0.647 | 944,709/1,011,320/1,011,320 us | 🟢 -8.1% / 🟢 -5.7% |
Baseline details
Latest main 4d7fd49 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 445 tuples/sec | 418 tuples/sec | 755.28 tuples/sec | +6.5% | -41.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.271 MB/s | 0.255 MB/s | 0.461 MB/s | +6.3% | -41.2% |
| bs=10 sw=10 sl=64 | p50 | 21,116 us | 22,643 us | 12,957 us | -6.7% | +63.0% |
| bs=10 sw=10 sl=64 | p95 | 32,087 us | 32,804 us | 16,134 us | -2.2% | +98.9% |
| bs=10 sw=10 sl=64 | p99 | 32,087 us | 32,804 us | 20,333 us | -2.2% | +57.8% |
| bs=100 sw=10 sl=64 | throughput | 930 tuples/sec | 956 tuples/sec | 980.1 tuples/sec | -2.7% | -5.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.568 MB/s | 0.583 MB/s | 0.598 MB/s | -2.6% | -5.0% |
| bs=100 sw=10 sl=64 | p50 | 104,789 us | 101,292 us | 101,894 us | +3.5% | +2.8% |
| bs=100 sw=10 sl=64 | p95 | 142,438 us | 128,520 us | 108,718 us | +10.8% | +31.0% |
| bs=100 sw=10 sl=64 | p99 | 142,438 us | 128,520 us | 122,482 us | +10.8% | +16.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,060 tuples/sec | 1,070 tuples/sec | 1,011 tuples/sec | -0.9% | +4.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.647 MB/s | 0.653 MB/s | 0.617 MB/s | -0.9% | +4.8% |
| bs=1000 sw=10 sl=64 | p50 | 944,709 us | 927,671 us | 996,422 us | +1.8% | -5.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,011,320 us | 1,100,560 us | 1,037,670 us | -8.1% | -2.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,011,320 us | 1,100,560 us | 1,072,152 us | -8.1% | -5.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,449.61,200,128000,445,0.271,21115.58,32086.74,32086.74
1,100,10,64,20,2150.02,2000,1280000,930,0.568,104788.93,142438.06,142438.06
2,1000,10,64,20,18865.44,20000,12800000,1060,0.647,944708.52,1011319.66,1011319.66
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8502 +/- ##
============================================
- Coverage 93.69% 93.69% -0.01%
+ Complexity 4826 4821 -5
============================================
Files 1209 1209
Lines 49871 49866 -5
Branches 6099 6099
============================================
- Hits 46727 46722 -5
Misses 1652 1652
Partials 1492 1492
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Sep 11, 2026
Contributor
Author
|
/request-review @aglinxinyuan |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Five operators implement
StandaloneCodeGenerator, which is what turns the translator from a mechanism into something that produces a script that runs: Distinct, Filter, Limit, Projection and Union. Four of them read a single input; Union reads a variadic port, so it names the whole list of upstreams rather than a fixed count, which a fixed count gets wrong in both directions.The endpoint the editor calls comes with them, because what it is worth testing on is a script that runs rather than one made of placeholders. It takes a plan, compiles it first so a scan source can read the schema off the file it points at, hands the translator the output schemas that gives, and returns the script. A failed compile is logged and translation goes on without them, so a workflow whose file is not chosen yet still exports.
pyStringLiteralcomes with them too. A generator has to write a column name into the source it emits, and writing the quotes by hand lets any quote, backslash or newline in the name close the literal early and change, or break, the emitted program.#8327 is the trait and the translator, and nothing that uses them.
Any related issues, documentation, discussions?
Part of #8325, 2 of 27; that issue lists the set in order.
Closes #8501, the task this change is the whole of.
How was this PR tested?
Each operator asserts the block it emits in its own spec.
PythonTemplateBuilderApiSpeccovers whatpyStringLiteralescapes, including the NUL that Python refuses to compile anywhere in a source file.WorkflowToPythonResourceSpecdrives the endpoint over a plan built from these operators and reads the script back.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with Claude Code