feat(frontend): add a panel that opens a workflow's own record (Hackathon UP) - #8525
Draft
Brb6754 wants to merge 1 commit into
Draft
feat(frontend): add a panel that opens a workflow's own record (Hackathon UP)#8525Brb6754 wants to merge 1 commit into
Brb6754 wants to merge 1 commit into
Conversation
Texera writes down every save and every run of a workflow and shows the user neither. `workflow_version` keeps one inverse JSON patch per save, and the runtime statistics of every run stay in Iceberg long after the results themselves are dropped. This adds a panel that reads both and answers, in plain language, the four questions people actually ask about a workflow they did not build, or built weeks ago: - How does this work? The flow drawn as one picture, each pipe as thick as the rows running through it, with a step inspector beside it. - What did I change? Every save as a block on a ribbon, and the canvas rebuilt at whichever one is picked. - What has been tried? Every run read as an experiment, so twenty-one runs collapse into the four distinct things that were actually tried. - Why did my results change? Two runs compared step by step, naming the first step whose output differs and the edit behind it. Two rules hold throughout. Every claim is derived from the record and can be checked; a model only puts those derived facts into words and never decides anything, so with no model the panel loses its sentences and nothing else. And a step that returns the same rows in a different order is not reported as a change, because that ordering comes from how work was split between workers rather than from any edit. The panel is opened from a button in the workspace menu. It is served by a new `spy-service`, which reads the database directly and holds no Texera session of its own; its single writing route carries the token of the user who has Texera open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqWWyuFAFLVswDYKXYyCwx
Contributor
|
👋 Thanks for your first contribution to Texera, @Brb6754! If you're looking for a good place to start, browse issues labeled You can drive common housekeeping yourself by commenting one of these commands on its own line:
Each command must match exactly: |
Contributor
Automated Reviewer SuggestionsBased on the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
A panel, opened from a button in the workspace menu, that shows the user what
Texera already records about a workflow and currently keeps to itself: every
save, and every run.
Nothing here is new data.
workflow_versionholds one inverse JSON patch persave, so the whole history of a canvas is recoverable by walking the patches
backwards. A run's results are dropped after thirty seconds, but its runtime
statistics are not: they stay in an Iceberg table with a row per operator per
sample, which is enough to say how many rows went in and out of every step of
every run ever made.
The panel reads both and answers four questions in plain language, which is what
its five views are named after:
rather than taken from the canvas, with each pipe as thick as the rows that
run through it, so where the flow narrows reads before any number does. A step
inspector alongside gives one step at a time: what it does, rows in and out,
its settings in words, and sample rows.
kind of save it was, with the canvas rebuilt at whichever block is picked and
that save's edits listed in words.
line, so the same settings run five times count as one thing tried. On the
workflow I developed against, twenty-one runs collapse to four.
first step whose output differs and the edit behind it.
hand because it costs a model call.
Two rules hold throughout, and they are the part I would most like reviewed:
already-derived facts into prose. It never decides which step diverged, which
edit caused it, or what any count means. With no model configured the panel
loses its sentences and nothing else, so it does not become a feature that
stops working when a key expires.
order is reported as exactly that. The engine splits work between workers
non-deterministically, and an earlier version of this did blame a join for it,
hiding the real edit further downstream.
Everything the panel writes is in plain language rather than in the system's own
terms: canvas names instead of identifiers, property paths translated through
the same JSON schema that draws the operator form (so a filter reads
amount > 400, never its JSON), and dates in words. The technical form is kept,one disclosure down.
Files. One new component under
frontend/src/app/workspace/component/spy/,three added lines' worth of button in the workspace menu, and a new
spy-servicethat answers it: it reads the database directly, holds no Texerasession of its own, and its single writing route carries the token of the user
who has Texera open, so whoever cannot run a workflow cannot run one from here.
Any related issues, documentation, discussions?
Proposed in #8524, which has the reasoning and the open question in full.
spy-service/README.mddocuments the modules and every environment variable.How was this PR tested?
No automated tests yet, which is the other thing I would like guidance on: the
parts worth testing are the patch application and the topological comparison,
and I would rather write them where the committers expect them to live than
guess at a home for them.
What was verified, by hand, against a workflow with 61 saves and 21 runs:
GET /api/version/{wid}/{vid}returns, compared one by one. That check turnedup a real bug in the process: Texera sometimes writes the same inverse patch
on several consecutive saves, typically
remove /operators/0/operatorProperties/envName,which appears because opening a user function's property panel writes the key
empty and the next save removes it. The first patch applies and the second
finds nothing to remove, which broke the chain and made everything earlier
look unrecoverable. Removing what is already gone leaves exactly the document
the patch describes, so it is tolerated and counted. With that, one workflow
went from 2 readable versions out of 56 to all 56. Worth flagging on its own:
restoring an old version has the same exposure.
threshold moved from 100 to 400, and a grouping key moved from segment to
city where one branch changes and the other does not.
run-and-keep button (9 steps, 2,731 rows kept).
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
The open design question
Comparing two runs row by row needs the rows, and Texera drops them thirty
seconds after the workflow goes idle. This draft keeps a copy as one JSON file
per run on the service's own disk. That is the weakest part of it: it does not
survive a restarted container, it does not respect the workflow's access
control, and nothing ever cleans it up. It should not survive review.
The rows do not need copying at all. When a run finishes they are already in an
Iceberg table. They disappear because
WorkflowService.clearExecutionResourcesclears them once
executionStateCleanUpInSecshas passed. So keeping a run forcomparison is a flag rather than a write: mark the execution as retained and let
the cleanup skip it, the way it already skips per-user warehouses through
WarehouseReadGuard.skipWhileDisabled. A retention policy would be neededalongside, since nothing would otherwise free the space.
That is likely worth having beyond this panel. "Keep this execution's results"
is something users ask for on its own.
I have left it as it is rather than guessing at the shape of that flag, and
would rather agree it first.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QqWWyuFAFLVswDYKXYyCwx