test(verify): compare the charts the two paths drew

[kz930]

What changes were proposed in this PR?

A visualization operator produces no table, so the table comparison has nothing to read. It writes a plotly figure and an HTML page, and this compares those.

A chart's meaning is in the numbers behind it, so the comparison reads those rather than the rendered picture, which would fail over a layout detail that carries none. The figure is compared trace by trace.

The page needs two things normalized away first, neither of which is markup the operator chose: a Styler id, which pandas regenerates on every run, and the line ending, which on Windows is CRLF for a file Python opened in text mode while the same markup carried through JSONL keeps the LF the engine wrote.

#8359 compares two tables, which is a different question and shares no code with this one.

Any related issues, documentation, discussions?

Part of #8325, 7 of 27; that issue lists the set in order.

Closes #8508, the task this change is the whole of.

How was this PR tested?

VisualizationHtmlComparatorSpec pins both normalizations: two pages differing only in a Styler id compare equal, and so do two differing only in line endings, while a page differing in the markup itself does not.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

🤖 Generated with Claude Code

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @mengw15
  You can notify them by mentioning @mengw15 in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.52%. Comparing base (1fbd346) to head (dfa14dc).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8509      +/-   ##
============================================
+ Coverage     93.69%   95.52%   +1.82%     
+ Complexity     4826      459    -4367     
============================================
  Files          1209      586     -623     
  Lines         49871    33436   -16435     
  Branches       6099     4147    -1952     
============================================
- Hits          46727    31939   -14788     
+ Misses         1652      936     -716     
+ Partials       1492      561     -931     

Flag	Coverage Δ		*Carryforward flag
access-control-service	80.18% <ø> (ø)
agent-service	99.32% <ø> (ø)		Carriedforward from 357b74c
amber	90.82% <ø> (+0.90%)	⬆️
computing-unit-managing-service	77.14% <ø> (ø)
config-service	87.12% <ø> (ø)
file-service	83.65% <ø> (ø)		Carriedforward from 357b74c
frontend	96.16% <ø> (ø)		Carriedforward from 357b74c
notebook-migration-service	83.73% <ø> (ø)
pyamber	98.47% <ø> (ø)		Carriedforward from 357b74c
workflow-compiling-service	74.09% <ø> (-3.10%)	⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[carloea2]

There is also a build dependency missing in this PR: VisualizationJsonComparator imports PythonWorkerPool from WorkflowOperator test sources, but WorkflowCompilingService does not depend on those test classes. The current CI job fails to compile that import. Please include the test dependency here so this PR builds independently. CI: https://github.com/apache/texera/actions/runs/34804943402/job/103854922027

[kz930]

Good catch, thanks. Added the test-scope dependency in dfa14dc, so the PR builds on its own now. The same line is in #8357 with identical wording, so the two merge without a conflict.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main fd09f20 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	618	0.377	15,698/24,269/24,269 us	🔴 +8.0% / 🔴 +65.2%
🔴	bs=100 sw=10 sl=64	1,344	0.82	73,897/104,072/104,072 us	🔴 +14.3% / 🟢 +22.7%
⚪	bs=1000 sw=10 sl=64	1,516	0.925	655,784/726,392/726,392 us	⚪ within ±5% / 🟢 +34.6%

Baseline details

Latest main fd09f20 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	618 tuples/sec	667 tuples/sec	838.23 tuples/sec	-7.3%	-26.3%
bs=10 sw=10 sl=64	MB/s	0.377 MB/s	0.407 MB/s	0.512 MB/s	-7.4%	-26.3%
bs=10 sw=10 sl=64	p50	15,698 us	14,533 us	11,878 us	+8.0%	+32.2%
bs=10 sw=10 sl=64	p95	24,269 us	25,493 us	14,690 us	-4.8%	+65.2%
bs=10 sw=10 sl=64	p99	24,269 us	25,493 us	18,483 us	-4.8%	+31.3%
bs=100 sw=10 sl=64	throughput	1,344 tuples/sec	1,360 tuples/sec	1,095 tuples/sec	-1.2%	+22.7%
bs=100 sw=10 sl=64	MB/s	0.82 MB/s	0.83 MB/s	0.668 MB/s	-1.2%	+22.7%
bs=100 sw=10 sl=64	p50	73,897 us	72,222 us	92,901 us	+2.3%	-20.5%
bs=100 sw=10 sl=64	p95	104,072 us	91,055 us	99,480 us	+14.3%	+4.6%
bs=100 sw=10 sl=64	p99	104,072 us	91,055 us	112,580 us	+14.3%	-7.6%
bs=1000 sw=10 sl=64	throughput	1,516 tuples/sec	1,519 tuples/sec	1,126 tuples/sec	-0.2%	+34.6%
bs=1000 sw=10 sl=64	MB/s	0.925 MB/s	0.927 MB/s	0.687 MB/s	-0.2%	+34.6%
bs=1000 sw=10 sl=64	p50	655,784 us	660,127 us	912,010 us	-0.7%	-28.1%
bs=1000 sw=10 sl=64	p95	726,392 us	695,996 us	950,922 us	+4.4%	-23.6%
bs=1000 sw=10 sl=64	p99	726,392 us	695,996 us	983,333 us	+4.4%	-26.1%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,323.61,200,128000,618,0.377,15698.30,24269.12,24269.12
1,100,10,64,20,1487.96,2000,1280000,1344,0.820,73897.11,104071.90,104071.90
2,1000,10,64,20,13196.61,20000,12800000,1516,0.925,655783.60,726391.66,726391.66