Skip to content

Reduce repeated full-projection lookups during workflow execution - #509

Merged
rmcdaniel merged 2 commits into
mainfrom
fix/projection-replay-cost
Sep 11, 2026
Merged

Reduce repeated full-projection lookups during workflow execution#509
rmcdaniel merged 2 commits into
mainfrom
fix/projection-replay-cost

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 11, 2026

Copy link
Copy Markdown
Member

Problem

Relates to #508. Repeated full monitoring projection rereads each existing timeline, wait, and timer row individually. Summary projection also derives activity and timer views more than once in the same pass. A small embedded workflow with three parallel activities followed by a timer per round exposes the growing cost.

Changes

  • Prefetch each run's existing projection rows using the configured model, table, and connection; reuse those Eloquent instances within that pass only.
  • Retain casts, model save hooks, duplicate-key recovery, point-delete stale cleanup, and history-based repair. Incremental child/claim projection paths are unchanged.
  • Reuse the activity metadata and timer views already computed by full summary projection.
  • Add fixed-query-count, unchanged-row/timestamp, repair, unrelated-run, custom-model/connection, and cold-reload workflow regression tests.

No new cache, background projector, schema, durable-history format, or public execution contract.

Validation

Local PHP 8.4.25 / Laravel 13 source checks:

  • 15 focused tests / 98 assertions passed, including cold activity metadata projection without external-payload reads and configured-model/table/secondary-connection cases.
  • Full phpstan analyse src tests and changed-file ECS passed.
  • Full source CI passed at c59828f4c7be77c6efd078419d0aed548c26c8fe: unit/coverage, all MySQL and PostgreSQL shards, MariaDB, Laravel 9-13 upgrade checks, regression corpus, contracts, and quality. PR checks also passed.
  • Final source review confirmed that prefetch is run-scoped and same-pass only; incremental child/claim paths, configured model connections, duplicate-key handling, and stale point-delete behavior are preserved.

Diagnostic only, not a published capacity benchmark: matched sequential runs on the same host, container limit 1 CPU / 1 GiB, SQLite in memory, fake no-op activity bodies, Workflow 2.0.12 versus this source overlay. Sixteen rounds produced 48 activity completions and 179 history events; both runs completed with the correct output and activity count.

Measurement Baseline Candidate
Total test duration 83.391 s 65.224 s
Round 15 duration 10.050 s 7.929 s
Round 15 SQL queries 5,860 1,196
PHP peak allocated memory 79 MB 79 MB

XHProf on a separate three-round baseline attributes about 75% of elapsed profiled time to full summary projection. Profiler overhead is excluded from the timing table. The repository-owned three-round correctness reproduction is tests/Unit/V2/ProjectionReplayTest.php; query-count regression is in ProjectionPrefetchTest.php.

This is a partial optimization: the remaining history-dependent PHP cost is still material. Keep #508 open until its acceptance criteria, broader correctness checks, and published-consumer follow-through are complete. No claim that this patch alone resolves the full performance problem.

Before Delivery

  • Source checks and review are complete. Publish the patch and verify installed-package behavior before calling it delivered.
  • Keep remaining history-dependent work explicit; this patch removes measured redundant work without claiming constant-time full history projection.
  • On publication, audit pinned Workflow consumers (including Server) and run affected conformance with published artifacts. Do not release unchanged language SDKs merely to align version numbers.

Current main consumer audit: Server requires Workflow 2.0.12 exactly and will need a dependency update and rebuilt release. Sample App's embedded lock is 2.0.12 under ^2.0, requiring a lock update and first-user verification. Waterline's Workflow dependency is development-only; refresh relevant embedded integration qualification without claiming a runtime package release is necessary. PHP SDK and CLI have no Workflow runtime dependency; there is no portable wire change requiring synchronized SDK releases. Deployment-specific follow-through stays in its owning private record.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The source change passed full local PHPStan and 15 focused cases in aggregate, covering configured model/table/connection routing, cold-reload projection consistency, metadata-only external payload behavior, existing unique-key recovery, and fixed projection read counts. PR functional/corpus checks passed; the first style check caught one chained-call indentation introduced by the formatter. Commit c59828f corrects it and a second local style check is clean. The first broad run was cancelled and one complete source validation has been requested for the corrected commit. This remains draft, with no published-consumer or production performance claim.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.06%. Comparing base (3994044) to head (c59828f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #509      +/-   ##
==========================================
+ Coverage   88.04%   88.06%   +0.02%     
==========================================
  Files         460      460              
  Lines       63036    63059      +23     
==========================================
+ Hits        55497    55531      +34     
+ Misses       7539     7528      -11     
Flag Coverage Δ
v2 88.06% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 11, 2026 18:59
@rmcdaniel
rmcdaniel merged commit 2d783ee into main Sep 11, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants