v0.1.10-4: Add Unified Request Tracing and Queue-Stage Observability

[rickisba]

Background

Later queueing and network-compute overlap work requires reproducible stage-level measurements. The current logs do not provide a stable request timeline or consistently separate estimates from measured values.

Proxy KVCache Management will primarily observe Instance cache state through LMCache lookup/status/controller APIs and cache events. Tracing must therefore preserve the difference between CacheRoute predictions, KDN policy decisions, LMCache observations, and vLLM execution measurements.

Objective

Introduce a unified request trace that records knowledge resolution, LMCache-observed cache preparation, and model-execution stages without changing current routing, injection, or forwarding behavior.

Implementation Scope

• Define a stable request-trace schema covering, when applicable:
  • KDN knowledge lookup and semantic resolution;
  • logical Artifact compatibility matching;
  • LMCache lookup/status query;
  • cache-plan construction;
  • management-operation queueing;
  • network queueing and KV transfer as reported by LMCache or a compatibility adapter;
  • LMCache cache loading/prefetch and confirmation;
  • prefill queueing and execution;
  • decode execution or completion tracking;
  • fusion preparation;
  • fallback selection and execution.
• Add explicit fields for:
  • predicted_* values;
  • actual_* or observed_* values;
  • source and source_endpoint;
  • observation timestamp, freshness/age, and endpoint generation where applicable;
  • start, end, and duration;
  • stage outcome, capability limitation, and error/fallback reason.
• Distinguish authoritative runtime observations from Proxy-local cached/inferred views.
• Use a monotonic clock for durations and wall-clock timestamps only for external correlation.
• Preserve trace context across Proxy, KDN, LMCache adapter, and Instance boundaries where those boundaries exist.
• Provide stable JSON export for debugging and benchmark ingestion.
• Allow tracing to be disabled or sampled without affecting correctness.

Acceptance Criteria

• A single request exports a deterministic, machine-readable stage timeline.
• Predicted, measured, observed, and inferred values cannot be confused in serialization.
• Every cache-state value identifies its collection source and freshness.
• LMCache lookup/status results can be represented without copying LMCache's internal block index into Proxy state.
• Missing, unsupported, stale, and skipped stages are explicit.
• Stage durations are non-negative and use consistent units.
• Trace collection does not alter scheduling, injection, or forwarding decisions.
• Text-only, Legacy KV, LMCache-backed KV, partial/fallback, and unsupported-capability paths are representable.
• Unit tests cover schema serialization, ordering, stale observations, skipped/failed stages, and predicted-versus-observed separation.
• At least one integration test produces an end-to-end timing breakdown consumable by later benchmarks.

Non-Goals

• Implementing ExecutionGraph scheduling.
• Maintaining a Proxy-owned authoritative KV block directory.
• Claiming network-compute overlap improvements in this release.
• Requiring an external tracing backend.

Parent Epic

• v0.1.10 Epic: Contract and Observability Baseline #137

Dependencies

• v0.1.10-2: Define Storage-Neutral Artifact, Replica, DataPlaneTask, and QueueWork State Models #139 for lifecycle and queue-work names.
• v0.1.10-3: Define a Versioned LMCache-Aligned KDN Management Protocol #140 for LMCache-aligned protocol and observation fields.