scripts: commit the show-spans generator for the telemetry demo - #558
Open
brentrager wants to merge 1 commit into
Open
brentrager wants to merge 1 commit into
brentrager wants to merge 1 commit into
Conversation
`.github/demo-telemetry.gif` is captioned "Not a mockup" and was generated
by an ad-hoc `show-spans.sh` that was never committed. By the time the GIF
needed regenerating for a model change, the script was gone — not in the
repo, not in any session transcript, not in git history. A reproducible
asset that nobody can reproduce is just an asset.
So this reconstructs it and commits it. One real turn, real spans, and
`gen_ai.request.model` is whatever `SMOOTH_AGENT_MODEL` says — which is
how the demo gets re-shot for a new model instead of retouched.
Three things hand you an EMPTY capture, and each looks like broken
telemetry rather than a harness mistake. All three are now handled by the
script and written down in scripts/README-show-spans.md:
1. RUST_LOG below info. `gen_ai.chat` and `gen_ai.tool` are info-level
SPANS — a `warn` filter does not hide them, it stops them being
created. This one cost the most time: the turn ran perfectly, the
collector was healthy, and the dump was empty.
2. Server started before the collector. The OTLP exporter connects at
boot, so it exports into a closed port and never retries.
3. Killing the server when the turn resolves. The exporter batches, so
the last batch dies with it.
A collector is required and that is not incidental: the fmt layer prints
tracing EVENTS, and spans only materialise on export — a span tree can
only be read off an OTLP consumer. scripts/otel/collector.yaml is a
throwaway one whose only job is to print what it receives. Note its
`service.telemetry.logs.level` must stay at info; the debug exporter
writes THROUGH that logger, so `warn` silences the very spans it exists
to show.
drive-one-turn.mjs records two protocol details the schemas do not:
`sessionId` comes back nested under `data`, and `requestId` is documented
optional on send_message but the server rejects the frame without one.
format-spans.mjs invents nothing — every attribute printed is read from
the collector dump, and the "parked for approval" note only appears when
the run actually parked (PARKED=1, set by the script because it sets
SMOOTH_AGENT_CONFIRM_TOOLS).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.github/demo-telemetry.gifis captioned "Not a mockup" and was generated by an ad-hocshow-spans.shthat was never committed. By the time it needed regenerating for a model change, the script was gone — not in the repo, not in git history, not in any session transcript. A reproducible asset nobody can reproduce is just an asset.This reconstructs and commits it. One real turn, real spans, and
gen_ai.request.modelis whateverSMOOTH_AGENT_MODELsays — which is how the demo gets re-shot for a new model instead of retouched.(cd rust && cargo build -p smooai-smooth-operator-server) SMOOAI_GATEWAY_KEY=sk-… SMOOTH_AGENT_MODEL=gpt-5.6-luna scripts/show-spans.shThree ways to get an empty capture
Each looks like broken telemetry rather than a harness mistake. All three are handled by the script and written down in
scripts/README-show-spans.md:RUST_LOGbelowinfofor the operator crates.gen_ai.chatandgen_ai.toolare info-level spans — awarnfilter doesn't hide them, it stops them being created. This one cost the most time: the turn ran perfectly, the collector was healthy, and the dump was empty.Why a collector is required
The
fmtlayer prints tracing events; spans only materialise on export. So a span tree can only be read off an OTLP consumer — which is whatotelcol:4317in the demo is.scripts/otel/collector.yamlis a throwaway one whose only job is to print what it receives.Its
service.telemetry.logs.levelmust stay atinfo: thedebugexporter writes through that logger, sowarnsilences the very spans it exists to show.Two protocol details the schemas don't state
Both cost a debugging round and are now recorded in
drive-one-turn.mjs:sessionIdcomes back nested underdata, not at the top level.requestIdis listed optional onsend_message, but the server rejects the frame without one (VALIDATION_ERROR).Honesty
format-spans.mjsinvents nothing — every attribute printed is read from the collector dump, and the "parked for approval" note only appears when the run actually parked (PARKED=1, which the script sets because it setsSMOOTH_AGENT_CONFIRM_TOOLS).Verification
Clean end-to-end run against
llm.smoo.ai: turn parked atknowledge_search, approved, answered with citations, 45 tokens streamed. Collector receivedgen_ai.chatwithgen_ai.request.model: Str(gpt-5.6-luna)and itsgen_ai.toolchild.🤖 Generated with Claude Code