Skip to content

[ctrace] Align trace-run metadata with specification #2581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jreineckearm merged 5 commits into from
Sep 2, 2026
Merged

[ctrace] Align trace-run metadata with specification #2581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6c4ef3d
[ctrace] Align trace-run metadata with specification
thorstendb-ARM Aug 31, 2026
78a5c4f
Merge branch 'main' into ctrace-spec-updates
thorstendb-ARM Aug 31, 2026
b7fa651
Merge branch 'main' into ctrace-spec-updates
thorstendb-ARM Sep 1, 2026
55ba90e
Merge branch 'main' into ctrace-spec-updates
thorstendb-ARM Sep 1, 2026
4e64639
[ctrace] Test CTF data-type diagnostic
thorstendb-ARM Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 15 additions & 61 deletions tools/ctrace/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,7 @@

`ctrace` converts CMSIS Cortex-M trace captures into outputs that can be inspected or processed by other tools. It
combines a `<solution-set>.ctrace-run.yml` description with matching raw trace files from one trace directory.

The current implementation decodes SWO/ITM and DWT data. Trace Bus (`*.TB.raw`) files are discovered and reported,
but deliberately skipped until that decoder is implemented.
ITM payload output supports stimulus ports `1` through `31`. Port `0` remains part of stream decoding but is
deliberately excluded from CSV and CTF event output. Trace Compass observes the same filtered CTF stream.

![ctrace architecture](docs/architecture.svg)
The [architecture description](docs/architecture.md) documents the supported feature profile and internal design.

## Usage

Expand All @@ -28,19 +22,17 @@ ctrace <trace-dir> [options]
Values for `--type` and `--stream` are space-separated, so place the trace directory before these multi-value options.
If the directory follows them, terminate option parsing explicitly, for example
`ctrace --type itm dwt -- .trace`. With no output option, `ctrace` validates and decodes the capture without writing
output files.
output files. Run `ctrace --help` for the current option details.

The `--type` option accepts the specification-defined selectors `itm`, `dwt`, `event`, `pmu`, `exception`,
`pcsample`, `global_ts`, `overflow`, and `error`. Decoded DWT event counters, PMU packets, and PC samples remain
disabled until their output semantics are implemented, so their selectors currently produce no rows.
## Trace directory

A trace directory uses solution-set-based file names:
Input and output files share a solution-set base name:

```text
.trace/
Board.ctrace-run.yml
Board.SWO.raw
Board.TB.raw # discovered, currently not decoded
Board.TB.raw # optional Trace Bus input
```

For `ctrace .trace --target Board --all`, the supported input produces:
Expand All @@ -54,30 +46,6 @@ For `ctrace .trace --target Board --all`, the supported input produces:
Board.SWO.traceanalysis.xml
```

The YAML reader validates fields consumed by `ctrace` and ignores unrelated or unknown fields. A malformed consumed
field is an error; an unneeded extension to the trace-run format does not break the tool.

## Project structure

| Path | Responsibility |
|---|---|
| `src/CtraceMain.h` | Platform-independent entry point used by the executable trampoline |
| `src/cli` | Command-line parsing and validation |
| `src/control` | Trace-directory orchestration, raw-file access, and per-file decode jobs |
| `src/tracerun` | Trace-run discovery, YAML parsing, and normalized metadata |
| `src/decode` | OpenCSD adapter, recovery, and Cortex-M semantic decoding |
| `src/model` | Backend-independent trace events and selections |
| `src/output/csv` | CSV schema and writer |
| `src/output/ctf` | CTF bundle and Trace Compass XML writers |
| `src/diagnostics` | Structured diagnostics and trace-issue reporting |
| `test/unit` | GoogleTest cases, arranged like the production modules |
| `test/integration` | GoogleTest integration suite for the application entry point and stable fixtures |
| `test/data` | Stable fixtures and expected output |

The module boundaries, dependency direction, runtime flow, and extension points are described in the
[architecture documentation](docs/architecture.md). See also the [verified constraints](docs/constraints.md) and the
compact [remaining TODO list](docs/todo.md).

## Build and test

Initialize all dependencies and configure the repository from its root:
Expand All @@ -97,27 +65,13 @@ ctest --test-dir build -C Debug -R '^(CtraceUnitTests|CtraceIntegTests|ctrace-)'
Editors using `clangd` should open the devtools repository root and configure into `build`. The tool-local
`.clangd` file points clangd at that compilation database.

## Dependencies and releases

`ctrace` uses the devtools copies of `cxxopts`, `yaml-cpp`, GoogleTest, and the `external/OpenCSD` submodule. It does
not carry private copies below `tools/ctrace`.
Known defects in the pinned OpenCSD revision that can affect ctrace are recorded
in the [OpenCSD issue notes](docs/opencsd-issues.md).

Publishing a GitHub Release for a tag named `tools/ctrace/<version>`, for example `tools/ctrace/0.0.1`, runs the ctrace
workflow. It builds Windows AMD64/Arm64, Linux AMD64/Arm64, and macOS Arm64 variants and attaches `ctrace.zip` to the
release. Unit and integration tests run on Windows AMD64 and Linux AMD64; ARM64 targets are compiled but not executed,
matching the other devtools workflows. Coverage runs on Linux AMD64. The executable version is derived from the same
tool-specific tag.

The project is licensed under Apache-2.0. Product dependencies and their licenses are recorded in the repository's
top-level `LICENSE.md`. The release archive contains the project license, third-party notices, OpenCSD copyright
notices, the application-dependency license texts, and checksums for all included files. Compiler and operating-system
runtime contents must still be inspected for each production build; the application dependency list does not claim to
cover them. Release artifacts are currently unsigned; signing, archive checksums, macOS notarization, and a formal
SBOM remain explicit release-hardening decisions.

The code, build, test, and packaging structure is ready for release-candidate validation. The Blinky SWO and TB
captures are approved redistributable test assets, as documented in the [fixture README](test/data/README.md).
Publishing the current statically linked Linux artifacts additionally requires the runtime-license and relinking work
tracked in the [release TODO](docs/todo.md).
## Further documentation

- [Architecture](docs/architecture.md): supported features, runtime flow, module boundaries, dependencies, tests,
and CI.
- [Constraints](docs/constraints.md): contracts that implementation changes must preserve.
- [TODO](docs/todo.md): planned work and pull-request boundaries.
- [OpenCSD issues](docs/opencsd-issues.md): known issues in the pinned decoder revision.
- [Third-party notices](docs/THIRD_PARTY_NOTICES.md): dependency versions, licenses, and build configuration.
- [Test data](test/data/README.md) and [integration tests](test/integration/README.md): fixture provenance and test
scope.
99 changes: 23 additions & 76 deletions tools/ctrace/docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# ctrace Architecture

This document describes the internal structure of `ctrace`, the runtime data flow, and the intended extension points.
For command-line usage, build instructions, and release information, see the [project README](../README.md). The
[verified constraints](constraints.md) record preserved contracts; the compact [TODO list](todo.md) tracks remaining
work.
For command-line usage and build instructions, see the [project README](../README.md). The [verified
constraints](constraints.md) record preserved contracts; the compact [TODO list](todo.md) tracks remaining work.

![ctrace architecture](architecture.svg)

Expand All @@ -19,18 +18,13 @@ packets are retained internally but are not mapped to their selectors yet; perio
Bus input is discovered so that a complete trace directory can be inspected, but `*.TB.raw` files are reported and
skipped until a decoder is implemented.

ITM stream decoding covers the architectural stimulus-port domain `0` through `31`. Public ITM payload output is
restricted to ports `1` through `31`; port `0` is decoded for stream integrity but excluded from CSV and CTF event
output. Trace Compass observes the same filtered CTF stream.

The architecture separates protocol decoding, semantic interpretation, and output generation. This keeps output
formats independent of OpenCSD and allows another raw trace channel to reuse the event model and output backends.

## How it works at a glance

`ctrace` processes one solution set at a time. The common base name joins configuration and trace data; for example,
`Board.ctrace-run.yml` describes the sources and timing metadata required to decode `Board.SWO.raw`. A matching
`Board.TB.raw` is discovered as part of the same solution set but is skipped by the first release profile.
`ctrace` processes one solution set at a time. The [README](../README.md#trace-directory) describes how configuration,
raw input, and generated output files are grouped by their common base name.

The main in-memory path is:

Expand Down Expand Up @@ -68,32 +62,6 @@ The `TraceEvent` boundary is the central design point. Before it, code handles b
recovery, and Cortex-M state. After it, code sees backend-independent events in decode order and does not depend on
OpenCSD types.

| Stage | Owner | Transformation |
| --- | --- | --- |
| Discover | `TraceDirectoryJob` | Trace directory and target selection to solution-set configuration and raw inputs |
| Prepare | `CtraceRunMeta`, `FileDecodeJob` | YAML representation to normalized runtime metadata and an output plan |
| Decode protocol | `DecodePipeline`, `OpenCsdItmDecoder` | Raw byte chunks to recoverable OpenCSD trace elements |
| Interpret | `CortexMStreamDecoder`, `CortexMPostDecoder` | Protocol elements to timestamped semantic events |
| Consume | `DecodeConsumers` | One ordered event stream to diagnostics and every enabled output backend |
| Complete | `TraceOutputLifecycle` | Complete active artifacts, or remove them after output or decoder failure |

## Runtime flow

1. `CtraceMain` parses and validates the command line.
2. `TraceRunDiscovery` finds one or all `<solution-set>.ctrace-run.yml` files in the selected trace directory.
3. `YmlTraceRunConfigReader` reads fields used by `ctrace`; unrelated and unknown YAML fields are ignored.
4. `CtraceRunMeta` normalizes processor, timestamp, route, ITM, and DWT source metadata.
5. `TraceDirectoryJob` associates the configuration with matching raw trace channels.
6. `FileDecodeJob` creates the requested output plan and reads supported raw files in 64 KiB blocks.
7. `DecodePipeline` receives non-owning `RawByteView` values while preserving decoder state across every file-read
boundary.
8. The OpenCSD adapter decodes ITM protocol elements and preserves decoder warnings, errors, and recovery boundaries.
9. The Cortex-M post-decoder converts protocol elements into semantic `TraceEvent` values.
10. `DecodeConsumers` forwards every event to diagnostics and the selected output backends.
11. Output lifecycle handling completes valid artifacts or removes incomplete artifacts after a failure.

Without `--csv`, `--ctf`, or `--all`, the same pipeline runs in validation-only mode without creating output files.

## Processing state and ownership

One `DecodePipeline` is created for each supported raw file. It owns the OpenCSD adapter and Cortex-M stream decoder,
Expand All @@ -105,12 +73,7 @@ buffer; each `RawByteView` borrows that buffer only for the synchronous `DecodeP
into the same `TraceEventSink`, preserving input order while keeping stream-specific timestamp and DWT state apart.

There is no application-wide event queue. `DecodeConsumers` forwards each event synchronously to the output
lifecycle and issue reporter. Output backends own their files and are isolated from one another: failure of one
backend aborts its incomplete artifact but does not directly stop another active backend. A non-recoverable decoder
error aborts every still-active output for that raw file.

The diagnostic sink lives for the complete command invocation. It therefore aggregates failures across solution sets
and determines the final process status after processing has continued wherever possible.
lifecycle and issue reporter.

## Recovery after damaged trace

Expand All @@ -124,8 +87,8 @@ the Cortex-M post-decoder flushes pending events, resets incomplete DWT correlat
until the stream supplies enough timing information again. The issue remains part of the ordered `TraceEvent` stream,
so diagnostics and enabled output backends observe the same recovery boundary.

Failure to reset OpenCSD, repeated lack of decoder progress, or an unsuccessful wait/flush operation aborts only the
current raw-file job. Other solution sets continue to be processed where possible.
Failure to reset OpenCSD, repeated lack of decoder progress, or an unsuccessful wait/flush operation aborts the
current raw-file job.

## Suggested code-reading path

Expand Down Expand Up @@ -165,9 +128,7 @@ must not depend on control jobs or command-line details.
| --- | --- |
| `src/tracerun` | File discovery, YAML parsing, schema subset validation, and normalized metadata |

The YAML reader intentionally consumes only data required by `ctrace`. A malformed consumed field is an error, while
an unknown field is ignored. This permits compatible trace-run format extensions without weakening validation of the
data used for decoding or output generation.
The YAML reader's validation and metadata rules are recorded in the [constraints](constraints.md#boundaries).

`CtraceRunMeta` is the boundary between the YAML representation and runtime processing. Decode and output modules use
normalized metadata instead of navigating YAML nodes.
Expand Down Expand Up @@ -211,27 +172,22 @@ Decoder issue packets remain part of the event stream. `DecodeConsumers` reports
of output filters and forwards all events to the backends. The backends apply stream and type selection internally;
selected issues become CSV error rows or CTF trace-status events. Repeated issues are not silently collapsed.

Processing continues with other solution sets where possible. Errors are rendered as `error` even when their impact
causes a non-zero exit status. Unhandled internal ctrace failures also terminate the command after an error diagnostic.
An invocation-wide diagnostic sink aggregates failures while other solution sets continue where possible, then
determines the final process status. Errors are rendered as `error` even when their impact causes a non-zero exit
status. Unhandled internal ctrace failures also terminate the command after an error diagnostic.

## External dependencies

| Dependency | Use |
| --- | --- |
| `cxxopts` | Command-line parsing |
| `yaml-cpp` | Trace-run YAML parsing |
| `OpenCSD` 1.8.3 | ITM protocol decoding; pinned as a repository submodule |
| GoogleTest | Unit-test framework; not linked into the product executable |
`cxxopts` provides command-line parsing, `yaml-cpp` is confined to the trace-run reader, OpenCSD is isolated behind
the decode adapters, and GoogleTest is used only by test targets. Exact revisions, licenses, and dependency build
configuration are documented in the [third-party notices](THIRD_PARTY_NOTICES.md).

Dependencies are provided by the devtools repository. `tools/ctrace` does not maintain private library copies.
The ITM adapter currently uses OpenCSD `common/` and `interfaces/` headers because the public OpenCSD 1.8.3 boundary
does not provide equivalent access: its installed headers omit the ITM configuration and packet types required by the
decoder callbacks, and its C API exposes only the last structured error rather than all errors from one data-path
operation. Moving the adapter to the public API therefore also requires resolving these two gaps.

The pinned OpenCSD source is built without a downstream source patch. A known unsafe empty-buffer access in an
upstream diagnostic path, its reachability, and a proposed upstream fix are recorded in the
[OpenCSD issue notes](opencsd-issues.md).
A known decoder defect and its proposed upstream fix are recorded in the [OpenCSD issue notes](opencsd-issues.md).

## Extension points

Expand All @@ -256,27 +212,18 @@ Do not introduce backend-specific state into the decode pipeline or event model.
Unit tests under `test/unit/src` mirror the production modules. Shared file, event, and diagnostic helpers live under
`test/unit/support`. One CTest entry runs the complete GoogleTest executable and writes its XML report.

The `CtraceIntegTests` GoogleTest target calls `CtraceMain` through the same `ctracelib` application object used by the
executable. It verifies command-line behavior, diagnostics, output cleanup, and fixture conversion. A small set of
`ctrace-` CTest smoke tests retains coverage of the platform executable and Windows manifest. Test data and expected
artifacts live under `test/data`; generated files are written only below the CMake build directory.

CI runs the unit and integration suites on Windows AMD64 and Linux AMD64. ARM64 targets are compiled but not executed,
matching the other devtools workflows.
Executable-level coverage and fixture ownership are documented next to the
[integration tests](../test/integration/README.md) and [test data](../test/data/README.md).

## Build and release structure
## Build and CI structure

The source tree has seven static library targets: `model`, `cli`, `trace-run`, `diagnostics`, `decode`, `output`, and
`control`. The shared `ctracelib` object contains `CtraceMain`; the executable adds only the platform trampoline and
manifest where required. Dependencies form a directed, cycle-free graph with `control` as the composition root.

The tool-specific GitHub workflow is selected by a `tools/ctrace/<version>` release tag. It builds Windows AMD64 and
Arm64, Linux AMD64 and Arm64, and macOS Arm64 binaries.
The release archive contains the Apache-2.0 project license, application-dependency notices and license texts,
retained OpenCSD copyright notices, and per-file SHA-256 checksums. The version compiled into the executable is derived
from the same tag. The actual compiler and operating-system runtime content still requires inspection for each
production release.

The checked-in SWO and TB captures are approved ctrace test assets and may be redistributed with devtools. Together
with the tool-specific build, test, packaging, versioning, and license integration, this forms the technical basis for
the first open-source release.
Arm64, Linux AMD64 and Arm64, and macOS Arm64 binaries. Unit and integration tests run on Windows AMD64 and Linux
AMD64; the remaining targets are compile-only.
The version compiled into the executable is derived from the same tag. Archive contents and license material are
described in the [third-party notices](THIRD_PARTY_NOTICES.md); unfinished release work remains in the
[TODO list](todo.md).
13 changes: 8 additions & 5 deletions tools/ctrace/docs/constraints.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# ctrace Constraints

This document records contracts that implementation changes must preserve. It intentionally does not describe the
runtime flow, module inventory, supported feature profile, build, or release process; those belong in the
[architecture description](architecture.md), [README](../README.md), and [TODO list](todo.md). The CMSIS-Toolbox
[trace specification](https://github.com/Open-CMSIS-Pack/cmsis-toolbox/blob/main/docs/Experimental-Features.md#file-structure-of-ctrace-runyml)
remains authoritative for the external `*.ctrace-run.yml` format.
This document records contracts that implementation changes must preserve. Runtime design and the supported feature
profile belong in the [architecture description](architecture.md), working instructions in the [README](../README.md),
and unfinished work in the [TODO list](todo.md). The CMSIS-Toolbox
[trace specification](https://open-cmsis-pack.github.io/cmsis-toolbox/Experimental-Features/) remains authoritative
for the external `*.ctrace-run.yml` format.

## Boundaries

Expand All @@ -16,6 +16,9 @@ remains authoritative for the external `*.ctrace-run.yml` format.
- YAML types remain inside the trace-run reader. The rest of ctrace consumes normalized configuration and metadata.
- The YAML reader validates fields consumed by ctrace; unrelated fields are outside its validation scope. Malformed
consumed fields remain errors. An ITM reference without `source` values is valid and contributes no source events.
- DWT data metadata comes from reference-level `address`, `size`, and `data-type`. When reference `size` is absent,
the referenced `ctrace-setup.data.size` supplies it. DWT instruction-control references may bind a processor stream
but do not create decoded data-source routes.
- Backend-specific requirements and failures remain independent; requesting CTF must not disable otherwise valid CSV
output, or vice versa.

Expand Down
Loading
Loading