Skip to content

fault_manager: debounce a fault code by the code, not only by who reported it - #690

Open
ahmedsleem109 wants to merge 1 commit into
selfpatch:mainfrom
ahmedsleem109:feat/per-fault-code-debounce-thresholds
Open

ahmedsleem109 wants to merge 1 commit into
selfpatch:mainfrom
ahmedsleem109:feat/per-fault-code-debounce-thresholds

Conversation

@ahmedsleem109

@ahmedsleem109 ahmedsleem109 commented Sep 15, 2026

Copy link
Copy Markdown

What this is

A debounce counter belongs to the fault code. A per-entity override is picked by the
reporting source. So when two entities report one code they share one counter under two
policies, and whichever report happens to arrive decides the transition - the other
entity's policy is bypassed, silently. That is #275, and the bypass being invisible is
#276. They are one mechanism seen from two sides, so they are fixed together here.

The TODO(#276) that sat on the resolve site in handle_report_fault is gone.

The layer (#275)

fault_thresholds.config_file takes a map of fault code to the same three fields an
entity override carries:

# fault_thresholds.yaml
MOTOR_OVERHEAT:
  confirmation_threshold: -5    # five events, whoever reports them
  healing_threshold: 10

LIDAR_FAIL:
  confirmation_threshold: -1
  healing_threshold: 1

Three layers, each applied on top of the last - global, then the entity override matching
source_id, then the fault code's own - so the priority the issue asks for,
fault_code > source_id > global, holds field by field: a code that pins only
confirmation_threshold still takes its healing fields from whichever entity matched.

Matching on the code is exact. A fault code is an identifier, not a path, so an entry for
MOTOR must not capture MOTOR_OVERHEAT the way /sensors captures /sensors/lidar;
there is a test pinning that.

A code that is not listed resolves exactly as it did before. The layer is opt-in and
changes nothing on its own - the existing per-entity suite passes untouched.

A separate file rather than a section in the entity file. The entity file's top level
is a map of entity prefixes, so a fault_thresholds: key in it would be ambiguous with an
entity literally named that, and would silently change meaning for anyone who already has
one. The issue allowed either; this way nothing existing can be reinterpreted.

The warning (#276)

Where no override settles a code, the node says so rather than let the bypass stay
invisible:

[WARN] Fault code 'OVERHEAT' is debounced two ways: '/powertrain/motor/left' resolves
confirmation=-5 healing_enabled=false healing=10, '/sensors/lidar/front' resolves
confirmation=-1 healing_enabled=true healing=1. The debounce counter belongs to the fault
code, so whichever source reports decides the transition and the other policy is bypassed.
Give the code an entry in fault_thresholds.config_file to settle it for every source.

It names both sources, both resolved policies and the remedy, because a warning that only
said "conflict" would leave the operator to reconstruct which two configs met.

Once per fault code, not throttled: the node keeps the first policy resolved for a
code and the source that produced it, and warns the first time a different one arrives. A
warning per report is a warning per fault event on a busy robot; a throttle would repeat
forever for a configuration that is not going to change until someone restarts the node.
The witness is one small entry per fault code seen, the same cardinality as the fault store
itself.

It is computed from the resolved policies, not from the presence of an entry, which
matters for a partial override: a code that pins confirmation_threshold but not the
healing fields is only half settled, and the warning still fires on what still differs.
That was not a design I anticipated - the integration test caught it, and it is now a case
of its own (test_04_a_half_pinned_code_is_still_reported) rather than something papered
over.

Comparison ignores auto_confirm_after_sec, which is global-only and therefore identical
for every source by construction.

One file rename

entity_threshold_resolver.{hpp,cpp} become threshold_resolver.{hpp,cpp}. Both resolvers
live there now and share the YAML field parsing and the sign correction (a
confirmation_threshold written positive, a healing_threshold written negative), so the
two loaders cannot drift apart. EntityThresholdResolver itself is unchanged - same class,
same behaviour, same tests.

Tests

  • test/test_fault_code_thresholds.cpp - 21 cases: resolution, exact-not-prefix matching,
    the three-layer merge, debounce_policy_equal, YAML loading (valid, missing, malformed,
    non-map root, non-map entry, sign correction), and the storage-level proof that lidar no
    longer confirms the motor's code early.
  • test/integration/test_fault_code_thresholds_integration.test.py - the node with both
    layers configured: a code pinned over its entity, one code debouncing alike from two
    entities, an unlisted code keeping entity behaviour, the half-pinned case, the warning
    firing exactly once, and - after shutdown, when the whole output is in hand - no warning
    for a settled code or for any code only one source reports. That last absence is the
    regression guard: a witness compared against the global config instead of against the
    first report would warn on every ordinary single-reporter robot.
  • REQ_INTEROP_107 and REQ_INTEROP_108 added to docs/requirements/specs/faults.rst,
    both traced by @verifies tags through generate_verification.py.

Verified locally

ROS 2 Jazzy / Ubuntu 24.04, colcon test on ros2_medkit_fault_manager: 32/32 pass,
linters included (clang_format, flake8, pep257, copyright, cppcheck, lint_cmake, xmllint).
clang-tidy run on both changed sources - no finding in the new code. Whole workspace built
(--packages-skip ros2_medkit_opcua) to confirm the header rename breaks no dependent
package.

Closes #275
Closes #276

…ho reported it

A debounce counter belongs to the fault code, while a per-entity override is
picked by the reporting source. Two entities reporting one code therefore share
one counter under two policies, and whichever report arrives decides the
transition - the other entity's policy is bypassed without a trace.

Add the layer that settles it. `fault_thresholds.config_file` takes a map of
fault code to the same three fields an entity override carries, matched exactly
on the code, and applied on top of whatever the entity layer produced: fault
code > source_id > global, field by field. A code that is not listed resolves
exactly as before, so the layer is opt-in and changes nothing on its own.

Where no override settles a code, say so rather than let the bypass stay
invisible. The node keeps the first policy resolved for each code and warns once
per code when a later source resolves a different one, naming both sources, both
policies and the remedy. Once per code, because a warning per report is a
warning per fault event on a busy robot. An override that names only some fields
settles only those, and the warning still fires on what still differs - it is
computed from the resolved policies, not from the presence of an entry.

entity_threshold_resolver.{hpp,cpp} become threshold_resolver.{hpp,cpp}: both
resolvers now live there, sharing the YAML field parsing and the sign
correction, so a change to one cannot drift from the other.

Closes selfpatch#275
Closes selfpatch#276
@ahmedsleem109
ahmedsleem109 force-pushed the feat/per-fault-code-debounce-thresholds branch from b48a005 to 8fe77fa Compare September 15, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: warn when different entity configs hit the same fault_code feat: per-fault_code debounce threshold configuration

1 participant