Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 97 additions & 1 deletion docs/config/fault-manager.rst
Original file line number Diff line number Diff line change
Expand Up @@ -221,12 +221,104 @@ threshold overrides:

When multiple entities report the same ``fault_code``, each event applies the
thresholds resolved from that event's ``source_id``. This means the debounce
behavior follows the reporting entity, not the fault.
behavior follows the reporting entity, not the fault. The debounce counter,
however, belongs to the fault code, so the two entities share one counter under
two policies - see `Per-Fault-Code Thresholds`_ for what that does and how to
settle it. The node warns when it happens.

``auto_confirm_after_sec`` is global-only and cannot be overridden per-entity.
Critical faults skip debounce and confirm on their first occurrence; that is
built in, not a parameter, so it can be neither disabled nor set per entity.

Per-Fault-Code Thresholds
~~~~~~~~~~~~~~~~~~~~~~~~~

Per-entity thresholds are the right tool when you know which subsystems are noisy but not
which codes they will emit. They have one limit: the debounce counter is kept per
``fault_code``, while the override is chosen per ``source_id``. Two entities reporting one
code therefore share a counter and debounce it under two policies, and the report that
happens to arrive decides the transition:

.. code-block:: text

motor (confirmation_threshold=-5) reports OVERHEAT -> counter=-1, PREFAILED
lidar (confirmation_threshold=-1) reports OVERHEAT -> counter=-2, CONFIRMED

The motor's policy was bypassed. Per-fault_code thresholds are the layer that removes
that: they are matched on the code itself, so they resolve the same whoever reports.

.. code-block:: yaml

fault_manager:
ros__parameters:
# Path to YAML file with per-fault_code overrides
fault_thresholds:
config_file: "/etc/ros2_medkit/fault_thresholds.yaml"

The file is a map of fault codes to threshold overrides, the same three fields an entity
override carries:

.. code-block:: yaml

# fault_thresholds.yaml
MOTOR_OVERHEAT:
confirmation_threshold: -5 # five events, whoever reports them
healing_threshold: 10

LIDAR_FAIL:
confirmation_threshold: -1 # instant
healing_threshold: 1

.. list-table::
:header-rows: 1
:widths: 35 15 50

* - Parameter
- Default
- Description
* - ``fault_thresholds.config_file``
- ``""``
- Path to YAML file with per-fault_code threshold overrides. Empty = disabled.

**How the layers combine:**

- Three layers, each applied on top of the last: the **global** defaults, then the
**entity** override whose prefix matches the reporting ``source_id``, then the
**fault code's** own override. A field a layer does not set is left as the layer below
had it, so ``fault_code`` > ``source_id`` > global for every field independently.
- Matching on the code is **exact**. A fault code is an identifier, not a path: an entry
for ``MOTOR`` does not capture ``MOTOR_OVERHEAT`` the way ``/sensors`` captures
``/sensors/lidar``.
- A code that is not listed resolves exactly as it did before: entity override if one
matches, global otherwise. The layer is opt-in and changes nothing on its own.
- Like the entity file, this one is loaded once at node startup. Changes require a restart.

.. note::

An override that names only some fields settles only those. If ``MOTOR_OVERHEAT`` pins
``confirmation_threshold`` but not ``healing_threshold``, the healing direction still
follows whichever entity reported, and the node still reports the conflict below. Pin
all three fields to settle a code completely.

**When one code is debounced two ways**

The node resolves the policy for every report, and warns the first time two sources resolve
different policies for one code:

.. code-block:: text

[WARN] Fault code 'OVERHEAT' is debounced two ways: '/powertrain/motor/left' resolves
confirmation=-5 healing_enabled=false healing=10, '/sensors/lidar/front' resolves
confirmation=-1 healing_enabled=true healing=1. The debounce counter belongs to the
fault code, so whichever source reports decides the transition and the other policy is
bypassed. Give the code an entry in fault_thresholds.config_file to settle it.

The warning names both sources and both resolved policies, and is emitted **once per fault
code** for the life of the node, so a busy reporter does not turn it into a log storm. It
is a diagnostic, not an error: the configuration is legal and the fault is still debounced,
just not under a policy an operator chose. Giving the code an entry in
``fault_thresholds.config_file`` that pins all three fields ends it.

Snapshot Configuration
----------------------

Expand Down Expand Up @@ -637,6 +729,10 @@ Complete Example
entity_thresholds:
config_file: "/etc/ros2_medkit/entity_thresholds.yaml"

# Per-fault_code debounce overrides (applied on top of the entity ones)
fault_thresholds:
config_file: "/etc/ros2_medkit/fault_thresholds.yaml"

# Snapshots
snapshots:
enabled: true
Expand Down
22 changes: 22 additions & 0 deletions docs/requirements/specs/faults.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,28 @@ Faults
shall take precedence over global defaults. Unspecified fields shall inherit from global
configuration. When no entity prefix matches, global defaults shall apply.

.. req:: Per-Fault-Code Debounce Thresholds
:id: REQ_INTEROP_107
:status: verified
:tags: Faults

The fault manager shall support per-fault_code debounce threshold configuration using
exact matching on the reported fault code. A fault-code override for
``confirmation_threshold``, ``healing_enabled``, or ``healing_threshold`` shall take
precedence over both the per-entity override resolved for the reporting source and the
global defaults. Unspecified fields shall inherit from the layer below. When a fault code
has no override, the resolved per-entity or global configuration shall apply unchanged.

.. req:: Conflicting Debounce Policies Are Reported
:id: REQ_INTEROP_108
:status: verified
:tags: Faults

The debounce counter belongs to the fault code while a per-entity override is resolved
from the reporting source. When two sources report one fault code and resolve to
different debounce policies, the fault manager shall warn, naming the fault code, both
sources and both resolved policies, at most once per fault code for the life of the node.

.. req:: Fault Snapshot and Rosbag Capture
:id: REQ_INTEROP_088
:status: verified
Expand Down
10 changes: 9 additions & 1 deletion src/ros2_medkit_fault_manager/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ add_library(fault_manager_lib STATIC
src/correlation/config_parser.cpp
src/correlation/pattern_matcher.cpp
src/correlation/correlation_engine.cpp
src/entity_threshold_resolver.cpp
src/threshold_resolver.cpp
)

target_include_directories(fault_manager_lib PUBLIC
Expand Down Expand Up @@ -173,6 +173,11 @@ if(BUILD_TESTING)
target_link_libraries(test_entity_thresholds fault_manager_lib)
medkit_target_dependencies(test_entity_thresholds rclcpp ros2_medkit_msgs)

# Fault-code threshold resolver tests
medkit_add_gtest(test_fault_code_thresholds test/test_fault_code_thresholds.cpp)
target_link_libraries(test_fault_code_thresholds fault_manager_lib)
medkit_target_dependencies(test_fault_code_thresholds rclcpp ros2_medkit_msgs)

# Integration tests
install(DIRECTORY test
DESTINATION share/${PROJECT_NAME}
Expand All @@ -192,6 +197,9 @@ if(BUILD_TESTING)
medkit_add_launch_test(test_entity_thresholds_integration test/integration/test_entity_thresholds_integration.test.py
TIMEOUT 60 LABELS "integration")

medkit_add_launch_test(test_fault_code_thresholds_integration
test/integration/test_fault_code_thresholds_integration.test.py TIMEOUT 60 LABELS "integration")

# Drives healing with the event counts a one-event-per-transition reporter
# actually sends: one FAILED per raise, one PASSED per clear. Parametrized over
# healing_threshold, so the node launches twice, and one case holds a settled
Expand Down
8 changes: 6 additions & 2 deletions src/ros2_medkit_fault_manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
when a cleared fault is raised again - and tracks all reporting sources
- **Severity escalation**: Fault severity is updated if a higher severity is reported
- **Persistent storage**: SQLite backend ensures faults survive node restarts
- **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
- **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless `snapshots.retain_on_clear` is set)
- **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
- **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below)
Expand All @@ -69,6 +69,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
| `healing_threshold` | int | `3` | Counter value at which faults are healed |
| `auto_confirm_after_sec` | double | `0.0` | Auto-confirm PREFAILED faults after timeout (0 = disabled) |
| `entity_thresholds.config_file` | string | `""` | Path to YAML file with per-entity debounce threshold overrides |
| `fault_thresholds.config_file` | string | `""` | Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
| `near_miss.max_per_fault` | int | `200` | Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |

### Snapshot Parameters
Expand Down Expand Up @@ -159,7 +160,10 @@ rows written before the field existed.

With per-entity thresholds the recorded `confirmation_threshold` is the one belonging to the
**reporting source**, while the debounce counter is shared by every source of that fault code. It
is therefore not by itself the distance to confirmation for the fault as a whole.
is therefore not by itself the distance to confirmation for the fault as a whole. Giving the code
an entry in `fault_thresholds.config_file` makes the two agree again: a fault-code override
resolves the same for every source, so the recorded threshold is the fault's own. The node warns
once per code when two sources resolve different policies for it.

Entries are kept and evicted in **arrival order**, not by their timestamps. Reporters carry their
own clocks, so a report can arrive carrying a timestamp behind one already stored; ordering the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@
#include "rclcpp/rclcpp.hpp"
#include "ros2_medkit_fault_manager/capture_thread_pool.hpp"
#include "ros2_medkit_fault_manager/correlation/correlation_engine.hpp"
#include "ros2_medkit_fault_manager/entity_threshold_resolver.hpp"
#include "ros2_medkit_fault_manager/fault_audit_log.hpp"
#include "ros2_medkit_fault_manager/fault_storage.hpp"
#include "ros2_medkit_fault_manager/rosbag_capture.hpp"
#include "ros2_medkit_fault_manager/snapshot_capture.hpp"
#include "ros2_medkit_fault_manager/threshold_resolver.hpp"
#include "ros2_medkit_msgs/msg/fault_event.hpp"
#include "ros2_medkit_msgs/srv/clear_fault.hpp"
#include "ros2_medkit_msgs/srv/get_fault.hpp"
Expand Down Expand Up @@ -187,9 +187,21 @@ class FaultManagerNode : public rclcpp::Node {
/// Extract topic name from full topic path (last segment)
static std::string extract_topic_name(const std::string & topic_path);

/// Resolve debounce config for a given source_id using entity threshold resolver.
/// Falls back to global config if no entity-specific overrides match.
DebounceConfig resolve_config(const std::string & source_id) const;
/// Resolve the debounce config a report is debounced under.
/// Three layers, each applied on top of the last: the global config, the
/// longest-prefix entity override matching @p source_id, and the exact-match
/// override for @p fault_code. A layer only sets the fields it configures.
DebounceConfig resolve_config(const std::string & source_id, const std::string & fault_code) const;

/// Warn once per fault code when two sources debounce it under different
/// policies. The counter belongs to the fault code and the entity override to
/// the source, so the report that arrives decides the transition and the other
/// source's policy is silently bypassed (issue #276).
/// @param fault_code The code just reported.
/// @param source_id The source that reported it.
/// @param resolved The config that report resolved to.
void warn_on_conflicting_debounce_policy(const std::string & fault_code, const std::string & source_id,
const DebounceConfig & resolved);

/// Create the tamper-evident audit log from parameters (nullptr if disabled).
std::unique_ptr<FaultAuditLog> create_audit_log();
Expand All @@ -215,7 +227,21 @@ class FaultManagerNode : public rclcpp::Node {
QueueFullPolicy capture_queue_full_policy_{QueueFullPolicy::kRejectNewest};
DebounceConfig global_config_; ///< Global debounce config (built from ROS params)
std::unique_ptr<FaultStorage> storage_;
std::unique_ptr<EntityThresholdResolver> threshold_resolver_; ///< Per-entity threshold overrides
std::unique_ptr<EntityThresholdResolver> threshold_resolver_; ///< Per-entity threshold overrides
std::unique_ptr<FaultCodeThresholdResolver> fault_code_resolver_; ///< Per-fault_code threshold overrides

/// The first debounce policy seen for a fault code, and who reported it.
/// Kept only to notice a second source resolving a different policy for the
/// same code, and warned about once. One entry per fault code the node has
/// seen, so it is bounded by the same thing the fault store is. Written from
/// the ReportFault callback, which the node's single-threaded executor
/// serialises with every other callback that touches node state.
struct DebouncePolicyWitness {
DebounceConfig config; ///< The policy the first report resolved to
std::string source_id; ///< The source that reported it
bool warned{false}; ///< Whether the conflict has already been reported
};
std::unordered_map<std::string, DebouncePolicyWitness> debounce_policy_witness_;

/// Tamper-evident audit log of fault transitions (nullptr when disabled).
std::unique_ptr<FaultAuditLog> audit_log_;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

#include <optional>
#include <string>
#include <unordered_map>
#include <utility>
#include <vector>

Expand All @@ -31,6 +32,15 @@ struct EntityDebounceOverride {
std::optional<int32_t> healing_threshold;
};

/// Per-fault_code debounce overrides. Unset fields inherit from the layer below:
/// the entity override if one matched, otherwise the global DebounceConfig.
struct FaultCodeDebounceOverride {
std::string fault_code; ///< Exact fault code (e.g. "MOTOR_OVERHEAT")
std::optional<int32_t> confirmation_threshold;
std::optional<bool> healing_enabled;
std::optional<int32_t> healing_threshold;
};

/// Resolves per-entity debounce thresholds using longest-prefix matching.
///
/// Given a source_id (entity FQN like "/powertrain/motor_left") and the global
Expand Down Expand Up @@ -62,4 +72,44 @@ class EntityThresholdResolver {
std::vector<EntityDebounceOverride> entries_;
};

/// Resolves per-fault_code debounce thresholds by exact match on the fault code.
///
/// The debounce counter lives on the fault code while an entity override is
/// selected by the reporting source, so two entities reporting one code debounce
/// it under two policies. A fault-code override is the layer that removes that:
/// it resolves the same whoever reports, and is applied on top of whatever the
/// entity layer produced.
class FaultCodeThresholdResolver {
public:
FaultCodeThresholdResolver() = default;

/// Construct with a list of fault-code overrides. A code repeated in the list
/// keeps its first entry; YAML loading cannot produce one, a caller can.
explicit FaultCodeThresholdResolver(std::vector<FaultCodeDebounceOverride> entries);

/// Resolve the effective DebounceConfig for a fault code.
/// `base` is what the layers below already produced (the global config, then
/// any entity override). Fields the code does not set are left as `base` has
/// them. A code with no override returns `base` unchanged.
DebounceConfig resolve(const std::string & fault_code, const DebounceConfig & base) const;

/// Number of configured fault-code entries.
size_t size() const;

/// Load fault-code threshold overrides from a YAML file.
/// Returns an empty vector on parse error (logs warning via rcutils).
/// YAML format: map of fault code -> {confirmation_threshold, healing_enabled, healing_threshold}
static std::vector<FaultCodeDebounceOverride> load_from_yaml(const std::string & path);

private:
std::unordered_map<std::string, FaultCodeDebounceOverride> entries_;
};

/// Whether two configs debounce a fault the same way.
///
/// Compares only the three fields an override can carry. `auto_confirm_after_sec`
/// is global-only, so it is the same for every source by construction and would
/// only add noise to the comparison.
bool debounce_policy_equal(const DebounceConfig & a, const DebounceConfig & b);

} // namespace ros2_medkit_fault_manager
Loading
Loading