From 2e583f0b617739dc3f2b3fa281d415c14dd673d4 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sun, 6 Sep 2026 12:43:01 +0200 Subject: [PATCH 01/25] fix(opcua): rescan while disconnected and heal PLC_COMMS_LOST on connect The plugin decided two things once, at startup, and never revised them while it ran. Both now recover after a bad start. Rescan while no session is up. Config-less discovery ran one scan a few seconds after start. A gateway that boots together with its PLC scans while the PLC is still coming up. It finds nothing, falls back to opc.tcp://localhost:4840 and retries that endpoint until someone restarts it. The reconnect arm now asks the plugin for a fresh scan. It adopts a newly found server for its next connect attempt and resets the backoff, so the new endpoint is tried at once. The rules that make discovery safe are unchanged. An explicitly configured endpoint_url still wins and is never rescanned. The scan stays a bounded read-only TCP sweep plus GetEndpoints. Nothing is scanned while a session is up. Cadence. discovery.interval_s sets the rescan cadence, 30 s by default. The cadence is stamped when a sweep ends, so a sweep of a /16, which takes minutes at the defaults, does not make the next one due at once. While discovery rescans, the reconnect backoff is capped at the cadence, so the real cadence is interval_s. An unset interval_s takes the default. An explicit 0 keeps discovery on with a one-shot start-up scan. A negative value is refused with a warning. The warning and the start-up line describe the cadence that applies. NetworkDiscovery::run() takes a cancel predicate, bound to the shutdown flag. It is checked before each probe and between the sweep and the identify phase, so shutdown() does not wait out a sweep. Discovery report. A pass whose outcome matches the previous pass reports at DEBUG. The first pass and every changed outcome report at INFO or WARN. A site with a secured-only server otherwise logs the same WARN every 30 s. Component identity. With no node map, the SOVD component is named from the device. When the start-up connect fails there is no device to ask, so the name comes from the fallback endpoint and an empty DeviceInfo. The identity is now derived again on the first poll of a new session, in config-less mode only. An explicit node map still owns the name. The derived alarms entity follows the rename, and the rename is logged at INFO. Clear PLC_COMMS_LOST on every successful connect. The fault manager keys faults by fault code and persists them. A fault raised before a gateway restart stays in the store, and the new process has no memory of raising it. The initial connect and every reconnect now send the clear, whatever this process raised. The clear is fire and forget, so a clear for a fault that is not there costs nothing. The debounce that governs raising is unchanged. Both clears set skip_correlation_auto_clear. They report the link state, so they must not cascade-clear the symptom faults that a rule attributes to PLC_COMMS_LOST. Pending buffer. While the fault manager is unmatched, the buffer keeps at most one pending clear per fault code. When it is full, it evicts a clear before a report. A clear that meets a buffer full of reports is refused, and the reports stay. A flapping link otherwise pushes real alarm reports out of the bounded buffer. Tests. Unit tests drive the discovery pass and the endpoint adoption rule with injected probes. They include the positive control that a configured endpoint refuses the server an unconfigured one accepts. The comms-lost heal runs against the live test server, because only a connect that succeeds reaches that arm. A docker scenario starts the gateway first, with discovery on and no endpoint configured. It asserts that the gateway settled on the fallback endpoint with no session. It then starts an OPC UA server on the same subnet and asserts that the endpoint is adopted within two rescan intervals. It asserts that the container never restarted, because a restart would pass the endpoint check. A config-less pass asserts the rename against a real server. The network has an explicit /24, so the sweep covers 254 hosts and ends in seconds. The README documents the rescan, the cadence rules and the scenario. --- .../ros2_medkit_opcua/README.md | 69 ++- .../docker/scripts/run_discovery_race_test.sh | 321 +++++++++++ .../ros2_medkit_opcua/network_discovery.hpp | 26 +- .../ros2_medkit_opcua/opcua_plugin.hpp | 203 ++++++- .../ros2_medkit_opcua/opcua_poller.hpp | 44 +- .../src/network_discovery.cpp | 33 +- .../ros2_medkit_opcua/src/opcua_plugin.cpp | 439 +++++++++++++-- .../ros2_medkit_opcua/src/opcua_poller.cpp | 56 +- .../test/test_network_discovery.cpp | 116 +++- .../test/test_opcua_identity.cpp | 82 ++- .../test/test_opcua_plugin.cpp | 516 ++++++++++++++++++ 11 files changed, 1807 insertions(+), 98 deletions(-) create mode 100755 src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 748d8da48..cb251c09c 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -656,7 +656,7 @@ ros2_medkit_gateway: | `subscription_interval_ms` | `500` | Publishing interval for OPC-UA subscriptions when `prefer_subscriptions: true` | | `condition_replay_strategy` | `auto` | Active-condition replay on reconnect: `method`, `read`, `auto`, `off` (see below) | | `require_confirm_for_clear` | `true` | Require both Acknowledge AND Confirm before a native alarm auto-clears. Set `false` for Confirm-less servers (e.g. Siemens S7-1500) so alarms clear on Acknowledge alone (see below) | -| `comms_lost_fault_enabled` | `true` | Raise a component-scoped `PLC_COMMS_LOST` fault when the connection stays down (issue #496) | +| `comms_lost_fault_enabled` | `true` | Raise a component-scoped `PLC_COMMS_LOST` fault when the connection stays down, and clear it on every successful connect (issue #496) | | `comms_lost_debounce_ms` | `5000` | Continuous down time before `PLC_COMMS_LOST` is raised (debounces reconnect blips; clamped to [0, 3600000] ms) | | `comms_lost_severity` | `ERROR` | SOVD severity bucket for the `PLC_COMMS_LOST` fault | | `discovery.enabled` | `false` | Opt-in read-only PLC network discovery (auto endpoint). See below | @@ -712,12 +712,18 @@ plugins.opcua.discovery: connect_timeout_ms: 600 # per-port TCP connect timeout scan_concurrency: 100 # bounded, polite concurrent connect count identify_timeout_ms: 6000 # per GetEndpoints identify - interval_s: 0 # 0 = one-shot at startup (periodic re-scan: TODO) + # re-scan cadence while disconnected. Omit the key for the built-in 30 s; + # set it to 0 to keep discovery on but never re-scan (start-up scan only). + interval_s: 30 anonymous_none_only: true # only auto-connect None/Anonymous servers ``` Environment overrides (Docker / appliance): `OPCUA_DISCOVERY_ENABLED`, `OPCUA_DISCOVERY_SUBNETS` (comma-separated CIDRs), `OPCUA_DISCOVERY_INTERVAL_S`. +Leaving `interval_s` (and `OPCUA_DISCOVERY_INTERVAL_S`) unset means "no cadence +stated" and takes the 30 s default; an explicit `0` is honoured as written and +turns the recurring sweep off. A negative value is refused with a warning and +leaves the cadence unset. How it works: 1. Bounded concurrent TCP connect sweep of the configured ports across the @@ -733,24 +739,49 @@ How it works: auto-selects the best None/Anonymous data server (deterministic, lowest ip:port) and connects to the **scanned ip:port** - not the advertised EndpointUrl, which a server may report as a non-resolvable hostname. +5. While no session is established, the reconnect loop scans again every + `interval_s` (default 30 s), measured from the END of the previous sweep, and + adopts a newly found server for its next connect attempt, logging the swap at + INFO. The re-scan is consulted once per reconnect attempt, and those are + spaced by an exponential backoff, so the backoff ceiling is capped at + `interval_s` while discovery is re-scanning - otherwise the real cadence + would be `max(interval_s, backoff)` rather than the stated one. This covers + the common race where the gateway and the PLC boot together: the startup scan + finds nothing because the PLC is still coming up, and without a re-scan the + plugin would retry the fallback endpoint until someone restarted it. +6. On the first session after such an adoption, a config-less deployment (no + node map) re-derives the SOVD component identity from the device itself, so + the component stops being served under the provisional `opcua-` name it + got when nothing answered. The change is logged at INFO. + +Re-scanning stops as soon as a session is up, and never starts at all when an +`endpoint_url` is configured. Safety / OT posture: - Everything is read-only: TCP connect + `GetEndpoints` only. No writes, no subscriptions, no second long-lived session. +- The scan is NOT one-shot: while the plugin has no session it repeats every + `interval_s` (default 30 s) for as long as it stays disconnected. Set + `interval_s: 0` (or `OPCUA_DISCOVERY_INTERVAL_S=0`) to keep discovery on with + the start-up scan only, or `enabled: false` to switch it off entirely. +- A sweep is cancelled when the plugin shuts down, so a stop does not have to + wait out a subnet the size of a /16. - An explicitly configured `endpoint_url` (or `OPCUA_ENDPOINT_URL`) always wins; discovery then does nothing, so it never opens a second session on a PLC the plugin already polls. -- Secured-only servers (no None/Anonymous endpoint) are surfaced in the startup - log as leads requiring operator credentials - never auto-connected or probed. +- Secured-only servers (no None/Anonymous endpoint) are surfaced in the log as + leads requiring operator credentials - never auto-connected or probed. A + re-scan whose outcome has not changed reports at DEBUG instead of repeating + the whole report, so a recurring sweep does not bury the rest of the log. - The scan is bounded (short connect timeout, capped concurrency) and CIDRs wider than /16 are rejected to prevent an accidental broad sweep. Note on passive discovery: a stock Siemens S7-1500 neither multicast-announces (mDNS `_opcua-tcp._tcp`) nor registers with an OPC-UA LDS, so passive sources find nothing there; the active scan is what discovers it. Passive mDNS / LDS -`FindServers` sources (useful on Kepware / Prosys / GDS estates) and periodic -re-scan + multi-endpoint registration are planned follow-ups; this iteration -delivers the active-scan core and single "auto endpoint" mode. +`FindServers` sources (useful on Kepware / Prosys / GDS estates) and +multi-endpoint registration are planned follow-ups. This iteration delivers the +active-scan core and a single "auto endpoint" mode. ### Active-condition replay on reconnect (issue #389/#478) @@ -802,6 +833,20 @@ so the alarm clears on `Acknowledge` alone. The default (`true`) is unchanged and spec-strict; the relaxed path still requires acknowledgement and needs real-S7-1500 validation. +### Connection loss and `PLC_COMMS_LOST` (issue #496) + +When the OPC-UA connection stays down for `comms_lost_debounce_ms` continuously, +the plugin raises one component-scoped `PLC_COMMS_LOST` fault (a shorter blip +during a normal reconnect does not flap it). + +The fault is cleared on **every** successful connect, both the initial one and +every later reconnect, whether or not this process was the one that raised it. +The fault manager keys faults by fault code and persists them, so a fault raised +before a gateway restart is still standing while the new process has no memory +of it. Clearing only what the running process remembered left exactly that fault +CONFIRMED for good. The clear is fire-and-forget, so a clear for a fault that is +not there is harmless. + Node map entries also support an optional `ros2_topic` field to override the auto-generated ROS 2 topic name for the PLC value bridge: ```yaml @@ -953,6 +998,16 @@ MEDKIT_OPCUA_VARIANT=write-capable bash scripts/test_all.sh bash scripts/stop.sh ``` +A separate scenario covers the config-less discovery start-up race, which the +suite above cannot see because it pins `OPCUA_ENDPOINT_URL` and so +short-circuits discovery. It starts the gateway before any server, with +discovery on and no endpoint configured, then brings a server up and asserts +the gateway adopts it without a restart: + +```bash +bash src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh +``` + ### Test Coverage | Category | read-only | write-capable | What it validates | diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh new file mode 100755 index 000000000..dc52d3823 --- /dev/null +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh @@ -0,0 +1,321 @@ +#!/usr/bin/env bash +# Copyright 2026 mfaferek93 +# +# Integration test for the config-less discovery start-up race. +# +# The field failure this reproduces: the gateway and the PLC power on +# together, the gateway's start-up scan runs while the PLC is still booting +# and finds nothing, and without a re-scan the plugin retries its fallback +# endpoint for as long as it runs. Only a restart ever found the PLC. +# +# So this scenario starts the gateway FIRST, with discovery on and no +# OPCUA_ENDPOINT_URL, asserts it settled on the fallback endpoint with no +# session, then starts an OPC-UA server and asserts the gateway adopts it +# within two re-scan intervals without being restarted. +# +# It then repeats the race with NO node map at all (the config-less deployment). +# There the component identity is derived from the device, and a gateway that +# scanned before its PLC existed can only name it after the fallback endpoint - +# so the second pass asserts the SOVD component stops being served under that +# provisional name once the PLC is adopted. +# +# Every other opcua docker scenario pins OPCUA_ENDPOINT_URL, which +# short-circuits discovery, so none of them can see this. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/../../../../.." && pwd)" +NET_NAME=discovery-race-net +NET_SUBNET=172.31.77.0/24 +SERVER_NAME=discovery-race-server +GATEWAY_NAME=discovery-race-gateway +SERVER_PORT=4840 +GATEWAY_PORT=8089 +RESCAN_INTERVAL_S=5 +CONFIG_DIR=/tmp/discovery_race_config + +# The fallback the plugin keeps when a scan selects nothing (OpcuaClientConfig). +FALLBACK_ENDPOINT="opc.tcp://localhost:4840" +# Config-less naming: with no node map the component id is derived from the +# device. Before any server exists that can only be the fallback endpoint's host; +# after adoption it is the test server's DI nameplate (Manufacturer "SelfPatch +# Devices" + Model "SPX-1000"), slugified. +FALLBACK_COMPONENT_ID="opcua-localhost" +DEVICE_COMPONENT_ID="selfpatch_devices_spx_1000" + +cleanup() { + local rc=$? + if [[ ${rc} -ne 0 ]]; then + for c in "${SERVER_NAME}" "${GATEWAY_NAME}"; do + echo "=== ${c} logs (cleanup trap) ===" >&2 + docker logs "${c}" >&2 2>&1 || true + done + fi + docker rm -f "${SERVER_NAME}" "${GATEWAY_NAME}" >/dev/null 2>&1 || true + docker network rm "${NET_NAME}" >/dev/null 2>&1 || true + rm -rf "${CONFIG_DIR}" +} +trap cleanup EXIT + +fail() { + echo " FAIL: $*" >&2 + exit 1 +} + +# x-plc-status of a named component (the node-map pass pins the id; the +# config-less pass has to look it up first). +status_json_for() { + curl -sf "http://localhost:${GATEWAY_PORT}/api/v1/components/$1/x-plc-status" || echo '{}' +} + +status_json() { + status_json_for discovery_race_runtime +} + +status_field() { + status_json | python3 -c "import json,sys; print(json.load(sys.stdin).get('$1', ''))" +} + +status_field_for() { + status_json_for "$1" | python3 -c "import json,sys; print(json.load(sys.stdin).get('$2', ''))" +} + +component_ids() { + curl -sf "http://localhost:${GATEWAY_PORT}/api/v1/components" 2>/dev/null | + python3 -c " +import json,sys +try: + print(' '.join(c.get('id','') for c in json.load(sys.stdin).get('items', []))) +except Exception: + print('') +" +} + +wait_for_rest_api() { + for _ in $(seq 1 60); do + if curl -sf "http://localhost:${GATEWAY_PORT}/api/v1/components" >/dev/null 2>&1; then + return 0 + fi + sleep 1 + done + curl -sf "http://localhost:${GATEWAY_PORT}/api/v1/components" >/dev/null \ + || fail "gateway REST API never came up" +} + +cd "${REPO_ROOT}" + +# Idempotent teardown of anything a hard-killed earlier run left behind. +docker rm -f "${SERVER_NAME}" "${GATEWAY_NAME}" >/dev/null 2>&1 || true +docker network rm "${NET_NAME}" >/dev/null 2>&1 || true + +echo "[1/9] Build images" +docker build --network=host \ + -f src/ros2_medkit_plugins/ros2_medkit_opcua/docker/test_alarm_server/Dockerfile \ + -t ros2_medkit_alarm_test_server:dev . >/dev/null +docker build --network=host \ + -f src/ros2_medkit_plugins/ros2_medkit_opcua/docker/Dockerfile.gateway \ + -t gateway-opcua:discovery-race . >/dev/null + +# A /24 keeps the read-only sweep to 254 hosts, so a scan finishes in seconds. +# Discovery rejects anything wider than /16 outright. +docker network create --subnet "${NET_SUBNET}" "${NET_NAME}" >/dev/null + +echo "[2/9] Start the gateway BEFORE any server, discovery on, no endpoint pinned" +mkdir -p "${CONFIG_DIR}" +cat >"${CONFIG_DIR}/discovery_nodes.yaml" <<'EOF' +area_id: plc_systems +component_id: discovery_race_runtime +# One node is enough: the assertion is on the session, not on any value. The +# node id need not resolve on the server, since a failed read does not drop +# the connection. +nodes: + - node_id: "ns=2;s=StatusWord" + entity_id: tank_process + data_name: status_word + data_type: int +EOF +cat >"${CONFIG_DIR}/manifest.yaml" <<'EOF' +manifest_version: "1.0" +EOF +cp src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml \ + "${CONFIG_DIR}/gateway_params.yaml" + +docker run -d --name "${GATEWAY_NAME}" --network "${NET_NAME}" \ + -p "${GATEWAY_PORT}:8080" \ + -v "${CONFIG_DIR}:/config:ro" \ + -e ROS_DOMAIN_ID=67 \ + -e OPCUA_DISCOVERY_ENABLED=1 \ + -e OPCUA_DISCOVERY_SUBNETS="${NET_SUBNET}" \ + -e OPCUA_DISCOVERY_INTERVAL_S="${RESCAN_INTERVAL_S}" \ + -e OPCUA_NODE_MAP_PATH=/config/discovery_nodes.yaml \ + gateway-opcua:discovery-race \ + bash -c ' + set -e + mkdir -p /var/lib/ros2_medkit/rosbags + source /opt/ros/jazzy/setup.bash + source /root/ws/install/setup.bash + ros2 run ros2_medkit_fault_manager fault_manager_node \ + > /var/lib/ros2_medkit/fault_manager.log 2>&1 & + PLUGIN_PATH=$(find /root/ws/install -name "libros2_medkit_opcua_plugin.so" | head -1) + exec ros2 run ros2_medkit_gateway gateway_node \ + --ros-args --params-file /config/gateway_params.yaml \ + -p plugins.opcua.path:="${PLUGIN_PATH}" \ + -p discovery.mode:=hybrid \ + -p discovery.manifest_path:=/config/manifest.yaml \ + -p discovery.manifest_strict_validation:=false' >/dev/null + +echo "[3/9] Wait for the REST API" +wait_for_rest_api + +echo "[4/9] Assert the start-up scan found nothing and left the fallback endpoint" +# The scan runs during set_context(), so by the time the API answers it has +# already completed against an empty network. +endpoint="$(status_field endpoint_url)" +connected="$(status_field connected)" +[[ "${endpoint}" == "${FALLBACK_ENDPOINT}" ]] \ + || fail "expected the fallback endpoint before the server exists, got '${endpoint}'" +[[ "${connected}" == "False" ]] \ + || fail "expected no session before the server exists, got connected='${connected}'" +echo " OK no server found at start-up, endpoint left at ${FALLBACK_ENDPOINT}" + +echo "[5/9] Start the OPC-UA server (the PLC finishing its boot)" +docker run -d --name "${SERVER_NAME}" --network "${NET_NAME}" \ + ros2_medkit_alarm_test_server:dev --port "${SERVER_PORT}" >/dev/null +for _ in $(seq 1 30); do + if docker logs "${SERVER_NAME}" 2>&1 | grep -q '^READY '; then + break + fi + sleep 1 +done +docker logs "${SERVER_NAME}" 2>&1 | grep -q '^READY ' || fail "test server never became ready" +SERVER_IP="$(docker inspect -f "{{(index .NetworkSettings.Networks \"${NET_NAME}\").IPAddress}}" "${SERVER_NAME}")" +echo " server up at ${SERVER_IP}:${SERVER_PORT}" + +echo "[6/9] Assert the gateway adopts it within two re-scan intervals, unrestarted" +# Budget arithmetic. The re-scan is consulted once per reconnect attempt, and +# those are spaced by the reconnect backoff, so the adoption cadence is +# max(interval_s, backoff). The backoff ceiling is capped at interval_s while +# discovery is re-scanning, which is what keeps this budget in terms of +# interval_s alone: worst case is one full interval before the sweep is due plus +# one for the attempt that follows it, and the +40 s covers the sweep of a /24, +# the connect and the REST refresh. Generous enough not to flake, far short of +# "never", which is what the bug did. +DEADLINE=$((SECONDS + 2 * RESCAN_INTERVAL_S + 40)) +adopted="" +while [[ ${SECONDS} -lt ${DEADLINE} ]]; do + endpoint="$(status_field endpoint_url)" + connected="$(status_field connected)" + if [[ "${endpoint}" == "opc.tcp://${SERVER_IP}:${SERVER_PORT}" && "${connected}" == "True" ]]; then + adopted="${endpoint}" + break + fi + sleep 2 +done +[[ -n "${adopted}" ]] \ + || fail "endpoint still '${endpoint}' (connected='${connected}') after $((2 * RESCAN_INTERVAL_S + 40))s" +echo " OK re-scan adopted ${adopted} without a gateway restart" + +# The gateway must have adopted the server in the process that started before +# it, not in a fresh one: a restarted container would pass the check above +# while proving nothing. +restarts="$(docker inspect -f '{{.RestartCount}}' "${GATEWAY_NAME}")" +[[ "${restarts}" == "0" ]] || fail "gateway restarted ${restarts} time(s) during the run" +echo " OK gateway never restarted" + +# --------------------------------------------------------------------------- +# Config-less variant: the same race with NO node map. +# +# Without a node map the SOVD component is named from the device itself. A +# gateway that scanned before its PLC existed has no device to ask, so it can +# only name the component after the fallback endpoint. The defect this covers is +# that identity being pinned once and never revisited: the component kept +# serving opcua-localhost for the life of the process while the plugin was +# happily polling the adopted PLC. +# --------------------------------------------------------------------------- + +echo "[7/9] Config-less pass: stop everything, start the gateway with no node map" +docker rm -f "${SERVER_NAME}" "${GATEWAY_NAME}" >/dev/null 2>&1 || true + +docker run -d --name "${GATEWAY_NAME}" --network "${NET_NAME}" \ + -p "${GATEWAY_PORT}:8080" \ + -v "${CONFIG_DIR}:/config:ro" \ + -e ROS_DOMAIN_ID=67 \ + -e OPCUA_DISCOVERY_ENABLED=1 \ + -e OPCUA_DISCOVERY_SUBNETS="${NET_SUBNET}" \ + -e OPCUA_DISCOVERY_INTERVAL_S="${RESCAN_INTERVAL_S}" \ + gateway-opcua:discovery-race \ + bash -c ' + set -e + mkdir -p /var/lib/ros2_medkit/rosbags + source /opt/ros/jazzy/setup.bash + source /root/ws/install/setup.bash + ros2 run ros2_medkit_fault_manager fault_manager_node \ + > /var/lib/ros2_medkit/fault_manager.log 2>&1 & + PLUGIN_PATH=$(find /root/ws/install -name "libros2_medkit_opcua_plugin.so" | head -1) + exec ros2 run ros2_medkit_gateway gateway_node \ + --ros-args --params-file /config/gateway_params.yaml \ + -p plugins.opcua.path:="${PLUGIN_PATH}" \ + -p discovery.mode:=hybrid \ + -p discovery.manifest_path:=/config/manifest.yaml \ + -p discovery.manifest_strict_validation:=false' >/dev/null + +wait_for_rest_api + +echo "[8/9] Assert the component is named after the fallback while nothing answers" +ids="" +DEADLINE=$((SECONDS + 30)) +while [[ ${SECONDS} -lt ${DEADLINE} ]]; do + ids="$(component_ids)" + if [[ " ${ids} " == *" ${FALLBACK_COMPONENT_ID} "* ]]; then + break + fi + sleep 2 +done +[[ " ${ids} " == *" ${FALLBACK_COMPONENT_ID} "* ]] \ + || fail "expected the provisional component '${FALLBACK_COMPONENT_ID}' before any server, got '${ids}'" +echo " OK config-less component provisionally named ${FALLBACK_COMPONENT_ID}" + +echo "[9/9] Start the server and assert the component is renamed from the device" +docker run -d --name "${SERVER_NAME}" --network "${NET_NAME}" \ + ros2_medkit_alarm_test_server:dev --port "${SERVER_PORT}" >/dev/null +for _ in $(seq 1 30); do + if docker logs "${SERVER_NAME}" 2>&1 | grep -q '^READY '; then + break + fi + sleep 1 +done +docker logs "${SERVER_NAME}" 2>&1 | grep -q '^READY ' || fail "test server never became ready" +SERVER_IP="$(docker inspect -f "{{(index .NetworkSettings.Networks \"${NET_NAME}\").IPAddress}}" "${SERVER_NAME}")" +echo " server up at ${SERVER_IP}:${SERVER_PORT}" + +# Same budget as step 6, plus the discovery refresh that republishes entities. +DEADLINE=$((SECONDS + 2 * RESCAN_INTERVAL_S + 60)) +renamed="" +while [[ ${SECONDS} -lt ${DEADLINE} ]]; do + ids="$(component_ids)" + if [[ " ${ids} " == *" ${DEVICE_COMPONENT_ID} "* ]]; then + renamed="${DEVICE_COMPONENT_ID}" + break + fi + sleep 2 +done +[[ -n "${renamed}" ]] \ + || fail "component still '${ids}' after adoption - expected the device-derived '${DEVICE_COMPONENT_ID}'" + +# The renamed component is the one actually polling the adopted PLC, so the id +# the operator sees is not a second, stale entity next to a live opcua-localhost. +endpoint="$(status_field_for "${renamed}" endpoint_url)" +connected="$(status_field_for "${renamed}" connected)" +[[ "${endpoint}" == "opc.tcp://${SERVER_IP}:${SERVER_PORT}" ]] \ + || fail "renamed component reports endpoint '${endpoint}', expected the adopted server" +[[ "${connected}" == "True" ]] \ + || fail "renamed component reports connected='${connected}', expected a live session" +[[ "${renamed}" != "${FALLBACK_COMPONENT_ID}" ]] \ + || fail "component id never moved off ${FALLBACK_COMPONENT_ID}" +echo " OK component renamed to ${renamed}, connected at ${endpoint}" + +restarts="$(docker inspect -f '{{.RestartCount}}' "${GATEWAY_NAME}")" +[[ "${restarts}" == "0" ]] || fail "gateway restarted ${restarts} time(s) during the config-less run" +echo " OK gateway never restarted" + +echo "Discovery race scenario passed." diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp index ed68d33e5..9f73d21d2 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp @@ -16,6 +16,7 @@ #include #include +#include #include #include @@ -105,10 +106,17 @@ struct OpcuaDiscoveryConfig { int scan_concurrency{100}; ///< bounded, polite concurrent connect count int identify_timeout_ms{6000}; - /// Re-scan cadence. 0 = one-shot at startup (the only mode implemented in - /// this iteration); a positive value is accepted and validated but periodic - /// re-scan is a documented follow-up. - int interval_s{0}; + /// Re-scan cadence, in seconds, while no OPC-UA session is established. + /// Unset (the key absent) selects the built-in default, an explicit 0 turns + /// re-scanning off and keeps the startup scan one-shot - the two are + /// deliberately distinct, so a deployment can keep discovery on and still + /// stop the recurring sweep (see OpcuaPlugin::effective_rescan_interval_s). + /// The startup scan always runs once. The cadence only governs how often the + /// disconnected reconnect loop scans again, so a gateway that started before + /// its PLC finished booting adopts the PLC when it appears instead of retrying + /// the fallback endpoint forever. Never used once an endpoint is configured + /// explicitly, and never while a session is up. + std::optional interval_s; /// Only auto-register endpoints that expose a None + Anonymous endpoint (what /// the plugin connects with today). Secured-only servers are surfaced as @@ -154,7 +162,15 @@ class NetworkDiscovery { /// Run one full discovery pass (blocking). Read-only: TCP connect + /// GetEndpoints only. Deduplicated by ApplicationUri (fallback ip:port), /// sorted deterministically by ip:port. - std::vector run(); + /// + /// @param cancelled optional abort predicate, polled before every probe and + /// between the sweep and identify phases. A sweep of a legal /16 is + /// tens of thousands of probes and takes minutes, and the caller runs + /// it on the poll thread that a shutdown has to join, so without this + /// a ``docker stop`` grace period would expire mid-sweep. A pass still + /// cancelled at the next phase boundary returns an empty result rather + /// than a partial one, within one in-flight probe per worker. + std::vector run(const std::function & cancelled = {}); /// Resolve the subnets to scan: configured ``subnets`` if any, else the /// derived local /24. Exposed for logging / tests. diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index a547740e8..4ad2da902 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -15,11 +15,14 @@ #pragma once #include "ros2_medkit_opcua/address_space_browser.hpp" +#include "ros2_medkit_opcua/device_identity.hpp" #include "ros2_medkit_opcua/network_discovery.hpp" #include "ros2_medkit_opcua/node_map.hpp" #include "ros2_medkit_opcua/opcua_client.hpp" #include "ros2_medkit_opcua/opcua_poller.hpp" +#include + #include #include #include @@ -31,10 +34,12 @@ #include #include +#include #include #include #include #include +#include #include #include #include @@ -156,6 +161,148 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, static void apply_auto_alarms_param(const nlohmann::json & value, AutoAlarmsConfig & cfg, const std::function & warn); + // Where one discovery pass reports to, plus the memory that keeps a repeated + // identical pass quiet. A rescan runs every ``interval_s`` for the life of a + // disconnected process, so re-emitting the same scan line, per-server lines, + // summary and "no auto-connectable server" WARN each time buries every other + // message in the log. ``previous_outcome`` is owned by the caller (the plugin + // keeps one across rescans): when it is non-null and the pass reaches the same + // outcome as the pass before it, the whole report goes to ``debug`` instead. + // The first pass, and every pass whose outcome changed, is always reported at + // info/warn. A null ``previous_outcome`` (the startup scan, and tests that do + // not care) reports every pass. + struct DiscoveryReporter { + std::function info; + std::function warn; + std::function debug; + std::string * previous_outcome{nullptr}; + }; + + // Run one read-only discovery pass and return the endpoint URL to adopt. + // + // Returns nullopt - meaning "keep the endpoint you have" - when discovery is + // disabled, when an endpoint was configured explicitly, when no subnet could + // be resolved, or when the pass found no auto-connectable None/Anonymous data + // server. Both the startup scan and the reconnect rescan go through here, so + // the two cannot drift apart. Static with injected probes and log sinks so + // both are unit-testable without a network. + // + // @param config discovery configuration (subnets, ports, timeouts, ...) + // @param endpoint_configured true when the operator pinned endpoint_url, so + // discovery then selects nothing and can neither override the + // operator's target nor open a second session on an already polled PLC + // @param scan injected TCP port probe + // @param identify injected OPC-UA GetEndpoints identify + // @param reporter operator-visible log sinks + repeat-suppression memory + // @param cancelled abort predicate handed to NetworkDiscovery::run, so a + // shutdown does not have to wait out a full sweep + static std::optional discover_endpoint(const OpcuaDiscoveryConfig & config, bool endpoint_configured, + const PortScanFn & scan, const IdentifyFn & identify, + const DiscoveryReporter & reporter, + const std::function & cancelled = {}); + + // Seconds between reconnect rescans, or 0 when the reconnect loop must never + // rescan. That is the answer when discovery is disabled, when an endpoint was + // configured explicitly, and when the operator set ``interval_s: 0``, which + // means "keep discovery on but leave the startup scan one-shot". An UNSET + // interval is the config-less case - it cannot name a cadence and is the one + // that most needs its PLC adopted once it finishes booting - so it takes the + // built-in default instead of never rescanning. + static int effective_rescan_interval_s(const OpcuaDiscoveryConfig & config, bool endpoint_configured); + + // Default reconnect rescan cadence, in seconds, when discovery is enabled with + // no explicit ``interval_s``. Long enough that a bounded subnet sweep stays a + // background cost on the poll thread, short enough that a PLC finishing its + // boot is picked up in well under a minute. + static constexpr int kDefaultRescanIntervalS = 30; + + // One rate-limited rescan step: run ``sweep`` when the cadence is due, + // otherwise do nothing. + // + // The cadence is measured from the END of the previous sweep, which + // ``*last_scan_end`` stores. A sweep of a legal /16 takes minutes, so + // stamping its start would make the next one due the moment it returned: the + // poll thread would sweep back to back and the reconnect attempt would drop to + // one per sweep. ``now`` is injected so the spacing is unit-testable without + // sleeping. + // + // @return whatever ``sweep`` returned, or nullopt when it was not due yet. + static std::optional rescan_step(int interval_s, + const std::function & now, + std::chrono::steady_clock::time_point * last_scan_end, + const std::function()> & sweep); + + // Ceiling for the poller's exponential reconnect backoff. + // + // Without discovery this is ``default_ceiling`` (60 s). While the reconnect + // loop is rescanning, the rescan is only consulted once per reconnect attempt, + // so the real adoption cadence is max(interval_s, backoff) - a documented + // "every 30 s" would silently become every 60 s. Capping the backoff at the + // rescan interval makes the documented cadence the true one. Never shorter + // than ``base`` (the configured reconnect interval), so a tiny interval cannot + // turn the backoff into a hot retry loop. + static std::chrono::milliseconds effective_max_reconnect_wait(std::chrono::milliseconds base, + std::chrono::milliseconds default_ceiling, + int rescan_interval_s); + + // The component identity a config-less deployment should serve after a + // connect, or nullopt when the identity it already has still holds. + // + // A gateway that starts before its PLC connects to nothing, so the identity + // derived at start-up comes from an empty DeviceInfo and the fallback + // endpoint: the neutral ``opcua-`` placeholder. Once discovery adopts + // the real server, the device can finally name itself, and the SOVD component + // must stop serving the placeholder. Pure / static so the rule is testable + // without a server. + static std::optional rederived_component_identity(const std::string & current_id, + const OpcuaClient::DeviceInfo & info, + const std::string & endpoint_url); + + // Build the ClearFault request for one fault code. ``link_state`` marks a + // clear that only reports the OPC-UA link came back (the connect-time + // ``PLC_COMMS_LOST`` clear). Such a clear must not cascade: a correlation rule + // may name PLC_COMMS_LOST as the root cause of every symptom the outage + // produced, and the link returning is not an operator resolving those. An + // operator-driven clear (the SOVD DELETE route) leaves the flag off and keeps + // the cascade. Static so the wire field is assertable without a fault manager. + static ros2_medkit_msgs::srv::ClearFault::Request make_clear_fault_request(const std::string & fault_code, + bool link_state); + + // One entry in the bounded buffer of fault dispatches held while the + // fault_manager service is unmatched. + struct PendingFaultDispatch { + enum class Kind { Report, Clear }; + Kind kind{Kind::Report}; + std::string fault_code; ///< dedup key for a Clear; diagnostic for a Report + std::function dispatch; + }; + + // What ``enqueue_pending_dispatch`` did, so the caller can log it. + enum class PendingEnqueueOutcome { + Buffered, ///< appended, nothing lost + ReplacedClear, ///< superseded the pending clear for the same fault code + EvictedClear, ///< buffer was full: dropped a pending clear to make room + EvictedReport, ///< buffer was full of reports and a report arrived + Refused ///< buffer was full of reports and a clear arrived + }; + + // Enqueue policy for the bounded pending-dispatch buffer. + // + // Reports outrank clears. A report is a one-shot edge from the PLC that + // nothing will re-send, while a clear is re-derivable: the link state is + // re-observed on the next reconnect. So at most ONE clear per fault code is + // ever pending (a newer one moves to the back, keeping report-then-clear + // order), a full buffer gives up its oldest pending clear first, and a clear + // arriving at a buffer full of reports is refused rather than evicting one. + // Without this a flapping link enqueued one connect-time clear per reconnect + // attempt and pushed real alarm reports out of the buffer. + static PendingEnqueueOutcome enqueue_pending_dispatch(std::vector & buffer, size_t max_size, + PendingFaultDispatch entry); + + // Bound on the pending-dispatch buffer, so a deployment with no fault_manager + // cannot grow it without limit. + static constexpr size_t kMaxPendingDispatches = 256; + private: // Route handlers void handle_plc_data(const ros2_medkit_gateway::PluginRequest & req, ros2_medkit_gateway::PluginResponse & res); @@ -177,11 +324,21 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // Report/clear fault via ROS 2 service (private helpers, not the FaultProvider overrides) void send_report_fault(const std::string & entity_id, const std::string & fault_code, const std::string & severity_str, const std::string & message); - void send_clear_fault(const std::string & fault_code); + // ``link_state`` marks a clear that reports the OPC-UA link came back rather + // than an operator resolving a root cause; see make_clear_fault_request. + void send_clear_fault(const std::string & fault_code, bool link_state = false); + + // Clear PLC_COMMS_LOST after the initial connect in set_context() succeeded. + // Unconditional on purpose: the fault manager keys faults by fault_code and + // persists them, so a comms-lost fault raised before a gateway restart is + // still standing in the store while this process has no memory of raising it. + // The poller's own reconnect clear can never reach that case, because a + // successful first connect means the reconnect arm is never entered. + void clear_comms_lost_on_connect(); // Dispatch now if the fault_manager service is matched, else buffer the // dispatch (bounded, order-preserving) to be flushed once it appears. - void send_or_buffer(std::function dispatch); + void send_or_buffer(PendingFaultDispatch entry); // Flush buffered fault dispatches when the fault_manager service is ready. void flush_pending_reports(); @@ -204,6 +361,16 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // node_map_mutex_. No-op (never called) when auto_browse is disabled. void run_auto_browse(); + // Poll-thread hook (from publish_values): re-derive the SOVD component + // identity from the device once a NEW session is up, in config-less mode only + // (an explicit node map owns the name). This is what stops a gateway that + // started before its PLC from serving the ``opcua-`` + // placeholder for the life of the process after discovery adopted the real + // server. Logs the change at INFO and rebuilds every derived reference (the + // ``_alarms`` entity, entity_defs) under the node-map lock. + // No-op when the identity is unchanged. + void maybe_rederive_component_identity(); + // Poll-thread hook (from publish_values): re-run auto_browse when the client // has established a new session since the last walk. Covers the field case // where the gateway starts before the PLC is reachable (initial connect @@ -225,6 +392,20 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // an endpoint is already configured. void run_startup_discovery(); + // Log sinks for a discovery pass: the plugin's operator-visible info/warn + // plus the named ``opcua.plugin`` debug logger a repeated identical pass falls + // back to. ``previous_outcome`` is the caller's repeat-suppression memory + // (null to report every pass in full). + DiscoveryReporter discovery_reporter(std::string * previous_outcome) const; + + // Poll-thread hook bound into PollerConfig::rediscover_endpoint whenever + // discovery runs without a configured endpoint. Called from the poller's + // reconnect arm, so only while no session is up, and rate-limited to one scan + // per effective_rescan_interval_s(). Returns the newly selected endpoint when + // a rescan found a different server (and logs the swap at INFO), nullopt when + // the rescan is not due yet or changed nothing. + std::optional rescan_endpoint_for_reconnect(); + // Build JSON response for data endpoint nlohmann::json build_data_response(const std::string & entity_id) const; @@ -250,6 +431,15 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // a network. Set in configure(), consumed in run_startup_discovery(). PortScanFn discovery_scan_fn_; IdentifyFn discovery_identify_fn_; + // When the last discovery pass FINISHED, so the reconnect rescan honours the + // cadence instead of sweeping the subnet on every reconnect attempt. Stamped + // by the startup scan, then only ever read/written on the poll thread. See + // rescan_step for why the end of the sweep is the reference point. + std::chrono::steady_clock::time_point last_discovery_scan_end_{}; + // Outcome digest of the previous discovery pass, so an unchanged rescan + // reports at DEBUG instead of repeating the whole report every interval_s. + // Poll thread only (the startup scan runs before the poller exists). + std::string last_discovery_outcome_; std::unique_ptr client_; NodeMap node_map_; @@ -278,6 +468,13 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, AssetIdentity device_identity_; uint64_t device_identity_generation_{0}; + // OpcuaClient::connection_generation the config-less component identity was + // derived at (0 = derived with no session, i.e. from the fallback endpoint and + // an empty DeviceInfo). The poll thread re-derives when the live generation + // differs, mirroring device_identity_generation_. Written on the set_context + // thread (happens-before the poller starts) then only on the poll thread. + uint64_t component_identity_generation_{0}; + // ROS 2 service clients for fault reporting struct FaultClients; std::unique_ptr fault_clients_; @@ -297,7 +494,7 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // never held across the actual dispatch (async_send_request) to keep ROS I/O out // of the critical section. std::mutex pending_reports_mutex_; - std::vector> pending_reports_; + std::vector pending_reports_; // Tracks which non-numeric nodes have already been warned about (avoids log spam). // Instance member instead of static to survive plugin reload (dlclose/dlopen). diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp index 711c3ec47..5e3c183fd 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -102,6 +103,13 @@ struct PollerConfig { double subscription_interval_ms{500.0}; std::chrono::milliseconds poll_interval{1000}; std::chrono::milliseconds reconnect_interval{5000}; + /// Ceiling for the exponential reconnect backoff (it doubles from + /// ``reconnect_interval`` up to this). A plugin whose reconnect arm also + /// rescans for a new endpoint lowers this to the rescan cadence: the rescan is + /// consulted once per reconnect attempt, so a backoff longer than the cadence + /// would silently stretch the documented "re-scan every interval_s" to the + /// backoff instead (see OpcuaPlugin::effective_max_reconnect_wait). + std::chrono::milliseconds max_reconnect_interval{60000}; /// Active-condition replay strategy on (re)subscribe (issue #389). /// Default Auto: ConditionRefresh with a read-based fallback so hardened /// servers that reject the method still recover their active alarms. @@ -133,8 +141,21 @@ struct PollerConfig { /// fire-and-forget report is never dropped-and-forgotten while the sink is /// unmatched - it retries on the next poll instead. Empty => assume ready. std::function report_sink_ready; + /// Optional endpoint rediscovery, bound by a plugin running config-less + /// network discovery with no endpoint configured. Called from the reconnect + /// arm - so only while no session is up - and expected to rate-limit itself. + /// Returns a new endpoint URL to reconnect against, nullopt to keep the + /// current one. Without it the reconnect loop retries the same endpoint + /// forever, which strands a gateway that scanned before its PLC had booted. + std::function()> rediscover_endpoint; }; +/// Fault code of the component-scoped OPC-UA connection fault the poller raises +/// on a sustained outage and clears on the next successful connect (issue #496). +/// Named here so the plugin can clear the same code from its own connect path +/// without keeping a second copy of the literal. +inline constexpr const char * kCommsLostFaultCode = "PLC_COMMS_LOST"; + /// Manages OPC-UA data collection via subscriptions (preferred) or polling class OpcuaPoller { public: @@ -237,6 +258,24 @@ class OpcuaPoller { std::chrono::steady_clock::time_point down_since, std::chrono::steady_clock::time_point now, std::chrono::milliseconds debounce); + /// Endpoint the next reconnect attempt should target. Asks + /// ``rediscover_endpoint`` (when bound) for a freshly discovered server and + /// returns it only when it names a DIFFERENT endpoint than ``current``. + /// nullopt means "keep the current one", which is also the answer when no + /// callback is bound, when the callback declines, or when it hands back an + /// empty string. Pure and static (the callback is injected) so the adoption + /// rule is unit-testable without a network. + static std::optional + adopt_rediscovered_endpoint(const std::string & current, + const std::function()> & rediscover); + + /// Wait before the next reconnect attempt: the current wait doubled, clamped + /// to ``max_wait`` (a wait already above the cap comes back down to it). Pure + /// and static so the backoff - and the cap that keeps a rescanning reconnect + /// loop on its documented cadence - is unit-testable. + static std::chrono::milliseconds next_reconnect_wait(std::chrono::milliseconds current, + std::chrono::milliseconds max_wait); + /// Zero-config native A&C (``auto_alarms``): the alarm sources that should /// actually be subscribed / replayed, i.e. every explicit ``event_alarms`` /// entry plus (when ``auto_cfg.enabled`` and no explicit entry already @@ -430,7 +469,10 @@ class OpcuaPoller { // Issue #496: comms-lost debounce state, touched only on the poll thread. // ``comms_down_since_`` is set the first poll iteration the connection is // observed down and cleared on reconnect; ``comms_lost_raised_`` guards the - // one-shot raise / matching clear so the fault is idempotent. + // one-shot RAISE only. The clear is deliberately not guarded by it: it is sent + // on every successful reconnect, because the fault manager persists faults by + // fault_code and a comms-lost fault raised before a restart is standing in the + // store with nothing in this process's memory to remember it. std::optional comms_down_since_; bool comms_lost_raised_{false}; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp index 40f7dd2e5..f3eaceea7 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -71,8 +72,12 @@ bool ipv4_less(const std::string & a, const std::string & b) { // ``max_workers`` threads (never more than ``count``) that pull indices off a // shared atomic cursor. The single primitive backs both the connect sweep and // the GetEndpoints identify so they share the same concurrency bound. +// +// ``cancelled`` is polled by every worker before it takes the next index, so an +// abort stops the fan-out after at most one more probe per worker instead of +// running the remaining tens of thousands. An empty predicate never cancels. template -void parallel_for(size_t count, int max_workers, Body && body) { +void parallel_for(size_t count, int max_workers, const std::function & cancelled, Body && body) { if (count == 0) { return; } @@ -80,6 +85,9 @@ void parallel_for(size_t count, int max_workers, Body && body) { std::atomic next{0}; const auto worker = [&]() { for (;;) { + if (cancelled && cancelled()) { + return; + } const size_t i = next.fetch_add(1); if (i >= count) { return; @@ -306,9 +314,12 @@ OpcuaDiscoveryConfig parse_discovery_config(const nlohmann::json & j, if (j.contains("interval_s") && j["interval_s"].is_number_integer()) { const int v = j["interval_s"].get(); if (v >= 0) { + // An explicit 0 is kept as an explicit 0: it means "keep discovery on but + // never re-scan", which an unset interval (the built-in cadence) cannot + // express. cfg.interval_s = v; } else { - warn_fn("discovery: interval_s must be >= 0 - keeping default (0 = one-shot)"); + warn_fn("discovery: interval_s must be >= 0 (0 disables re-scanning) - keeping the default cadence"); } } if (j.contains("anonymous_none_only") && j["anonymous_none_only"].is_boolean()) { @@ -333,7 +344,7 @@ std::vector NetworkDiscovery::resolve_subnets() const { return {local}; } -std::vector NetworkDiscovery::run() { +std::vector NetworkDiscovery::run(const std::function & cancelled) { // "passive" has no active scan implementation yet (mDNS / LDS FindServers // are a documented follow-up) - a no-op stub rather than silently running // the active scan mode wasn't asked for. parse_discovery_config() already @@ -365,7 +376,7 @@ std::vector NetworkDiscovery::run() { // only the polite fan-out lives here. std::vector open_hits; std::mutex hits_mu; - parallel_for(targets.size(), cfg_.scan_concurrency, [&](size_t i) { + parallel_for(targets.size(), cfg_.scan_concurrency, cancelled, [&](size_t i) { const Target & t = targets[i]; if (scan_(t.ip, t.port, cfg_.connect_timeout_ms)) { std::lock_guard lk(hits_mu); @@ -373,6 +384,12 @@ std::vector NetworkDiscovery::run() { } }); + // Cancelled mid-sweep: the hit list is partial, so do not spend an identify + // round-trip per hit on a pass whose caller is shutting down. + if (cancelled && cancelled()) { + return {}; + } + // Deterministic identify order (numerically lowest ip:port first). std::sort(open_hits.begin(), open_hits.end(), [](const Target & a, const Target & b) { if (a.ip != b.ip) { @@ -388,7 +405,7 @@ std::vector NetworkDiscovery::run() { // sweep, writing each result by index so the deterministic ip:port order // (open_hits was sorted above) survives regardless of completion order. std::vector built(open_hits.size()); - parallel_for(open_hits.size(), cfg_.scan_concurrency, [&](size_t i) { + parallel_for(open_hits.size(), cfg_.scan_concurrency, cancelled, [&](size_t i) { const Target & t = open_hits[i]; DiscoveredEndpoint ep; ep.ip = t.ip; @@ -415,6 +432,12 @@ std::vector NetworkDiscovery::run() { built[i] = std::move(ep); }); + // Cancelled during the identify phase: ``built`` holds default-constructed + // (empty) entries for the hits no worker reached, which is not a result set. + if (cancelled && cancelled()) { + return {}; + } + // 4. Dedup by ApplicationUri (fallback ip:port), sequentially over the // deterministic order so the lowest ip:port always wins. std::vector ordered; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 3dff059a6..5010f2046 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -617,6 +617,7 @@ void OpcuaPlugin::set_context(PluginContext & context) { const bool connected = client_->connect(client_config_); if (connected) { log_info("Connected to OPC-UA server: " + client_config_.endpoint_url); + clear_comms_lost_on_connect(); } else { log_warn("Failed to connect to OPC-UA server: " + client_config_.endpoint_url); } @@ -635,6 +636,13 @@ void OpcuaPlugin::set_context(PluginContext & context) { node_map_.set_component_identity(ci.id, ci.name); log_info("Component identity derived from device: id='" + ci.id + "', name='" + ci.name + "'"); } + // Which session this identity speaks for. 0 when the connect failed: the id + // above then came from the fallback endpoint and an empty DeviceInfo, so it + // is provisional and the poll thread re-derives it on the first session + // (maybe_rederive_component_identity). Without that a gateway which started + // before its PLC would serve the placeholder for the life of the process, + // even after discovery adopted the real server. + component_identity_generation_ = connected ? client_->connection_generation() : 0; // Zero-config native A&C: with no node map and no explicit auto_alarms // block, subscribe the Server EventNotifier by default so discovered @@ -692,6 +700,22 @@ void OpcuaPlugin::set_context(PluginContext & context) { poller_config_.report_sink_ready = [this]() { return fault_clients_->report && fault_clients_->report->service_is_ready(); }; + // Config-less discovery with no configured endpoint: let the poller's + // reconnect arm ask for a fresh scan while it is down. Without this the + // startup scan is the only one that ever runs, so a gateway that scanned + // while its PLC was still booting retries the fallback endpoint forever and + // only a restart finds the PLC. + const int rescan_interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); + if (rescan_interval_s > 0) { + poller_config_.rediscover_endpoint = [this]() { + return rescan_endpoint_for_reconnect(); + }; + // The rescan is consulted once per reconnect attempt, so the real adoption + // cadence is max(interval_s, backoff). Cap the backoff at the cadence so + // the documented "re-scan every interval_s while down" is the true one. + poller_config_.max_reconnect_interval = effective_max_reconnect_wait( + poller_config_.reconnect_interval, poller_config_.max_reconnect_interval, rescan_interval_s); + } poller_->start(poller_config_); log_info("OPC-UA poller started (mode: " + std::string(poller_->using_subscriptions() ? "subscription" : "poll") + ")"); @@ -767,7 +791,11 @@ IntrospectionResult OpcuaPlugin::introspect(const IntrospectionInput & /*input*/ // Fault scope grants bare-id ownership only to external entities; the poller // reports PLC_COMMS_LOST under this component's own id. comp.external = true; - comp.description = "PLC runtime connected at " + client_config_.endpoint_url; + // Read the endpoint off the client, not off client_config_: a reconnect + // rescan can adopt a different server after startup, and the client is the + // one that holds the endpoint actually being connected to. + const std::string live_endpoint = client_ ? client_->endpoint_url() : client_config_.endpoint_url; + comp.description = "PLC runtime connected at " + live_endpoint; // INV2: fill the asset-identity nameplate from the live server's device-info // (ServerStatus/BuildInfo + optional OPC-UA DI nameplate). Read once per @@ -778,7 +806,7 @@ IntrospectionResult OpcuaPlugin::introspect(const IntrospectionInput & /*input*/ if (client_ && client_->is_connected()) { const uint64_t session_generation = client_->connection_generation(); if (session_generation != device_identity_generation_) { - device_identity_ = opcua_device_info_to_identity(client_->read_device_info(), client_config_.endpoint_url); + device_identity_ = opcua_device_info_to_identity(client_->read_device_info(), live_endpoint); device_identity_generation_ = session_generation; if (!device_identity_.empty()) { log_info("Populated asset identity from OPC-UA device-info (manufacturer='" + device_identity_.manufacturer + @@ -1062,7 +1090,11 @@ void OpcuaPlugin::on_alarm_change(const std::string & entity_id, send_report_fault(entity_id, signal.fault_code, signal.severity, signal.message); } else { log_info("Alarm cleared: " + signal.fault_code + " on " + entity_id); - send_clear_fault(signal.fault_code); + // The poller's own comms-lost clear on a successful reconnect is the same + // link-state clear as the connect-time one, so it does not cascade either. + // Every other code is a real alarm going inactive on the device and keeps + // the default correlation behaviour. + send_clear_fault(signal.fault_code, /*link_state=*/signal.fault_code == kCommsLostFaultCode); } } @@ -1276,42 +1308,109 @@ void OpcuaPlugin::send_report_fault(const std::string & entity_id, const std::st request->severity = ros2_medkit_msgs::msg::Fault::SEVERITY_INFO; } - send_or_buffer([this, request]() { - fault_clients_->report->async_send_request(request); - }); + send_or_buffer({PendingFaultDispatch::Kind::Report, fault_code, [this, request]() { + fault_clients_->report->async_send_request(request); + }}); } -void OpcuaPlugin::send_clear_fault(const std::string & fault_code) { +ros2_medkit_msgs::srv::ClearFault::Request OpcuaPlugin::make_clear_fault_request(const std::string & fault_code, + bool link_state) { + ros2_medkit_msgs::srv::ClearFault::Request request; + request.fault_code = fault_code; + // A link-state clear reports that the OPC-UA session came back. It is not an + // operator resolving a root cause, so it must not trip the correlation + // engine's auto_clear_with_root cascade: a rule naming PLC_COMMS_LOST as the + // root cause would otherwise clear every symptom fault the outage produced, + // none of which this plugin has any evidence about. + request.skip_correlation_auto_clear = link_state; + return request; +} + +void OpcuaPlugin::send_clear_fault(const std::string & fault_code, bool link_state) { if (!fault_clients_->clear) { log_warn("ClearFault service client not available"); return; } - auto request = std::make_shared(); - request->fault_code = fault_code; + auto request = + std::make_shared(make_clear_fault_request(fault_code, link_state)); - send_or_buffer([this, request]() { - fault_clients_->clear->async_send_request(request); - }); + send_or_buffer({PendingFaultDispatch::Kind::Clear, fault_code, [this, request]() { + fault_clients_->clear->async_send_request(request); + }}); } -void OpcuaPlugin::send_or_buffer(std::function dispatch) { +void OpcuaPlugin::clear_comms_lost_on_connect() { + if (!poller_config_.comms_lost_fault_enabled) { + return; + } + // ClearFault is idempotent from this side: send_clear_fault is + // fire-and-forget, so a "Fault not found" answer for a code that was never + // raised costs nothing here and is the normal case on a healthy start. + log_info(std::string("OPC-UA connection established; clearing any standing ") + kCommsLostFaultCode); + send_clear_fault(kCommsLostFaultCode, /*link_state=*/true); +} + +OpcuaPlugin::PendingEnqueueOutcome OpcuaPlugin::enqueue_pending_dispatch(std::vector & buffer, + size_t max_size, PendingFaultDispatch entry) { + const bool is_clear = entry.kind == PendingFaultDispatch::Kind::Clear; + + // At most one pending clear per fault code. A repeat moves to the BACK rather + // than overwriting in place, so an interleaved report-then-clear for the same + // code still flushes in the order the PLC produced it. + bool replaced = false; + if (is_clear) { + const auto same_code = std::find_if(buffer.begin(), buffer.end(), [&entry](const PendingFaultDispatch & pending) { + return pending.kind == PendingFaultDispatch::Kind::Clear && pending.fault_code == entry.fault_code; + }); + if (same_code != buffer.end()) { + buffer.erase(same_code); + replaced = true; + } + } + + PendingEnqueueOutcome outcome = replaced ? PendingEnqueueOutcome::ReplacedClear : PendingEnqueueOutcome::Buffered; + if (buffer.size() >= max_size) { + // A report is a one-shot edge from the PLC that nothing will re-send; a + // clear is re-derivable from the next reconnect. So a full buffer gives up a + // pending clear first, and refuses an incoming clear rather than evicting a + // report for it. + const auto oldest_clear = std::find_if(buffer.begin(), buffer.end(), [](const PendingFaultDispatch & pending) { + return pending.kind == PendingFaultDispatch::Kind::Clear; + }); + if (oldest_clear != buffer.end()) { + buffer.erase(oldest_clear); + outcome = PendingEnqueueOutcome::EvictedClear; + } else if (is_clear) { + return PendingEnqueueOutcome::Refused; + } else { + buffer.erase(buffer.begin()); + outcome = PendingEnqueueOutcome::EvictedReport; + } + } + + buffer.push_back(std::move(entry)); + return outcome; +} + +void OpcuaPlugin::send_or_buffer(PendingFaultDispatch entry) { // Bound the buffer so a deployment with no fault_manager cannot grow it - // without limit; drop the oldest (least relevant) pending dispatch. - // Runs on both the poll thread and the REST clear_fault thread, so the vector - // mutation is serialised by pending_reports_mutex_. - constexpr size_t kMaxPendingReports = 256; - bool dropped_oldest = false; + // without limit. Runs on both the poll thread and the REST clear_fault thread, + // so the vector mutation is serialised by pending_reports_mutex_. + PendingEnqueueOutcome outcome = PendingEnqueueOutcome::Buffered; { std::lock_guard lock(pending_reports_mutex_); - if (pending_reports_.size() >= kMaxPendingReports) { - pending_reports_.erase(pending_reports_.begin()); - dropped_oldest = true; - } - pending_reports_.push_back(std::move(dispatch)); - } - if (dropped_oldest) { - log_warn("pending fault report buffer full (" + std::to_string(kMaxPendingReports) + "), dropping oldest"); + outcome = enqueue_pending_dispatch(pending_reports_, kMaxPendingDispatches, std::move(entry)); + } + if (outcome == PendingEnqueueOutcome::EvictedReport) { + log_warn("pending fault dispatch buffer full (" + std::to_string(kMaxPendingDispatches) + + "), dropping the oldest report"); + } else if (outcome == PendingEnqueueOutcome::EvictedClear) { + log_warn("pending fault dispatch buffer full (" + std::to_string(kMaxPendingDispatches) + + "), dropping the oldest pending clear"); + } else if (outcome == PendingEnqueueOutcome::Refused) { + log_warn("pending fault dispatch buffer full of reports (" + std::to_string(kMaxPendingDispatches) + + "), dropping this clear instead of a report"); } // Drains immediately (in order) if the sink is already matched. flush_pending_reports(); @@ -1327,7 +1426,7 @@ void OpcuaPlugin::flush_pending_reports() { // the vector is never being reallocated by a concurrent send_or_buffer while it // is iterated here (the use-after-free that corrupted the heap), and so the ROS // service call never runs under the mutex. - std::vector> batch; + std::vector batch; { std::lock_guard lock(pending_reports_mutex_); if (pending_reports_.empty()) { @@ -1335,8 +1434,8 @@ void OpcuaPlugin::flush_pending_reports() { } batch.swap(pending_reports_); } - for (auto & dispatch : batch) { - dispatch(); + for (auto & entry : batch) { + entry.dispatch(); } } @@ -1396,6 +1495,56 @@ void OpcuaPlugin::run_auto_browse() { (result.depth_cap_hit ? " [depth cap reached on at least one branch]" : "")); } +std::optional OpcuaPlugin::rederived_component_identity(const std::string & current_id, + const OpcuaClient::DeviceInfo & info, + const std::string & endpoint_url) { + const ComponentIdentity ci = derive_component_identity(info, endpoint_url); + if (ci.id.empty() || ci.id == current_id) { + return std::nullopt; + } + return ci; +} + +void OpcuaPlugin::maybe_rederive_component_identity() { + // An explicit node map owns the component name; only the config-less path + // derives it from the device. + if (!node_map_path_.empty() || !client_ || !client_->is_connected()) { + return; + } + const uint64_t generation = client_->connection_generation(); + if (generation == component_identity_generation_) { + return; // identity already speaks for this session + } + + const std::string live_endpoint = client_->endpoint_url(); + const auto rederived = + rederived_component_identity(node_map_.component_id(), client_->read_device_info(), live_endpoint); + component_identity_generation_ = generation; + if (!rederived) { + return; + } + + const std::string previous_id = node_map_.component_id(); + { + // Serialize against the REST read paths, which hold references into + // node_map_ (entity_defs) while they answer. + std::unique_lock lock(node_map_mutex_); + // The auto_alarms fallback entity is derived from the component id, so a + // default-derived one has to follow the rename. An operator-chosen entity_id + // does not match the derived form and is left alone. + auto & auto_alarms = node_map_.mutable_auto_alarms(); + if (auto_alarms.entity_id == previous_id + "_alarms") { + auto_alarms.entity_id.clear(); + } + node_map_.set_component_identity(rederived->id, rederived->name); + // Re-derives the fallback entity id and rebuilds entity_defs, so every + // reference to the component id moves together. + node_map_.finalize_auto_alarms_overlay(); + } + log_info("Component identity re-derived from the adopted device at " + live_endpoint + ": id='" + rederived->id + + "', name='" + rederived->name + "' (was '" + previous_id + "')"); +} + void OpcuaPlugin::maybe_rebrowse_on_reconnect() { if (!node_map_.auto_browse_config().enabled || !client_ || !client_->is_connected()) { return; @@ -1414,6 +1563,11 @@ void OpcuaPlugin::publish_values(const PollSnapshot & snap) { // Poll-thread hook: drain any fault reports buffered before fault_manager was // discovered, so a late sink still receives them. flush_pending_reports(); + // Poll-thread hook: re-derive the config-less component identity from the + // device once a session is up, so an adopted PLC stops being served under the + // provisional endpoint-derived id. Runs BEFORE the re-walk below, which + // rebuilds entity_defs off the component id. + maybe_rederive_component_identity(); // Poll-thread hook: (re)run auto_browse after a fresh session so a PLC that // came up (or restarted) after the initial connect still gets walked. maybe_rebrowse_on_reconnect(); @@ -1508,37 +1662,107 @@ void OpcuaPlugin::log_security_profile() const { } } -void OpcuaPlugin::run_startup_discovery() { - if (!discovery_config_.enabled) { - return; +int OpcuaPlugin::effective_rescan_interval_s(const OpcuaDiscoveryConfig & config, bool endpoint_configured) { + if (!config.enabled || endpoint_configured) { + return 0; } - // Never override an explicitly configured endpoint: discovery must not open a - // second session on a PLC the operator already targets (and already polls). - if (endpoint_configured_) { - log_info("OPC-UA discovery enabled but endpoint_url is explicitly configured (" + client_config_.endpoint_url + - "); skipping auto-discovery to avoid a second session."); - return; + // Unset means "discovery is on but no cadence was stated" - the config-less + // deployment, which takes the built-in default. An explicit 0 is an operator + // saying "do not re-scan", and is honoured as written. + return config.interval_s.value_or(kDefaultRescanIntervalS); +} + +std::chrono::milliseconds OpcuaPlugin::effective_max_reconnect_wait(std::chrono::milliseconds base, + std::chrono::milliseconds default_ceiling, + int rescan_interval_s) { + if (rescan_interval_s <= 0) { + return default_ceiling; // no rescan: the plain backoff ceiling applies } - if (discovery_config_.interval_s > 0) { - log_warn("OPC-UA discovery interval_s=" + std::to_string(discovery_config_.interval_s) + - " set, but periodic re-scan is not implemented yet; running a one-shot scan at startup."); + const auto cadence = std::chrono::milliseconds(static_cast(rescan_interval_s) * 1000); + return std::max(base, std::min(default_ceiling, cadence)); +} + +std::optional OpcuaPlugin::rescan_step(int interval_s, + const std::function & now, + std::chrono::steady_clock::time_point * last_scan_end, + const std::function()> & sweep) { + if (interval_s <= 0 || !now || last_scan_end == nullptr || !sweep) { + return std::nullopt; + } + if (now() - *last_scan_end < std::chrono::seconds(interval_s)) { + return std::nullopt; + } + const auto result = sweep(); + // Stamp the END of the sweep: a legal /16 runs for minutes, and stamping its + // start would make the next one due the moment this one returned - the poll + // thread would sweep back to back and only attempt a reconnect once a sweep. + *last_scan_end = now(); + return result; +} + +std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryConfig & config, bool endpoint_configured, + const PortScanFn & scan, const IdentifyFn & identify, + const DiscoveryReporter & reporter, + const std::function & cancelled) { + if (!config.enabled) { + return std::nullopt; } + // Never override an explicitly configured endpoint: discovery must not open a + // second session on a PLC the operator already targets (and already polls). + if (endpoint_configured) { + return std::nullopt; + } + + // The pass reports through a buffer rather than straight to the log: whether + // the report is operator-visible or a DEBUG trace depends on the outcome, + // which is only known once the sweep is done. A rescan runs for the life of a + // disconnected process, so an unchanged outcome must not repeat its whole + // report (a secured-only site would log the same WARN every interval_s). + struct ReportLine { + bool warning; + std::string text; + }; + std::vector report; + std::string outcome; + const auto info_line = [&report, &outcome](const std::string & text) { + report.push_back({false, text}); + outcome += text; + outcome += '\n'; + }; + const auto warn_line = [&report, &outcome](const std::string & text) { + report.push_back({true, text}); + outcome += text; + outcome += '\n'; + }; + const auto emit = [&report, &outcome, &reporter]() { + const bool repeat = reporter.previous_outcome != nullptr && *reporter.previous_outcome == outcome; + if (reporter.previous_outcome != nullptr) { + *reporter.previous_outcome = outcome; + } + for (const auto & line : report) { + const auto & sink = repeat ? reporter.debug : (line.warning ? reporter.warn : reporter.info); + if (sink) { + sink(line.text); + } + } + }; - NetworkDiscovery discovery(discovery_config_, discovery_scan_fn_, discovery_identify_fn_); + NetworkDiscovery discovery(config, scan, identify); const auto subnets = discovery.resolve_subnets(); if (subnets.empty()) { - log_warn("OPC-UA discovery: no subnet configured and could not derive a local /24; nothing to scan."); - return; + warn_line("OPC-UA discovery: no subnet configured and could not derive a local /24; nothing to scan."); + emit(); + return std::nullopt; } std::string subnet_list; for (const auto & s : subnets) { subnet_list += (subnet_list.empty() ? "" : ", ") + s; } - log_info("OPC-UA discovery: read-only active scan of [" + subnet_list + "] on " + - std::to_string(discovery_config_.ports.size()) + " port(s)..."); + info_line("OPC-UA discovery: read-only active scan of [" + subnet_list + "] on " + + std::to_string(config.ports.size()) + " port(s)..."); - const std::vector found = discovery.run(); + const std::vector found = discovery.run(cancelled); // Summarize what was found and what was skipped (leads, LDS, secured-only). size_t data_servers = 0; @@ -1562,26 +1786,121 @@ void OpcuaPlugin::run_startup_discovery() { if (!ep.anonymous_none_available) { ++secured_only; } - log_info("OPC-UA discovery: found data server " + ep.endpoint_url + " (uri='" + ep.application_uri + - "', product='" + ep.product_uri + "', None/Anonymous=" + (ep.anonymous_none_available ? "yes" : "no") + - ")"); + info_line("OPC-UA discovery: found data server " + ep.endpoint_url + " (uri='" + ep.application_uri + + "', product='" + ep.product_uri + "', None/Anonymous=" + (ep.anonymous_none_available ? "yes" : "no") + + ")"); } - log_info("OPC-UA discovery summary: " + std::to_string(data_servers) + " data server(s), " + - std::to_string(discovery_servers) + " discovery server(s)/LDS, " + std::to_string(secured_only) + - " secured-only (need credentials), " + std::to_string(leads) + " non-OPC-UA/unidentified lead(s)."); + info_line("OPC-UA discovery summary: " + std::to_string(data_servers) + " data server(s), " + + std::to_string(discovery_servers) + " discovery server(s)/LDS, " + std::to_string(secured_only) + + " secured-only (need credentials), " + std::to_string(leads) + " non-OPC-UA/unidentified lead(s)."); - const DiscoveredEndpoint * chosen = - NetworkDiscovery::select_auto_endpoint(found, discovery_config_.anonymous_none_only); + const DiscoveredEndpoint * chosen = NetworkDiscovery::select_auto_endpoint(found, config.anonymous_none_only); if (chosen == nullptr) { - log_warn( - "OPC-UA discovery: no auto-connectable None/Anonymous data server found; leaving endpoint at default. " + warn_line( + "OPC-UA discovery: no auto-connectable None/Anonymous data server found; leaving the endpoint unchanged. " "Secured-only servers require operator credentials."); + emit(); + return std::nullopt; + } + + info_line("OPC-UA discovery: selected endpoint " + chosen->endpoint_url + " (uri='" + chosen->application_uri + "')"); + emit(); + return chosen->endpoint_url; +} + +void OpcuaPlugin::run_startup_discovery() { + if (!discovery_config_.enabled) { + return; + } + if (endpoint_configured_) { + log_info("OPC-UA discovery enabled but endpoint_url is explicitly configured (" + client_config_.endpoint_url + + "); skipping auto-discovery to avoid a second session."); + return; + } + + // The startup scan is always reported in full (no previous outcome to compare + // against) and always cancellable, so a shutdown during set_context does not + // wait out a whole sweep. + const auto chosen = discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, + discovery_identify_fn_, discovery_reporter(&last_discovery_outcome_), [this]() { + return shutdown_requested_.load(); + }); + // Stamp when the sweep FINISHED: the rescan cadence is measured from the end + // of the previous sweep, so a long sweep is not immediately followed by + // another one. + last_discovery_scan_end_ = std::chrono::steady_clock::now(); + + if (!chosen) { + // The startup scan can legitimately find nothing - a gateway that boots + // alongside its PLC routinely scans while the PLC is still coming up. With a + // cadence the endpoint stays at its default and the reconnect arm rescans, + // so this is a delay rather than a dead end. With re-scanning switched off + // (an explicit interval_s: 0) it IS the end, and the operator has to be told + // which of the two they configured. + const int startup_interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); + if (startup_interval_s > 0) { + log_info("OPC-UA discovery: startup scan selected no endpoint; the reconnect loop rescans every " + + std::to_string(startup_interval_s) + "s while down."); + } else { + log_warn( + "OPC-UA discovery: startup scan selected no endpoint and re-scanning is off (interval_s: 0); the endpoint " + "stays at " + + client_config_.endpoint_url + " until the plugin is restarted."); + } return; } - client_config_.endpoint_url = chosen->endpoint_url; - log_info("OPC-UA discovery: auto-selected endpoint " + chosen->endpoint_url + " (uri='" + chosen->application_uri + - "') - handing to the connect + introspect path."); + client_config_.endpoint_url = *chosen; + log_info("OPC-UA discovery: auto-selected endpoint " + *chosen + " - handing to the connect + introspect path."); +} + +OpcuaPlugin::DiscoveryReporter OpcuaPlugin::discovery_reporter(std::string * previous_outcome) const { + DiscoveryReporter reporter; + reporter.info = [this](const std::string & m) { + log_info(m); + }; + reporter.warn = [this](const std::string & m) { + log_warn(m); + }; + reporter.debug = [](const std::string & m) { + RCLCPP_DEBUG(opcua_plugin_logger(), "%s", m.c_str()); + }; + reporter.previous_outcome = previous_outcome; + return reporter; +} + +std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { + // A sweep is a bounded but multi-second blocking call on the poll thread, and + // stop() has to wait for whatever it is in the middle of. Do not start one the + // shutdown is going to throw away. + if (shutdown_requested_.load()) { + return std::nullopt; + } + const int interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); + + const auto chosen = rescan_step( + interval_s, + []() { + return std::chrono::steady_clock::now(); + }, + &last_discovery_scan_end_, + [this]() { + return discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, + discovery_reporter(&last_discovery_outcome_), [this]() { + return shutdown_requested_.load(); + }); + }); + // The live client config, not client_config_: this runs on the poll thread + // and client_config_ is read by the refresh thread in introspect(). The + // client owns the endpoint once connect() has been called with it, and its + // accessors are mutex-guarded. + const std::string current = client_ ? client_->endpoint_url() : client_config_.endpoint_url; + if (!chosen || *chosen == current) { + return std::nullopt; + } + + log_info("OPC-UA discovery: rescan while disconnected adopted endpoint " + *chosen + " (was " + current + ")"); + return chosen; } nlohmann::json OpcuaPlugin::build_data_response(const std::string & entity_id) const { diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp index c24f9ca0b..019d0604c 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp @@ -1142,9 +1142,27 @@ bool OpcuaPoller::comms_lost_should_raise(bool enabled, bool already_raised, return (now - down_since) >= debounce; } +std::optional +OpcuaPoller::adopt_rediscovered_endpoint(const std::string & current, + const std::function()> & rediscover) { + if (!rediscover) { + return std::nullopt; + } + const std::optional found = rediscover(); + if (!found || found->empty() || *found == current) { + return std::nullopt; + } + return found; +} + +std::chrono::milliseconds OpcuaPoller::next_reconnect_wait(std::chrono::milliseconds current, + std::chrono::milliseconds max_wait) { + return std::min(current * 2, max_wait); +} + void OpcuaPoller::emit_comms_lost(bool active) { ros2_medkit::fault_detection::FaultSignal signal; - signal.fault_code = "PLC_COMMS_LOST"; + signal.fault_code = kCommsLostFaultCode; signal.severity = config_.comms_lost_severity; signal.message = active ? ("OPC-UA connection lost to " + client_.endpoint_url()) : ("OPC-UA connection restored to " + client_.endpoint_url()); @@ -1157,7 +1175,6 @@ void OpcuaPoller::emit_comms_lost(bool active) { void OpcuaPoller::poll_loop() { auto reconnect_wait = config_.reconnect_interval; - constexpr auto max_reconnect_wait = std::chrono::milliseconds(60000); while (running_.load()) { // Handle reconnection @@ -1173,15 +1190,32 @@ void OpcuaPoller::poll_loop() { comms_down_since_ = std::chrono::steady_clock::now(); } - // Attempt reconnect with original config (preserves timeout, etc.) - if (client_.connect(client_.current_config())) { + // Reconnect with the original config (preserves timeout, security, ...). + // The endpoint is the one exception: when a rediscovery callback is bound + // and offers a different server, adopt it for this attempt. connect() + // stores the config it is given, so current_config() carries the adopted + // endpoint from here on and every later retry targets the new server. + OpcuaClientConfig reconnect_config = client_.current_config(); + if (auto adopted = adopt_rediscovered_endpoint(reconnect_config.endpoint_url, config_.rediscover_endpoint)) { + reconnect_config.endpoint_url = *adopted; + // A freshly discovered server deserves a prompt attempt: without this + // reset the backoff (up to 60 s) would keep the newly found PLC waiting + // for as long as the old dead endpoint had earned. + reconnect_wait = config_.reconnect_interval; + } + + if (client_.connect(reconnect_config)) { reconnect_wait = config_.reconnect_interval; // reset on success - // Issue #496: connection restored - clear the comms-lost fault if it - // was raised, then reset the debounce timer. - if (comms_lost_raised_) { + // Issue #496: connection restored - clear the comms-lost fault. Sent on + // EVERY successful reconnect, not only when this process raised it: the + // fault manager keys faults by fault_code and persists them, so a fault + // raised before a restart is standing in the store with nothing in + // memory to remember it. The clear is fire-and-forget and the store + // answers "not found" harmlessly when there is nothing to clear. + if (config_.comms_lost_fault_enabled) { emit_comms_lost(/*active=*/false); - comms_lost_raised_ = false; } + comms_lost_raised_ = false; comms_down_since_.reset(); if (config_.prefer_subscriptions) { setup_subscriptions(); @@ -1214,14 +1248,16 @@ void OpcuaPoller::poll_loop() { comms_lost_raised_ = true; } } - // Exponential backoff capped at 60s. condition_variable so stop() wakes immediately. + // Exponential backoff, capped at config_.max_reconnect_interval (60 s by + // default, the rescan cadence while the reconnect arm also rescans). + // condition_variable so stop() wakes immediately. { std::unique_lock lock(stop_mutex_); stop_cv_.wait_for(lock, reconnect_wait, [this] { return !running_.load(); }); } - reconnect_wait = std::min(reconnect_wait * 2, max_reconnect_wait); + reconnect_wait = next_reconnect_wait(reconnect_wait, config_.max_reconnect_interval); continue; } } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp index 03643beb3..96c32e339 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp @@ -17,6 +17,7 @@ #include #include +#include #include #include #include @@ -106,7 +107,32 @@ TEST(ParseDiscoveryConfig, DefaultsDisabled) { ASSERT_EQ(cfg.ports.size(), 1u); EXPECT_EQ(cfg.ports[0], 4840); EXPECT_TRUE(cfg.anonymous_none_only); - EXPECT_EQ(cfg.interval_s, 0); + // Unset, NOT 0: an absent key means "no cadence stated" (the caller takes its + // built-in default), while an explicit 0 means "never re-scan". + EXPECT_FALSE(cfg.interval_s.has_value()); +} + +TEST(ParseDiscoveryConfig, ExplicitZeroIntervalIsKeptAsAnExplicitZero) { + std::vector warnings; + const auto cfg = parse_discovery_config(nlohmann::json{{"interval_s", 0}}, [&](const std::string & m) { + warnings.push_back(m); + }); + ASSERT_TRUE(cfg.interval_s.has_value()); + EXPECT_EQ(*cfg.interval_s, 0); + EXPECT_TRUE(warnings.empty()); +} + +TEST(ParseDiscoveryConfig, NegativeIntervalWarnsAndLeavesTheCadenceUnset) { + std::vector warnings; + const auto cfg = parse_discovery_config(nlohmann::json{{"interval_s", -5}}, [&](const std::string & m) { + warnings.push_back(m); + }); + EXPECT_FALSE(cfg.interval_s.has_value()); + ASSERT_EQ(warnings.size(), 1u); + EXPECT_NE(warnings[0].find("interval_s"), std::string::npos); + // The warning must not tell the operator the kept default is one-shot: an + // unset interval re-scans on the built-in cadence, only an explicit 0 stops. + EXPECT_EQ(warnings[0].find("0 = one-shot"), std::string::npos) << warnings[0]; } TEST(ParseDiscoveryConfig, ReadsAllKnownKeys) { @@ -133,7 +159,8 @@ TEST(ParseDiscoveryConfig, ReadsAllKnownKeys) { EXPECT_EQ(cfg.connect_timeout_ms, 300); EXPECT_EQ(cfg.scan_concurrency, 64); EXPECT_EQ(cfg.identify_timeout_ms, 2000); - EXPECT_EQ(cfg.interval_s, 900); + ASSERT_TRUE(cfg.interval_s.has_value()); + EXPECT_EQ(*cfg.interval_s, 900); EXPECT_FALSE(cfg.anonymous_none_only); EXPECT_TRUE(warnings.empty()); } @@ -377,6 +404,91 @@ TEST(NetworkDiscoveryRun, IdentifyFailureRecordedAsLead) { EXPECT_EQ(NetworkDiscovery::select_auto_endpoint(eps, true), nullptr); } +// --------------------------------------------------------------------------- // +// run(cancelled): a shutdown must not wait out a whole sweep +// --------------------------------------------------------------------------- // +TEST(NetworkDiscoveryRun, CancelStopsTheSweepInsteadOfProbingEveryHost) { + // A /24 is 254 probes and a legal /16 is 65k; the caller runs them on the poll + // thread a shutdown has to join. With scan_concurrency 1 the sweep is + // sequential, so the probe count is exactly what the cancel predicate allowed. + std::atomic probes{0}; + std::atomic stop{false}; + auto scan = [&probes, &stop](const std::string &, uint16_t, int) { + if (probes.fetch_add(1) + 1 >= 5) { + stop.store(true); // the shutdown flag flipping mid-sweep + } + return false; + }; + OpcuaDiscoveryConfig cfg; + cfg.enabled = true; + cfg.subnets = {"192.168.1.0/24"}; + cfg.ports = {4840}; + cfg.scan_concurrency = 1; + + NetworkDiscovery disc(cfg, scan, make_identify({})); + const auto eps = disc.run([&stop]() { + return stop.load(); + }); + + EXPECT_TRUE(eps.empty()); + // One in-flight probe per worker may still complete after the flag flips. + EXPECT_GE(probes.load(), 5); + EXPECT_LE(probes.load(), 6) << "the sweep kept probing after it was cancelled"; +} + +TEST(NetworkDiscoveryRun, WithoutCancellationEveryHostIsStillProbed) { + // Positive control for the test above on the same harness: the identical + // sweep with no cancel predicate visits all 254 hosts, so a low probe count + // there is the cancellation and not a broken fake. + std::atomic probes{0}; + auto scan = [&probes](const std::string &, uint16_t, int) { + probes.fetch_add(1); + return false; + }; + OpcuaDiscoveryConfig cfg; + cfg.enabled = true; + cfg.subnets = {"192.168.1.0/24"}; + cfg.ports = {4840}; + cfg.scan_concurrency = 1; + + NetworkDiscovery disc(cfg, scan, make_identify({})); + const auto eps = disc.run(); + EXPECT_TRUE(eps.empty()); + EXPECT_EQ(probes.load(), 254); +} + +TEST(NetworkDiscoveryRun, CancelBetweenSweepAndIdentifySkipsTheIdentifyRoundTrips) { + // The identify phase is a separate batch, and each GetEndpoints blocks for up + // to identify_timeout_ms. A cancel that arrives once the sweep is done must + // not still pay for one round-trip per hit. + std::atomic stop{false}; + auto scan = [&stop](const std::string & ip, uint16_t, int) { + const bool hit = ip == "192.168.1.10" || ip == "192.168.1.11"; + if (ip == "192.168.1.254") { + stop.store(true); // sweep finished, shutdown requested + } + return hit; + }; + std::atomic identifies{0}; + auto identify = [&identifies](const std::string &, int) { + identifies.fetch_add(1); + return IdentifyResult{}; + }; + + OpcuaDiscoveryConfig cfg; + cfg.enabled = true; + cfg.subnets = {"192.168.1.0/24"}; + cfg.ports = {4840}; + cfg.scan_concurrency = 1; + + NetworkDiscovery disc(cfg, scan, identify); + const auto eps = disc.run([&stop]() { + return stop.load(); + }); + EXPECT_TRUE(eps.empty()); + EXPECT_EQ(identifies.load(), 0); +} + // --------------------------------------------------------------------------- // // select_auto_endpoint // --------------------------------------------------------------------------- // diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 48294503f..93b84ee1f 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -12,15 +12,22 @@ // See the License for the specific language governing permissions and // limitations under the License. -// INV2 end-to-end (no HW): boot the test_alarm_server OPC-UA fixture, connect, -// and prove the asset-identity nameplate is filled from the server's device-info -// (ServerStatus/BuildInfo + the OPC-UA DI DeviceSet nameplate) with no manual -// entry. Exercises both the raw OpcuaClient::read_device_info read and the full -// OpcuaPlugin::introspect() path that lands identity on the SOVD Component. +// End-to-end against a live OPC-UA server (no HW): boot the test_alarm_server +// fixture and exercise the paths that only a real session can reach. +// +// INV2 identity: prove the asset-identity nameplate is filled from the server's +// device-info (ServerStatus/BuildInfo + the OPC-UA DI DeviceSet nameplate) with +// no manual entry, through both the raw OpcuaClient::read_device_info read and +// the full OpcuaPlugin::introspect() path that lands identity on the SOVD +// Component. +// +// Connection lifecycle: prove a successful connect clears the standing +// PLC_COMMS_LOST fault, which needs a connect that actually succeeds. #include "ros2_medkit_opcua/device_identity.hpp" #include "ros2_medkit_opcua/opcua_client.hpp" #include "ros2_medkit_opcua/opcua_plugin.hpp" +#include "ros2_medkit_opcua/opcua_poller.hpp" #include @@ -33,13 +40,16 @@ #include #include +#include #include #include #include #include +#include #include #include #include +#include #include #include @@ -559,4 +569,66 @@ TEST_F(OpcuaIdentityE2ETest, DiNameplateReadFollowsBrowseContinuationPoints) { client.disconnect(); } +// A gateway that restarts after a comms outage never raised PLC_COMMS_LOST in +// THIS process, yet the fault manager keys faults by fault_code alone and +// persists them, so the fault raised before the restart is still standing. +// The reconnect arm used to clear only when its own in-memory +// ``comms_lost_raised_`` flag was set, which no restart can satisfy, so the +// fault stayed CONFIRMED for good. The clear now goes out on every successful +// connect. Driven against the live fixture because the arm can only be reached +// by a connect that actually succeeds. +TEST_F(OpcuaIdentityE2ETest, SuccessfulConnectClearsCommsLostNeverRaisedHere) { + OpcuaClient client; + OpcuaClientConfig config; + config.endpoint_url = endpoint_; + config.connect_timeout = std::chrono::milliseconds(5000); + // Connect once to seed the client's stored config (what the poller reconnects + // with), then drop the session so the poll loop starts in its reconnect arm - + // the state a freshly started gateway is in while the PLC is already up. + ASSERT_TRUE(client.connect(config)); + client.disconnect(); + ASSERT_FALSE(client.is_connected()); + + NodeMap node_map; // config-less: no entries, nothing to poll + OpcuaPoller poller(client, node_map); + + std::mutex signals_mutex; + std::vector> signals; // (fault_code, active) + poller.set_alarm_callback( + [&signals_mutex, &signals](const std::string &, const ros2_medkit::fault_detection::FaultSignal & signal) { + std::lock_guard lock(signals_mutex); + signals.emplace_back(signal.fault_code, signal.active); + }); + + PollerConfig poller_config; + poller_config.poll_interval = std::chrono::milliseconds(100); + poller_config.reconnect_interval = std::chrono::milliseconds(100); + poller_config.comms_lost_fault_enabled = true; + poller.start(poller_config); + + bool cleared = false; + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(15); + while (!cleared && std::chrono::steady_clock::now() < deadline) { + { + std::lock_guard lock(signals_mutex); + cleared = std::find(signals.begin(), signals.end(), std::make_pair(std::string(kCommsLostFaultCode), false)) != + signals.end(); + } + if (!cleared) { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + } + poller.stop(); + + EXPECT_TRUE(cleared) << "a successful connect must clear PLC_COMMS_LOST even when this process never raised it"; + + // Absence control on the same harness: the connect succeeded, so nothing may + // have RAISED the fault. Without this a clear-everything-always regression + // would still pass the assertion above. + std::lock_guard lock(signals_mutex); + EXPECT_EQ(std::find(signals.begin(), signals.end(), std::make_pair(std::string(kCommsLostFaultCode), true)), + signals.end()) + << "comms-lost must not be raised while the connection is up"; +} + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp index 0c6b58b64..84d17f99e 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp @@ -27,7 +27,11 @@ #include #include #include +#include +#include #include +#include +#include #include #include #include @@ -800,6 +804,518 @@ TEST(CommsLostShouldRaise, IdempotentAndDisabled) { EXPECT_FALSE(OpcuaPoller::comms_lost_should_raise(/*enabled=*/false, false, t0, late, debounce)); } +// --------------------------------------------------------------------------- +// Endpoint rediscovery while disconnected (config-less discovery) +// --------------------------------------------------------------------------- + +namespace { + +// Fake port scanner backed by a set of open "ip:port" hosts. +PortScanFn fake_scan(std::set open) { + return [open = std::move(open)](const std::string & ip, uint16_t port, int) { + return open.count(ip + ":" + std::to_string(port)) > 0; + }; +} + +// Fake GetEndpoints identify keyed by connect URL. Anything else is unreachable. +IdentifyFn fake_identify(std::map table) { + return [table = std::move(table)](const std::string & url, int) -> IdentifyResult { + const auto it = table.find(url); + if (it != table.end()) { + return it->second; + } + IdentifyResult r; + r.error = "unreachable"; + return r; + }; +} + +IdentifyResult plc_identity() { + IdentifyResult r; + r.ok = true; + r.advertised_url = "opc.tcp://192.168.1.10:4840"; + r.application_uri = "urn:SIMATIC.S7-1500.OPC-UA.Application:Software PLC_1"; + r.product_uri = "https://www.siemens.com/s7-1500"; + r.application_name = "SIMATIC.S7-1500"; + r.application_type = 0; // Server + r.security_policies = {{"None", 1}}; + r.anonymous_none_available = true; + return r; +} + +OpcuaDiscoveryConfig rescan_cfg() { + OpcuaDiscoveryConfig cfg; + cfg.enabled = true; + cfg.subnets = {"192.168.1.0/24"}; // explicit, so no local-interface derivation + cfg.ports = {4840}; + return cfg; +} + +// Discards log output. The tests assert on the selected endpoint, not the text. +const std::function kSilent = [](const std::string &) {}; + +// Silent reporter: no repeat-suppression memory, so every pass reports in full +// (into the void). Tests that assert on the log build their own. +OpcuaPlugin::DiscoveryReporter silent_reporter() { + OpcuaPlugin::DiscoveryReporter reporter; + reporter.info = kSilent; + reporter.warn = kSilent; + return reporter; +} + +} // namespace + +TEST(DiscoverEndpoint, ScanBeforeThePlcIsUpSelectsNothingAndALaterRescanAdoptsIt) { + // The field race: the gateway scans 2 s after start while the PLC is still + // booting. Nothing answers, so nothing is selected and the caller keeps the + // default endpoint. + const auto empty_pass = OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({}), + fake_identify({}), silent_reporter()); + EXPECT_FALSE(empty_pass.has_value()); + + // The PLC finishes booting. The same call with the same config now finds it, + // which is what the reconnect arm applies to the next connect attempt. + const auto later_pass = OpcuaPlugin::discover_endpoint( + rescan_cfg(), /*endpoint_configured=*/false, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), silent_reporter()); + ASSERT_TRUE(later_pass.has_value()); + EXPECT_EQ(*later_pass, "opc.tcp://192.168.1.10:4840"); +} + +TEST(DiscoverEndpoint, AnExplicitEndpointIsNeverRescanned) { + // Positive control: the very scan that DOES find a server above finds the + // same server here, and is still refused because the operator pinned an + // endpoint. Discovery must not open a second session on a polled PLC. + const auto chosen = OpcuaPlugin::discover_endpoint( + rescan_cfg(), /*endpoint_configured=*/true, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), silent_reporter()); + EXPECT_FALSE(chosen.has_value()); +} + +TEST(DiscoverEndpoint, DisabledDiscoveryScansNothing) { + OpcuaDiscoveryConfig cfg = rescan_cfg(); + cfg.enabled = false; + bool scanned = false; + auto counting_scan = [&scanned](const std::string &, uint16_t, int) { + scanned = true; + return true; + }; + const auto chosen = OpcuaPlugin::discover_endpoint(cfg, /*endpoint_configured=*/false, counting_scan, + fake_identify({}), silent_reporter()); + EXPECT_FALSE(chosen.has_value()); + EXPECT_FALSE(scanned) << "a disabled discovery must not touch the network"; +} + +TEST(EffectiveRescanInterval, DefaultsWhenDiscoveryIsOnWithNoCadenceAndIsOffOtherwise) { + OpcuaDiscoveryConfig cfg = rescan_cfg(); + // Config-less: discovery on, no interval stated -> the built-in cadence, not + // "never rescan". This is the deployment that most needs the rescan. + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(cfg, /*endpoint_configured=*/false), + OpcuaPlugin::kDefaultRescanIntervalS); + // An operator-stated cadence wins. + cfg.interval_s = 120; + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(cfg, false), 120); + // An explicit endpoint, or discovery off, means no rescan at all. + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(cfg, /*endpoint_configured=*/true), 0); + cfg.enabled = false; + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(cfg, false), 0); +} + +TEST(EffectiveRescanInterval, ExplicitZeroKeepsDiscoveryOnAndStopsRescanning) { + // The three states an operator can be in, all with discovery enabled and no + // endpoint pinned. + OpcuaDiscoveryConfig unset = rescan_cfg(); // (1) unset -> the built-in cadence + EXPECT_FALSE(unset.interval_s.has_value()); + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(unset, false), OpcuaPlugin::kDefaultRescanIntervalS); + + OpcuaDiscoveryConfig explicit_zero = rescan_cfg(); // (2) explicit 0 -> one-shot + explicit_zero.interval_s = 0; + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(explicit_zero, false), 0) + << "an explicit interval_s: 0 must stop the rescan, not fall back to the default"; + + // (3) A negative value never reaches here: the parse warns and leaves the + // cadence unset, so what arrives is case (1). + std::vector warnings; + const auto parsed = + parse_discovery_config(nlohmann::json{{"enabled", true}, {"interval_s", -1}}, [&warnings](const std::string & m) { + warnings.push_back(m); + }); + EXPECT_EQ(warnings.size(), 1u); + EXPECT_EQ(OpcuaPlugin::effective_rescan_interval_s(parsed, false), OpcuaPlugin::kDefaultRescanIntervalS); +} + +// --------------------------------------------------------------------------- +// Rescan cadence: measured from the END of the previous sweep +// --------------------------------------------------------------------------- + +TEST(RescanStep, SpacesSweepsFromTheEndOfThePreviousOne) { + // A legal /16 sweep runs for minutes. With the cadence stamped at the START, + // the next sweep is due the instant the current one returns, so the poll + // thread sweeps back to back and the reconnect attempt drops to one a sweep. + const auto t0 = std::chrono::steady_clock::time_point{}; + const auto sweep_duration = std::chrono::seconds(390); // a /16 at the defaults + auto clock_now = t0; + const auto now = [&clock_now]() { + return clock_now; + }; + + int sweeps = 0; + const auto sweep = [&sweeps, &clock_now, sweep_duration]() -> std::optional { + ++sweeps; + clock_now += sweep_duration; // the sweep blocks for its whole duration + return std::nullopt; + }; + + auto last_end = t0; + clock_now = t0 + std::chrono::seconds(30); + OpcuaPlugin::rescan_step(30, now, &last_end, sweep); + ASSERT_EQ(sweeps, 1); + EXPECT_EQ(last_end, clock_now) << "the cadence must be stamped when the sweep finished"; + + // One second after the sweep returned: not due, even though it STARTED 391 s + // ago. + clock_now += std::chrono::seconds(1); + OpcuaPlugin::rescan_step(30, now, &last_end, sweep); + EXPECT_EQ(sweeps, 1) << "a rescan ran less than one interval after the previous sweep ended"; + + // A full interval after the end: due again. + clock_now += std::chrono::seconds(29); + OpcuaPlugin::rescan_step(30, now, &last_end, sweep); + EXPECT_EQ(sweeps, 2); +} + +TEST(RescanStep, DoesNothingWithoutACadence) { + const auto t0 = std::chrono::steady_clock::time_point{}; + auto last_end = t0; + int sweeps = 0; + const auto now = [t0]() { + return t0 + std::chrono::hours(1); + }; + const auto sweep = [&sweeps]() -> std::optional { + ++sweeps; + return std::string("opc.tcp://192.168.1.10:4840"); + }; + // 0 is the operator's "do not re-scan" (and also discovery off / endpoint + // pinned, both of which effective_rescan_interval_s maps to 0). + EXPECT_FALSE(OpcuaPlugin::rescan_step(0, now, &last_end, sweep).has_value()); + EXPECT_EQ(sweeps, 0); + // Positive control on the same harness: with a cadence the very same call + // sweeps and hands the endpoint back. + const auto adopted = OpcuaPlugin::rescan_step(30, now, &last_end, sweep); + ASSERT_TRUE(adopted.has_value()); + EXPECT_EQ(*adopted, "opc.tcp://192.168.1.10:4840"); + EXPECT_EQ(sweeps, 1); +} + +// --------------------------------------------------------------------------- +// Reconnect backoff ceiling while the reconnect arm also rescans +// --------------------------------------------------------------------------- + +TEST(EffectiveMaxReconnectWait, CapsTheBackoffAtTheRescanCadence) { + using namespace std::chrono_literals; + // No rescan: the plain 60 s ceiling. + EXPECT_EQ(OpcuaPlugin::effective_max_reconnect_wait(5000ms, 60000ms, /*rescan_interval_s=*/0), 60000ms); + // Rescanning every 30 s: an uncapped backoff would make the real adoption + // cadence max(30 s, 60 s), not the documented 30 s. + EXPECT_EQ(OpcuaPlugin::effective_max_reconnect_wait(5000ms, 60000ms, 30), 30000ms); + // A cadence longer than the ceiling does not raise the ceiling. + EXPECT_EQ(OpcuaPlugin::effective_max_reconnect_wait(5000ms, 60000ms, 900), 60000ms); + // A cadence shorter than the configured reconnect interval does not turn the + // backoff into a hot retry loop. + EXPECT_EQ(OpcuaPlugin::effective_max_reconnect_wait(5000ms, 60000ms, 1), 5000ms); +} + +TEST(NextReconnectWait, DoublesUpToTheCeiling) { + using namespace std::chrono_literals; + EXPECT_EQ(OpcuaPoller::next_reconnect_wait(5000ms, 60000ms), 10000ms); + EXPECT_EQ(OpcuaPoller::next_reconnect_wait(40000ms, 60000ms), 60000ms); + EXPECT_EQ(OpcuaPoller::next_reconnect_wait(60000ms, 60000ms), 60000ms); + // Capped at a 30 s rescan cadence: the wait never exceeds it, so the rescan is + // consulted every cadence instead of every max(cadence, backoff). + EXPECT_EQ(OpcuaPoller::next_reconnect_wait(20000ms, 30000ms), 30000ms); + EXPECT_EQ(OpcuaPoller::next_reconnect_wait(30000ms, 30000ms), 30000ms); +} + +// --------------------------------------------------------------------------- +// Discovery report: quiet while the outcome does not change +// --------------------------------------------------------------------------- + +TEST(DiscoverEndpoint, AnUnchangedRescanReportsAtDebugInsteadOfRepeatingItself) { + // A secured-only site rescans for the life of the process and would otherwise + // log the whole report - scan line, per-server line, summary and the + // "no auto-connectable server" WARN - every interval_s. + IdentifyResult secured = plc_identity(); + secured.anonymous_none_available = false; + + std::vector info; + std::vector warn; + std::vector debug; + std::string outcome; + OpcuaPlugin::DiscoveryReporter reporter; + reporter.info = [&info](const std::string & m) { + info.push_back(m); + }; + reporter.warn = [&warn](const std::string & m) { + warn.push_back(m); + }; + reporter.debug = [&debug](const std::string & m) { + debug.push_back(m); + }; + reporter.previous_outcome = &outcome; + + const auto pass = [&]() { + return OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", secured}}), reporter); + }; + + EXPECT_FALSE(pass().has_value()); + const size_t first_info = info.size(); + const size_t first_warn = warn.size(); + EXPECT_GT(first_info, 0u); + EXPECT_EQ(first_warn, 1u) << "the first pass always reports the secured-only outcome"; + EXPECT_TRUE(debug.empty()); + + // Same network, same outcome: nothing new at INFO/WARN, the report goes to + // the debug logger instead. + EXPECT_FALSE(pass().has_value()); + EXPECT_EQ(info.size(), first_info) << "an unchanged rescan repeated its report at INFO"; + EXPECT_EQ(warn.size(), first_warn) << "an unchanged rescan repeated its WARN"; + EXPECT_EQ(debug.size(), first_info + first_warn) << "the repeated report must still be traceable at DEBUG"; + + // The server opens up an anonymous endpoint: the outcome changed, so the + // operator hears about it at INFO again. + const auto chosen = + OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), reporter); + ASSERT_TRUE(chosen.has_value()); + EXPECT_GT(info.size(), first_info) << "a changed outcome must be reported at INFO"; +} + +TEST(DiscoverEndpoint, WithNoRepeatMemoryEveryPassIsReported) { + // Positive control for the test above: the same two identical passes with no + // previous_outcome (the startup scan's own reporter) report in full twice, so + // the silence above is the suppression and not a dead sink. + std::vector info; + OpcuaPlugin::DiscoveryReporter reporter; + reporter.info = [&info](const std::string & m) { + info.push_back(m); + }; + reporter.warn = kSilent; + + const auto pass = [&]() { + return OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), reporter); + }; + EXPECT_TRUE(pass().has_value()); + const size_t first = info.size(); + EXPECT_GT(first, 0u); + EXPECT_TRUE(pass().has_value()); + EXPECT_EQ(info.size(), 2 * first); +} + +// --------------------------------------------------------------------------- +// Config-less component identity across an adoption +// --------------------------------------------------------------------------- + +TEST(RederivedComponentIdentity, AdoptionReplacesTheProvisionalEndpointDerivedId) { + // The config-less race, end to end over the derivation path: the gateway + // starts before the PLC, its start-up scan finds nothing, and the identity is + // derived from the fallback endpoint plus an empty DeviceInfo. + const auto startup_pass = OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({}), + fake_identify({}), silent_reporter()); + ASSERT_FALSE(startup_pass.has_value()); + const std::string fallback_endpoint = "opc.tcp://localhost:4840"; // OpcuaClientConfig's default + const ComponentIdentity provisional = derive_component_identity(OpcuaClient::DeviceInfo{}, fallback_endpoint); + EXPECT_EQ(provisional.id, "opcua-localhost"); + + // The PLC finishes booting and the rescan adopts it. + const auto adopted = OpcuaPlugin::discover_endpoint( + rescan_cfg(), /*endpoint_configured=*/false, fake_scan({"192.168.1.10:4840"}), + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), silent_reporter()); + ASSERT_TRUE(adopted.has_value()); + + // The session is up, so the device can finally name itself: the component + // must stop being served under the placeholder. + OpcuaClient::DeviceInfo info; + info.di_manufacturer = "Siemens AG"; + info.di_model = "CPU 1505SP F"; + const auto rederived = OpcuaPlugin::rederived_component_identity(provisional.id, info, *adopted); + ASSERT_TRUE(rederived.has_value()) << "an adopted device with a nameplate must replace opcua-localhost"; + EXPECT_EQ(rederived->id, "siemens_ag_cpu_1505sp_f"); + EXPECT_EQ(rederived->name, "Siemens AG CPU 1505SP F"); +} + +TEST(RederivedComponentIdentity, KeepsTheIdentityWhenNothingChanged) { + // Same device on a later reconnect: no rename, so no entity churn and no INFO + // line claiming an identity change that did not happen. + OpcuaClient::DeviceInfo info; + info.di_manufacturer = "Siemens AG"; + info.di_model = "CPU 1505SP F"; + EXPECT_FALSE(OpcuaPlugin::rederived_component_identity("siemens_ag_cpu_1505sp_f", info, "opc.tcp://192.168.1.10:4840") + .has_value()); + + // A nameplate-less server on an adopted endpoint still moves off the + // fallback host it was provisionally named after. + const auto host_derived = OpcuaPlugin::rederived_component_identity("opcua-localhost", OpcuaClient::DeviceInfo{}, + "opc.tcp://192.168.1.10:4840"); + ASSERT_TRUE(host_derived.has_value()); + EXPECT_EQ(host_derived->id, "opcua-192_168_1_10"); +} + +// --------------------------------------------------------------------------- +// ClearFault: a link-state clear does not cascade +// --------------------------------------------------------------------------- + +TEST(MakeClearFaultRequest, LinkStateClearSkipsTheCorrelationCascade) { + // The connect-time PLC_COMMS_LOST clear says the link came back. A + // correlation rule may name PLC_COMMS_LOST as the root cause of every symptom + // the outage produced, and clearing those is an operator's call, not a link + // event's. + const auto link_state = OpcuaPlugin::make_clear_fault_request(kCommsLostFaultCode, /*link_state=*/true); + EXPECT_EQ(link_state.fault_code, kCommsLostFaultCode); + EXPECT_TRUE(link_state.skip_correlation_auto_clear); + + // Positive control on the same request builder: an operator-driven clear (the + // SOVD DELETE route) leaves the cascade alone, so the flag above is the + // link-state rule and not a hardcoded true. + const auto operator_clear = OpcuaPlugin::make_clear_fault_request("PLC_TANK_HIGH", /*link_state=*/false); + EXPECT_EQ(operator_clear.fault_code, "PLC_TANK_HIGH"); + EXPECT_FALSE(operator_clear.skip_correlation_auto_clear); +} + +// --------------------------------------------------------------------------- +// Pending fault dispatch buffer: reports outrank clears +// --------------------------------------------------------------------------- + +namespace { + +OpcuaPlugin::PendingFaultDispatch report_entry(const std::string & code) { + return {OpcuaPlugin::PendingFaultDispatch::Kind::Report, code, []() {}}; +} + +OpcuaPlugin::PendingFaultDispatch clear_entry(const std::string & code) { + return {OpcuaPlugin::PendingFaultDispatch::Kind::Clear, code, []() {}}; +} + +size_t count_kind(const std::vector & buffer, + OpcuaPlugin::PendingFaultDispatch::Kind kind) { + return static_cast( + std::count_if(buffer.begin(), buffer.end(), [kind](const OpcuaPlugin::PendingFaultDispatch & entry) { + return entry.kind == kind; + })); +} + +} // namespace + +TEST(EnqueuePendingDispatch, ReconnectClearsNeverEvictABufferedAlarmReport) { + // A flapping link with no fault_manager: 300 reconnects, each enqueueing a + // connect-time clear, while ten real alarm reports wait to be flushed. The + // reports are one-shot edges from the PLC; the clears are re-derivable. + std::vector buffer; + for (int i = 0; i < 10; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + report_entry("PLC_ALARM_" + std::to_string(i))); + } + for (int i = 0; i < 300; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry(kCommsLostFaultCode)); + } + + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Report), 10u) + << "connect-time clears evicted buffered alarm reports"; + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 1u) + << "at most one clear per fault code may be pending"; + for (int i = 0; i < 10; ++i) { + EXPECT_EQ(buffer[static_cast(i)].fault_code, "PLC_ALARM_" + std::to_string(i)); + } +} + +TEST(EnqueuePendingDispatch, AFullReportBufferRefusesAClearInsteadOfDroppingAReport) { + std::vector buffer; + for (size_t i = 0; i < OpcuaPlugin::kMaxPendingDispatches; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + report_entry("PLC_ALARM_" + std::to_string(i))); + } + ASSERT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); + + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + clear_entry(kCommsLostFaultCode)), + OpcuaPlugin::PendingEnqueueOutcome::Refused); + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Report), OpcuaPlugin::kMaxPendingDispatches); + EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_0") << "the oldest report must survive an incoming clear"; + + // A report arriving at a full buffer still drops the oldest one: reports do + // not outrank each other, so the bound still holds. + EXPECT_EQ( + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_NEW")), + OpcuaPlugin::PendingEnqueueOutcome::EvictedReport); + EXPECT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); + EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_1"); + EXPECT_EQ(buffer.back().fault_code, "PLC_ALARM_NEW"); +} + +TEST(EnqueuePendingDispatch, AFullBufferGivesUpAPendingClearBeforeAReport) { + std::vector buffer; + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_OLD_CLEAR")); + for (size_t i = 1; i < OpcuaPlugin::kMaxPendingDispatches; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + report_entry("PLC_ALARM_" + std::to_string(i))); + } + ASSERT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); + + EXPECT_EQ( + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_NEW")), + OpcuaPlugin::PendingEnqueueOutcome::EvictedClear); + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 0u); + EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_1") << "the clear went, not the oldest report"; +} + +TEST(EnqueuePendingDispatch, ARequeuedClearMovesToTheBackSoOrderStillHolds) { + // Report-then-clear for one code must still flush in that order after the + // clear is re-enqueued, or the flush would leave the fault standing. + std::vector buffer; + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_FLAP")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_FLAP")); + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_FLAP")), + OpcuaPlugin::PendingEnqueueOutcome::ReplacedClear); + + ASSERT_EQ(buffer.size(), 2u); + EXPECT_EQ(buffer[0].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Report); + EXPECT_EQ(buffer[1].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear) + << "the newest clear must flush after the report it supersedes"; + // Clears for DIFFERENT codes are independent. + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_OTHER")); + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 2u); +} + +TEST(AdoptRediscoveredEndpoint, AdoptsOnlyADifferentNonEmptyUrl) { + const std::string current = "opc.tcp://localhost:4840"; + + // No callback bound (an explicit endpoint, or discovery off) -> keep current. + EXPECT_FALSE(OpcuaPoller::adopt_rediscovered_endpoint(current, nullptr).has_value()); + + // Rescan not due, or found nothing -> keep current. + EXPECT_FALSE(OpcuaPoller::adopt_rediscovered_endpoint(current, [] { + return std::optional{}; + }).has_value()); + + // Same server as before -> nothing to adopt, so no needless reconnect churn. + EXPECT_FALSE(OpcuaPoller::adopt_rediscovered_endpoint(current, [¤t] { + return std::optional{current}; + }).has_value()); + + // An empty URL is not an endpoint. + EXPECT_FALSE(OpcuaPoller::adopt_rediscovered_endpoint(current, [] { + return std::optional{""}; + }).has_value()); + + // A different server -> adopt it for the next connect attempt. + const auto adopted = OpcuaPoller::adopt_rediscovered_endpoint(current, [] { + return std::optional{"opc.tcp://192.168.1.10:4840"}; + }); + ASSERT_TRUE(adopted.has_value()); + EXPECT_EQ(*adopted, "opc.tcp://192.168.1.10:4840"); +} + // Issue #478 safety-gate: an empty scan from a source that has NEVER yielded a // condition instance node (EventNotifier-only server, e.g. S7-1500) must NOT // clear the still-active tracked fault. This is the single most important From 1117e3960b6e2ebf3d7fc94fc7574535ffcf1f5b Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sun, 6 Sep 2026 12:43:01 +0200 Subject: [PATCH 02/25] fix(gateway): say where a plugin entity's freeze-frame values came from A freeze-frame captured for a plugin-backed entity reaches a client with an empty topic and an empty message_type. Both are correct. The values are the plugin's live entity data, and no ROS message carries them. But no field names the origin of the snapshot. Two such frames from different bridges look the same. A frame also looks like a topic capture that lost its metadata. Frame now carries the capture path that read it, and the gateway serves it as x-medkit.source. The value is plugin_data_provider for a read through the owning plugin's DataProvider. It is plugin_x_plc_data_route for the in-process dispatch of the plugin's own x-plc-data route, which bridges without a DataProvider use. topic and message_type stay empty, because neither names a ROS topic here. When the capture names no path, the key is absent from the payload. A fault manager freeze-frame taken from a real topic is unchanged and carries its topic and message_type as before. The snapshots tutorial describes the field. --- docs/tutorials/snapshots.rst | 21 ++++++++ .../entity_freeze_frame_capture.hpp | 19 +++++++- .../src/entity_freeze_frame_capture.cpp | 7 +-- .../src/http/handlers/fault_handlers.cpp | 13 +++++ .../test/test_entity_freeze_frame_capture.cpp | 31 ++++++++++++ .../test/test_fault_handlers.cpp | 48 +++++++++++++++++++ 6 files changed, 134 insertions(+), 5 deletions(-) diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index 1070aa373..98f54b757 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -239,6 +239,26 @@ with: ros2 run ros2_medkit_gateway gateway_node --ros-args \ -p entity_freeze_frame.enabled:=false +A plugin entity's values are not a ROS message, so ``topic`` and +``message_type`` are empty on these frames. ``x-medkit.source`` names the +capture path instead, so a consumer can still tell where the values came +from: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - ``x-medkit.source`` + - Meaning + * - ``plugin_data_provider`` + - Read through the owning plugin's ``DataProvider::list_data``. + * - ``plugin_x_plc_data_route`` + - Read by dispatching the owning plugin's own ``x-plc-data`` route + in-process (plugins that export no ``DataProvider``). + +The field is absent on freeze-frames captured by the fault manager from a ROS +topic. Those carry a real ``topic`` and ``message_type`` instead. + Example plugin-entity freeze-frame in the fault response: .. code-block:: json @@ -250,6 +270,7 @@ Example plugin-entity freeze-frame in the fault response: "x-medkit": { "topic": "", "message_type": "", + "source": "plugin_x_plc_data_route", "full_data": {"tank_level": 87.5, "pump_running": true}, "captured_at": "2026-07-14T12:00:00.000Z" } diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp index 7dc081dc0..d3c041ee4 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp @@ -60,6 +60,13 @@ namespace ros2_medkit_gateway { */ class EntityFreezeFrameCapture { public: + /// Capture-path identifiers stored in Frame::source and served as + /// ``x-medkit.source``. The plugin's own DataProvider, and the in-process + /// dispatch of the plugin's `x-plc-data` route for plugins that export no + /// DataProvider. + static constexpr const char * kSourceDataProvider = "plugin_data_provider"; + static constexpr const char * kSourceXPlcDataRoute = "plugin_x_plc_data_route"; + /// One captured frame: the entity's data values at fault-confirm time. /// captured_at_ns dates the capture, not the values - a disconnected entity /// serves its last known values, whose age is bounded only by the outage. @@ -72,6 +79,13 @@ class EntityFreezeFrameCapture { bool startup_catchup{false}; std::optional connected; ///< payload's top-level link flag, when reported nlohmann::json source_timestamp; ///< payload's own "timestamp" field verbatim (null when absent) + /// Which capture path read the values (kSourceDataProvider / + /// kSourceXPlcDataRoute), served as ``x-medkit.source``. These values are + /// entity data, not a ROS message, so ``topic`` and ``message_type`` are + /// empty on the wire and would otherwise leave a consumer with nothing at + /// all saying where the numbers came from. Empty when the caller named no + /// path. + std::string source; }; /// Resolves an entity id to its owning plugin's DataProvider (nullptr when @@ -187,9 +201,10 @@ class EntityFreezeFrameCapture { bool capture_for_event(const ros2_medkit_msgs::msg::FaultEvent & event, bool startup_catchup = false); /// Build a frame from list-data-shaped content, enforcing the shared - /// no-row-of-nulls invariant on both capture paths. + /// no-row-of-nulls invariant on both capture paths. @p source names the path + /// that read the content and is stored verbatim in Frame::source. std::optional frame_from_content(const std::string & entity_id, const std::string & fault_code, - const nlohmann::json & content); + const nlohmann::json & content, const std::string & source); /// Capture via the plugin's own x-plc-data route (no DataProvider exported). /// Returns nullopt when the route yields nothing usable. diff --git a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp index fe060826c..01bde9674 100644 --- a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp @@ -188,13 +188,14 @@ EntityFreezeFrameCapture::standing_faults_from_list_reply(const nlohmann::json & std::optional EntityFreezeFrameCapture::frame_from_content(const std::string & entity_id, const std::string & fault_code, - const nlohmann::json & content) { + const nlohmann::json & content, const std::string & source) { if (!content_has_live_data(content)) { log_fallback_failure_once(fault_code, "entity '" + entity_id + "' returned no data items"); return std::nullopt; } Frame frame; frame.entity_id = entity_id; + frame.source = source; frame.values = values_from_list_content(content); if (!values_have_data(frame.values)) { // Items present but nothing usable in them (all-null values, or no usable @@ -228,7 +229,7 @@ EntityFreezeFrameCapture::capture_via_route(const std::string & entity_id, const if (!content) { return std::nullopt; // not plugin-owned, no x-plc-data route, or handler error } - return frame_from_content(entity_id, fault_code, *content); + return frame_from_content(entity_id, fault_code, *content, kSourceXPlcDataRoute); } void EntityFreezeFrameCapture::log_fallback_failure_once(const std::string & fault_code, const std::string & message) { @@ -410,7 +411,7 @@ bool EntityFreezeFrameCapture::capture_for_event(const ros2_medkit_msgs::msg::Fa log_fallback_failure_once(fault_code, "list_data('" + source + "') failed: " + result.error().message); continue; } - if (auto frame = frame_from_content(source, fault_code, result->content)) { + if (auto frame = frame_from_content(source, fault_code, result->content, kSourceDataProvider)) { frames.push_back(std::move(*frame)); } } catch (const std::exception & e) { diff --git a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp index b235acc53..c7282fd7b 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp @@ -263,6 +263,13 @@ json FaultHandlers::merge_entity_freeze_frames(json env_data, snap["topic"] = ""; // entity data values, not a ROS topic snap["message_type"] = ""; snap["captured_at_ns"] = frame.captured_at_ns; + // Capture provenance. topic/message_type stay empty because these values + // are not a ROS message, which leaves "source" as the only field naming + // where the numbers came from - so carry it whenever the capture named a + // path. + if (!frame.source.empty()) { + snap["source"] = frame.source; + } if (frame.startup_catchup) { // Values were read at gateway start, not when the fault confirmed; // absent marker = captured on the confirm edge. @@ -344,6 +351,12 @@ dto::FaultDetail FaultHandlers::build_sovd_fault_response(const json & fault_jso snap["x-medkit"]["capture_origin"] = s["capture_origin"]; } // Entity-frame provenance (merge_entity_freeze_frames), only when known. + // "source" names the capture path (a plugin DataProvider or the + // plugin's x-plc-data route). A consumer reads it instead of the + // empty topic/message_type an entity frame necessarily carries. + if (s.contains("source") && s["source"].is_string()) { + snap["x-medkit"]["source"] = s["source"]; + } if (s.contains("connected") && s["connected"].is_boolean()) { snap["x-medkit"]["connected"] = s["connected"]; } diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index b9a44b2e0..152297ba0 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -787,6 +787,37 @@ TEST(MergeEntityFreezeFrames, AppendsWhenNoConfiguredFreezeFrame) { EXPECT_FALSE(snap.contains("capture_origin")); // confirm-edge frames carry no marker } +TEST(MergeEntityFreezeFrames, CarriesCapturePathAsSource) { + // An entity frame has no ROS topic, so topic/message_type are necessarily + // empty, so "source" is the only field left saying where they came from. + json env_data = {{"snapshots", json::array()}}; + EntityFreezeFrameCapture::Frame frame; + frame.entity_id = "plc_app"; + frame.values = {{"temperature", 42.5}}; + frame.captured_at_ns = 1234; + frame.source = EntityFreezeFrameCapture::kSourceXPlcDataRoute; + + auto merged = FaultHandlers::merge_entity_freeze_frames(env_data, {frame}); + ASSERT_EQ(merged["snapshots"].size(), 1u); + const auto & snap = merged["snapshots"][0]; + EXPECT_EQ(snap["source"], EntityFreezeFrameCapture::kSourceXPlcDataRoute); + EXPECT_EQ(snap["topic"], ""); + EXPECT_EQ(snap["message_type"], ""); +} + +TEST(MergeEntityFreezeFrames, OmitsSourceWhenTheCaptureNamedNoPath) { + // Absence control for the test above, on the same harness: a frame whose + // capture path is unknown must not have one invented for it. + json env_data = {{"snapshots", json::array()}}; + EntityFreezeFrameCapture::Frame frame; + frame.entity_id = "plc_app"; + frame.values = {{"temperature", 42.5}}; + + auto merged = FaultHandlers::merge_entity_freeze_frames(env_data, {frame}); + ASSERT_EQ(merged["snapshots"].size(), 1u); + EXPECT_FALSE(merged["snapshots"][0].contains("source")); +} + TEST(MergeEntityFreezeFrames, StartupCatchUpFrameCarriesCaptureOrigin) { json env_data = {{"snapshots", json::array()}}; EntityFreezeFrameCapture::Frame frame; diff --git a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp index 9c7f08a91..29b53ef22 100644 --- a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp @@ -142,6 +142,54 @@ TEST_F(FaultHandlersTest, BuildSovdFaultResponsePropagatesCaptureOrigin) { EXPECT_EQ(snap["x-medkit"]["capture_origin"], "startup"); } +TEST_F(FaultHandlersTest, BuildSovdFaultResponseServesEntityFrameSource) { + // A plugin-captured entity frame reaches the wire with an empty topic and + // message_type (the values are not a ROS message) plus x-medkit.source + // naming the capture path that read them. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "PLC_ALARM"; + + json env_data = {{"snapshots", json::array({{{"type", "freeze_frame"}, + {"snapshot_type", "freeze_frame"}, + {"name", "plc_app"}, + {"data", R"({"tank_level": 87.5})"}, + {"topic", ""}, + {"message_type", ""}, + {"captured_at_ns", 1234}, + {"source", "plugin_x_plc_data_route"}}})}}; + + auto response = to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_data, "/apps/plc_app")); + + auto & snap = response["environment_data"]["snapshots"][0]; + EXPECT_EQ(snap["x-medkit"]["source"], "plugin_x_plc_data_route"); + EXPECT_EQ(snap["x-medkit"]["topic"], ""); + EXPECT_EQ(snap["x-medkit"]["message_type"], ""); +} + +TEST_F(FaultHandlersTest, BuildSovdFaultResponseOmitsSourceWhenTheSnapshotHasNone) { + // Absence control for the test above, on the same harness: a topic-captured + // freeze frame (the fault_manager's own) carries no source, and none is + // invented for it. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "TEMP_FAULT"; + + ros2_medkit_msgs::msg::EnvironmentData env_data; + ros2_medkit_msgs::msg::Snapshot freeze_frame; + freeze_frame.type = "freeze_frame"; + freeze_frame.name = "temperature"; + freeze_frame.data = R"({"temperature": 85.5})"; + freeze_frame.topic = "/motor/temperature"; + freeze_frame.message_type = "sensor_msgs/msg/Temperature"; + env_data.snapshots.push_back(freeze_frame); + + auto response = + to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_json(env_data), "/apps/motor")); + + auto & snap = response["environment_data"]["snapshots"][0]; + EXPECT_FALSE(snap["x-medkit"].contains("source")); + EXPECT_EQ(snap["x-medkit"]["topic"], "/motor/temperature"); +} + // Conversion layer must emit an explicit "snapshot_type" discriminator so // downstream consumers (handler, SSE, MCP) can dispatch on a single key // regardless of which optional payload fields are present. From e8a1b96d81eda6a28de1372148b73835b33d89f0 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sun, 6 Sep 2026 12:43:01 +0200 Subject: [PATCH 03/25] fix(gateway): stop listing the gateway's own nodes as apps The gateway runs four nodes in its own process, all named after itself: the gateway node, "_sub" for the subscription executor, "_fault_clients" for the fault-service transport, and "_lifecycle_state_reader" for the lifecycle reader. None of them starts with an underscore, so the ROS 2 hidden-node convention does not cover them. Runtime introspection returned all four as ordinary Apps. An operator who browsed /api/v1/apps saw four entries that answer nothing useful. count_peer_nodes knew the gateway's own FQN plus "_sub" and "_fault_clients", but not the lifecycle reader. The app filter knew only the underscore rule, so it dropped none of the four. Both now use one predicate, is_own_gateway_node, so a fifth helper is declared in one place. The match is exact for each suffix. A real peer named "_monitor" or "2" stays visible, because hiding a real node is the worse error. A fault_manager in the same process is not ours and stays listed. Remote entities are left alone. A peer's helper nodes carry the same fully qualified names, and the peer's own filter handles them. --- .../ros2_medkit_gateway/gateway_node.hpp | 30 ++++++- src/ros2_medkit_gateway/src/gateway_node.cpp | 47 ++++++++-- .../test/test_handler_context.cpp | 90 +++++++++++++++++-- 3 files changed, 150 insertions(+), 17 deletions(-) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index a46a11bf5..d4585d99a 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -497,6 +497,27 @@ class GatewayNode : public rclcpp::Node { std::unique_ptr server_thread_; }; +/** + * @brief Is this node FQN the gateway's own, rather than a diagnosable peer? + * + * True for the gateway node itself and for the helper nodes it creates inside + * its own process: the subscription executor's `_sub`, the fault-service + * transport's `_fault_clients`, and the lifecycle reader's + * `_lifecycle_state_reader`. None of these begins with '_', so the ROS 2 + * hidden-node convention does not cover them and the gateway would otherwise + * count them as peers and list them as diagnosable Apps - reporting on itself. + * + * A fault_manager node sharing the process is NOT ours: it is a separate, + * diagnosable component and stays visible. + * + * Exact matches only. A prefix test would also claim a genuine peer named + * `_monitor` or `2`, and dropping a real node is the worse error. + * + * @param node_fqn Fully qualified node name to test ("/ns/node") + * @param self_fqn The gateway node's own FQN; an empty value matches nothing + */ +bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_fqn); + /** * @brief Filter ROS 2 internal nodes from an app list * @@ -505,11 +526,18 @@ class GatewayNode : public rclcpp::Node { * before checking for the underscore prefix, using the routing table for precise * prefix detection. * + * Also removes local apps bound to one of the gateway's own nodes + * (is_own_gateway_node), which the underscore rule cannot see. The test is on + * the bound node FQN, and only for apps with no routing-table entry: a peer's + * helper nodes are the peer's business and are left to the peer's own filter. + * * @param apps App vector to filter in place * @param peer_routing_table Maps entity_id -> peer_name for remote entities + * @param self_fqn The gateway node's own FQN; empty disables the self check * @return Number of apps removed */ size_t filter_internal_node_apps(std::vector & apps, - const std::unordered_map & peer_routing_table); + const std::unordered_map & peer_routing_table, + const std::string & self_fqn); } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index a8f32986e..0d4e8c5ff 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -15,6 +15,7 @@ #include "ros2_medkit_gateway/gateway_node.hpp" #include +#include #include #include #include @@ -1596,6 +1597,27 @@ GatewayNode::GatewayNode(const rclcpp::NodeOptions & options) : Node("ros2_medki }); } +bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_fqn) { + if (self_fqn.empty() || node_fqn.empty()) { + return false; + } + if (node_fqn == self_fqn) { + return true; + } + // The helper nodes the gateway creates inside its own process, each named + // after this node plus a fixed suffix. Where each one is set: + // "_sub" Ros2SubscriptionExecutor::Config + // (subscription_node_name_suffix) + // "_fault_clients" Ros2FaultServiceTransport + // "_lifecycle_state_reader" Ros2LifecycleStateReader + // Exact matches only: a prefix test would also claim a genuine peer named + // "_monitor" or "2", and hiding a real node is the worse error. + static constexpr std::array kHelperSuffixes{"_sub", "_fault_clients", "_lifecycle_state_reader"}; + return std::any_of(kHelperSuffixes.begin(), kHelperSuffixes.end(), [&](const char * suffix) { + return node_fqn == self_fqn + suffix; + }); +} + size_t GatewayNode::count_peer_nodes(const std::vector> & nodes_and_namespaces, const std::string & self_fqn) { size_t count = 0; @@ -1608,10 +1630,7 @@ size_t GatewayNode::count_peer_nodes(const std::vector_monitor" or "2"). - if (fqn == self_fqn || fqn == self_fqn + "_sub" || fqn == self_fqn + "_fault_clients") { + if (is_own_gateway_node(fqn, self_fqn)) { continue; } ++count; @@ -2457,14 +2476,15 @@ void GatewayNode::refresh_cache() { } } - // Filter ROS 2 internal nodes (underscore prefix convention). + // Filter ROS 2 internal nodes (underscore prefix convention) and this + // gateway's own helper nodes. // Controlled by discovery.runtime.filter_internal_nodes parameter (default: true). // Covers local heuristic apps (which bypass the merge pipeline orphan filter // in runtime_only mode) and any peer apps that slipped through fetch_entities. if (filter_internal_nodes_) { - auto removed = filter_internal_node_apps(apps, peer_routing_table); + auto removed = filter_internal_node_apps(apps, peer_routing_table, get_fully_qualified_name()); if (removed > 0) { - RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix)", removed); + RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix or own helper node)", removed); } } @@ -2561,9 +2581,10 @@ void GatewayNode::stop_rest_server() { } size_t filter_internal_node_apps(std::vector & apps, - const std::unordered_map & peer_routing_table) { + const std::unordered_map & peer_routing_table, + const std::string & self_fqn) { auto before = apps.size(); - auto end = std::remove_if(apps.begin(), apps.end(), [&peer_routing_table](const App & app) { + auto end = std::remove_if(apps.begin(), apps.end(), [&peer_routing_table, &self_fqn](const App & app) { std::string original_id = app.id; auto rt_it = peer_routing_table.find(app.id); if (rt_it != peer_routing_table.end()) { @@ -2573,6 +2594,14 @@ size_t filter_internal_node_apps(std::vector & apps, if (original_id.size() > prefix.size() && original_id.compare(0, prefix.size(), prefix) == 0) { original_id = original_id.substr(prefix.size()); } + } else if (is_own_gateway_node(app.effective_fqn(), self_fqn)) { + // A local app bound to one of this gateway's own nodes. Those names do + // not start with '_' ("_sub", "_fault_clients", ...), + // so only the FQN test catches them, and without it the gateway + // advertises its own plumbing as diagnosable apps. Remote entities are + // skipped deliberately: a peer's helper nodes carry the same FQNs and are + // the peer's own filter's business. + return true; } // ROS 2 internal nodes use _ prefix convention return !original_id.empty() && original_id[0] == '_'; diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 968f29469..573c356c0 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -874,7 +874,7 @@ TEST(FilterInternalNodeAppsTest, FiltersLocalInternalNodes) { apps.push_back(another_internal); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 2u); ASSERT_EQ(apps.size(), 1u); @@ -896,7 +896,7 @@ TEST(FilterInternalNodeAppsTest, PreservesAllNormalNodes) { apps.push_back(a3); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); EXPECT_EQ(apps.size(), 3u); @@ -921,7 +921,7 @@ TEST(FilterInternalNodeAppsTest, FiltersPeerPrefixedInternalNodes) { routing["peer_subsystem___ros2cli_daemon"] = "peer_subsystem"; routing["peer_subsystem__lidar_driver"] = "peer_subsystem"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 1u); ASSERT_EQ(apps.size(), 1u); @@ -939,7 +939,7 @@ TEST(FilterInternalNodeAppsTest, DoesNotStripPrefixWithoutRoutingEntry) { apps.push_back(ambiguous); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); ASSERT_EQ(apps.size(), 1u); @@ -950,7 +950,7 @@ TEST(FilterInternalNodeAppsTest, HandlesEmptyAppList) { std::vector apps; std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); EXPECT_TRUE(apps.empty()); @@ -981,7 +981,7 @@ TEST(FilterInternalNodeAppsTest, MixedLocalAndRemoteInternalNodes) { routing["sub_b__actuator"] = "sub_b"; routing["sub_b___parameter_bridge"] = "sub_b"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 2u); ASSERT_EQ(apps.size(), 2u); @@ -1006,12 +1006,88 @@ TEST(FilterInternalNodeAppsTest, PeerPrefixMatchMustBeExact) { std::unordered_map routing; routing["my_peer__sensor"] = "my_peer"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); ASSERT_EQ(apps.size(), 1u); } +namespace { + +App bound_app(const std::string & id, const std::string & fqn) { + App app; + app.id = id; + app.name = id; + app.bound_fqn = fqn; + return app; +} + +} // namespace + +TEST(FilterInternalNodeAppsTest, DropsTheGatewaysOwnHelperNodes) { + // The gateway creates "_sub", "_fault_clients" and + // "_lifecycle_state_reader" in its own process. None starts with '_', + // so runtime introspection returns them as ordinary apps and the gateway ends + // up listing its own plumbing as diagnosable. + const std::string self_fqn = "/ros2_medkit_gateway"; + std::vector apps{ + bound_app("ros2_medkit_gateway", self_fqn), + bound_app("ros2_medkit_gateway_sub", self_fqn + "_sub"), + bound_app("ros2_medkit_gateway_fault_clients", self_fqn + "_fault_clients"), + bound_app("ros2_medkit_gateway_lifecycle_state_reader", self_fqn + "_lifecycle_state_reader"), + // Positive controls on the same harness: a similarly suffixed FOREIGN + // node, a node whose name merely extends the gateway's, and the + // fault_manager, which is a separate diagnosable component even when it + // shares the process. + bound_app("other_gateway_sub", "/other_gateway_sub"), + bound_app("ros2_medkit_gateway_monitor", self_fqn + "_monitor"), + bound_app("fault_manager", "/fault_manager"), + }; + + std::unordered_map routing; + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 4u); + std::set remaining; + for (const auto & app : apps) { + remaining.insert(app.id); + } + EXPECT_EQ(remaining, (std::set{"other_gateway_sub", "ros2_medkit_gateway_monitor", "fault_manager"})); +} + +TEST(FilterInternalNodeAppsTest, LeavesPeerHelperNodesToThePeer) { + // A remote entity carrying the same FQN belongs to the peer that reported + // it, so this gateway must not reach across and filter it. + const std::string self_fqn = "/ros2_medkit_gateway"; + std::vector apps{bound_app("sub_b__ros2_medkit_gateway_sub", self_fqn + "_sub")}; + + std::unordered_map routing; + routing["sub_b__ros2_medkit_gateway_sub"] = "sub_b"; + + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 0u); + ASSERT_EQ(apps.size(), 1u); +} + +TEST(IsOwnGatewayNodeTest, MatchesSelfAndHelpersExactlyAndNothingElse) { + const std::string self_fqn = "/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_node(self_fqn, self_fqn)); + EXPECT_TRUE(is_own_gateway_node(self_fqn + "_sub", self_fqn)); + EXPECT_TRUE(is_own_gateway_node(self_fqn + "_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_node(self_fqn + "_lifecycle_state_reader", self_fqn)); + + // Prefix neighbours are genuine peers, not ours. + EXPECT_FALSE(is_own_gateway_node(self_fqn + "_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_node(self_fqn + "2", self_fqn)); + EXPECT_FALSE(is_own_gateway_node("/other" + self_fqn + "_sub", self_fqn)); + EXPECT_FALSE(is_own_gateway_node("/fault_manager", self_fqn)); + + // An unknown self FQN must claim nothing rather than everything. + EXPECT_FALSE(is_own_gateway_node(self_fqn, "")); + EXPECT_FALSE(is_own_gateway_node("", self_fqn)); +} + // ============================================================================= // Area fault/log aggregation handler tests (via REST API) // ============================================================================= From f1220ab8a22388832ae47797445886316800a7c7 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sun, 6 Sep 2026 16:35:01 +0200 Subject: [PATCH 04/25] test(gateway): assert the freeze-frame capture path from a real capture The route and DataProvider loss-of-comms tests now each assert the Frame::source constant that their own capture path must produce. They also assert the literal wire value. A comparison of symbol against symbol stays equal when the two constants swap values, and the string is what every consumer of x-medkit.source reads. The merge test that omits the key stays. It covers the helper contract for a frame that a caller built without naming a path. The capture paths always name one, and the test comment says so. The peer-node count test lists all three helper nodes that the gateway creates in its own process, the lifecycle reader included. A lone gateway with a lifecycle reader must count zero peers and give the empty-graph warning. The REST fault snapshot reference documents the source field of an entity frame. An entity frame carries no topic or message type, so source is the only provenance a consumer gets. --- docs/api/rest.rst | 20 ++++++++++------ .../test/test_entity_freeze_frame_capture.cpp | 23 ++++++++++++++++--- .../test/test_gateway_node.cpp | 4 ++++ 3 files changed, 37 insertions(+), 10 deletions(-) diff --git a/docs/api/rest.rst b/docs/api/rest.rst index 5f55fb862..489d25d1b 100644 --- a/docs/api/rest.rst +++ b/docs/api/rest.rst @@ -1517,13 +1517,19 @@ Query and manage faults. - ``freeze_frame``: Data captured at fault confirmation. Entity frames for faults that were already confirmed when the gateway started are captured at gateway start instead and carry ``"capture_origin": "startup"`` in - their ``x-medkit`` block. For a plugin-backed entity that reports its - link down, the values are the plugin's last known ones and may predate - the confirmation by the length of the outage; such entries carry - ``connected`` (the payload's link flag, ``false`` for the loss-of-comms - case) and ``source_timestamp`` (the payload's own timestamp, verbatim) - in ``x-medkit``, both only when the plugin's payload reports them. - ``captured_at`` always dates the capture, not the values. + their ``x-medkit`` block. An entity frame also carries ``source`` in + ``x-medkit``, naming the path that read the values + (``plugin_data_provider`` for the owning plugin's DataProvider, + ``plugin_x_plc_data_route`` for its ``x-plc-data`` route). These values + are not a ROS message, so ``topic`` and ``message_type`` are empty and + ``source`` is the only field saying where the numbers came from. For a + plugin-backed entity that reports its link down, the values are the + plugin's last known ones and may predate the confirmation by the length of + the outage; such entries carry ``connected`` (the payload's link flag, + ``false`` for the loss-of-comms case) and ``source_timestamp`` (the + payload's own timestamp, verbatim) in ``x-medkit``, both only when the + plugin's payload reports them. ``captured_at`` always dates the capture, + not the values. - ``rosbag``: Recording file available via bulk-data endpoint **Response codes:** diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index 152297ba0..8ef14dbc4 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -573,6 +573,15 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedEntityWithLastKnownValuesIsCapt ASSERT_TRUE(frames[0].connected.has_value()); EXPECT_FALSE(*frames[0].connected); EXPECT_EQ(frames[0].source_timestamp, 1234567890); + // Which path read the values. This capture had no DataProvider and went + // through the route fallback, so the frame must name that path; the + // DataProvider flavour of the same case asserts the other constant, which is + // what stops the two from being swapped at their call sites unnoticed. + EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceXPlcDataRoute); + // The wire value itself, not just the symbol: swapping what the two constants + // hold is an API break for every consumer of x-medkit.source, and comparing + // symbol against symbol would not see it. + EXPECT_EQ(frames[0].source, "plugin_x_plc_data_route"); } /// @verifies REQ_INTEROP_088 @@ -642,6 +651,11 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedDataProviderWithLastKnownValues ASSERT_TRUE(frames[0].connected.has_value()); EXPECT_FALSE(*frames[0].connected); EXPECT_TRUE(frames[0].source_timestamp.is_null()); // provider content has no timestamp field + // The provider path names itself, and the route path (same case, above) names + // the other constant: the pair is what makes a swap of the two call sites + // visible. The literal pins the wire value the API reference documents. + EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceDataProvider); + EXPECT_EQ(frames[0].source, "plugin_data_provider"); } /// @verifies REQ_INTEROP_088 @@ -805,9 +819,12 @@ TEST(MergeEntityFreezeFrames, CarriesCapturePathAsSource) { EXPECT_EQ(snap["message_type"], ""); } -TEST(MergeEntityFreezeFrames, OmitsSourceWhenTheCaptureNamedNoPath) { - // Absence control for the test above, on the same harness: a frame whose - // capture path is unknown must not have one invented for it. +TEST(MergeEntityFreezeFrames, OmitsSourceForAFrameThatNamesNoPath) { + // A merge-helper contract, not a control for the capture tests: both capture + // paths always name themselves (asserted from real captures in + // Disconnected{Entity,DataProvider}WithLastKnownValuesIsCaptured), so this + // frame is one only a caller can build. The helper must then leave the key + // out rather than invent a provenance the wire consumer would trust. json env_data = {{"snapshots", json::array()}}; EntityFreezeFrameCapture::Frame frame; frame.entity_id = "plc_app"; diff --git a/src/ros2_medkit_gateway/test/test_gateway_node.cpp b/src/ros2_medkit_gateway/test/test_gateway_node.cpp index 2f81dbc6c..557aef176 100644 --- a/src/ros2_medkit_gateway/test/test_gateway_node.cpp +++ b/src/ros2_medkit_gateway/test/test_gateway_node.cpp @@ -1058,10 +1058,14 @@ TEST(GatewayStartupSummary, CountPeerNodesExcludesOwnAndHidden) { } TEST(GatewayStartupSummary, CountPeerNodesZeroWhenOnlyOwnNodes) { + // Every helper the gateway creates inside its own process. A gateway alone on + // the graph must report zero peers, so each helper has to be recognized - + // including the lifecycle reader, which the list previously omitted. const std::vector> nodes = { {"ros2_medkit_gateway", "/"}, {"ros2_medkit_gateway_sub", "/"}, {"ros2_medkit_gateway_fault_clients", "/"}, + {"ros2_medkit_gateway_lifecycle_state_reader", "/"}, }; // Zero peers is the condition that triggers the empty-graph warning. EXPECT_EQ(ros2_medkit_gateway::GatewayNode::count_peer_nodes(nodes, "/ros2_medkit_gateway"), 0u); From e80fa08e344c29ef4919d40d68b87825f41788fe Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sun, 6 Sep 2026 18:51:20 +0200 Subject: [PATCH 05/25] fix(opcua): keep a scoped clear from cascading and rank the pending buffer Scoped clear. The per-entity route DELETE /{entity}/faults/{code} calls FaultProvider::clear_fault when a plugin owns the entity. On its own clear path the gateway sets skip_correlation_auto_clear. An operator scoped to one entity must not cascade-clear correlated symptoms that apps in other entities reported, and the ClearFault contract documents that guarantee. This plugin sent the flag as false, so the guarantee failed wherever a PLC was involved. The plugin now sets the flag. A ClearOrigin travels with every clear and says why: - the device reported the condition inactive. This is a real resolution, so the cascade stays. - the link came back. - an operator cleared the fault through the scoped route. The poller's own clear and a device alarm use the same callback. clear_origin_for_signal tells them apart by an exact fault-code match. Pending buffer. The buffer gave up any clear before any report. Only the link-state clear can be derived again, because the next reconnect sends it again. A device alarm's inactive edge is as one-shot as its raise. When the buffer evicted it, the flush replayed the raise alone, and the fault stood while the device said inactive. A full buffer now gives up the link-state clear first. Everything else ages out oldest-first. Sweep cancellation. The start-up sweep runs inside set_context(), during node construction, before the gateway's executor spins. Nothing can set the shutdown flag then, so a SIGTERM during a wide sweep waited for the sweep to finish. Both sweeps now ask discovery_cancelled(). It reads rclcpp::ok() next to the shutdown flag and applies the rule in discovery_cancelled_for. rclcpp's own signal handler turns rclcpp::ok() false, so the start-up sweep ends on the signal. The rescan still ends on shutdown(). The README says what ends each sweep. A rescan sweep that throws now stamps the cadence on its way out, so the next poll iteration does not start another sweep at once. The "scanning [subnets]" line is logged before the sweep, at INFO on the first pass and at DEBUG on a rescan. A long sweep with no output looks like a hung process. The comment on the start-up scan gives the right reason why its report comes out in full. Tests. The two clear-origin sites that need a live session run against the test_alarm_server fixture, with stub fault-manager services on the ROS graph, so the tests read the flag off the wire. One test requires a successful connect to clear PLC_COMMS_LOST without a cascade. A second fires and clears a condition through the fixture's CLI and requires that clear to keep the cascade, next to the connect-time clear of the same run. The fixture harness gained a stdin pipe for these commands. clear_origin_for_signal and discovery_cancelled_for are tested on both branches. A test shuts a private rclcpp context down and reads rclcpp::ok() back. The comment on the cancellation test states what it exercises. Prose semicolons become periods and commas in plugin comments, operator-visible log strings, the README, the docker scenario script, rest.rst, gateway_node.hpp and test comments. The doc comment of the Refused outcome says what it means. --- docs/api/rest.rst | 2 +- .../ros2_medkit_gateway/gateway_node.hpp | 4 +- .../test/test_entity_freeze_frame_capture.cpp | 2 +- .../ros2_medkit_opcua/README.md | 13 +- .../docker/scripts/run_discovery_race_test.sh | 8 +- .../ros2_medkit_opcua/opcua_plugin.hpp | 142 +++++-- .../ros2_medkit_opcua/src/opcua_plugin.cpp | 147 ++++--- .../test/test_network_discovery.cpp | 4 +- .../test/test_opcua_identity.cpp | 253 ++++++++++++ .../test/test_opcua_plugin.cpp | 375 ++++++++++++++++-- 10 files changed, 814 insertions(+), 136 deletions(-) diff --git a/docs/api/rest.rst b/docs/api/rest.rst index 489d25d1b..00aceb0aa 100644 --- a/docs/api/rest.rst +++ b/docs/api/rest.rst @@ -1525,7 +1525,7 @@ Query and manage faults. ``source`` is the only field saying where the numbers came from. For a plugin-backed entity that reports its link down, the values are the plugin's last known ones and may predate the confirmation by the length of - the outage; such entries carry ``connected`` (the payload's link flag, + the outage. Such entries carry ``connected`` (the payload's link flag, ``false`` for the loss-of-comms case) and ``source_timestamp`` (the payload's own timestamp, verbatim) in ``x-medkit``, both only when the plugin's payload reports them. ``captured_at`` always dates the capture, diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index d4585d99a..89a153d1b 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -514,7 +514,7 @@ class GatewayNode : public rclcpp::Node { * `_monitor` or `2`, and dropping a real node is the worse error. * * @param node_fqn Fully qualified node name to test ("/ns/node") - * @param self_fqn The gateway node's own FQN; an empty value matches nothing + * @param self_fqn The gateway node's own FQN. An empty value matches nothing */ bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_fqn); @@ -533,7 +533,7 @@ bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_ * * @param apps App vector to filter in place * @param peer_routing_table Maps entity_id -> peer_name for remote entities - * @param self_fqn The gateway node's own FQN; empty disables the self check + * @param self_fqn The gateway node's own FQN. Empty disables the self check * @return Number of apps removed */ size_t filter_internal_node_apps(std::vector & apps, diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index 8ef14dbc4..b97372530 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -574,7 +574,7 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedEntityWithLastKnownValuesIsCapt EXPECT_FALSE(*frames[0].connected); EXPECT_EQ(frames[0].source_timestamp, 1234567890); // Which path read the values. This capture had no DataProvider and went - // through the route fallback, so the frame must name that path; the + // through the route fallback, so the frame must name that path. The // DataProvider flavour of the same case asserts the other constant, which is // what stops the two from being swapped at their call sites unnoticed. EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceXPlcDataRoute); diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index cb251c09c..8cb6920bc 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -712,8 +712,8 @@ plugins.opcua.discovery: connect_timeout_ms: 600 # per-port TCP connect timeout scan_concurrency: 100 # bounded, polite concurrent connect count identify_timeout_ms: 6000 # per GetEndpoints identify - # re-scan cadence while disconnected. Omit the key for the built-in 30 s; - # set it to 0 to keep discovery on but never re-scan (start-up scan only). + # re-scan cadence while disconnected. Omit the key for the built-in 30 s, + # or set it to 0 to keep discovery on but never re-scan (start-up scan only). interval_s: 30 anonymous_none_only: true # only auto-connect None/Anonymous servers ``` @@ -721,7 +721,7 @@ plugins.opcua.discovery: Environment overrides (Docker / appliance): `OPCUA_DISCOVERY_ENABLED`, `OPCUA_DISCOVERY_SUBNETS` (comma-separated CIDRs), `OPCUA_DISCOVERY_INTERVAL_S`. Leaving `interval_s` (and `OPCUA_DISCOVERY_INTERVAL_S`) unset means "no cadence -stated" and takes the 30 s default; an explicit `0` is honoured as written and +stated" and takes the 30 s default. An explicit `0` is honoured as written and turns the recurring sweep off. A negative value is refused with a warning and leaves the cadence unset. @@ -764,8 +764,11 @@ Safety / OT posture: `interval_s` (default 30 s) for as long as it stays disconnected. Set `interval_s: 0` (or `OPCUA_DISCOVERY_INTERVAL_S=0`) to keep discovery on with the start-up scan only, or `enabled: false` to switch it off entirely. -- A sweep is cancelled when the plugin shuts down, so a stop does not have to - wait out a subnet the size of a /16. +- A sweep is cancellable, so a stop does not have to wait out a subnet the size + of a /16. The start-up sweep runs while the gateway node is still being + constructed, so what ends it is `SIGINT` / `SIGTERM`, which the plugin sees + through `rclcpp::ok()`. A re-scan sweep runs on the poll thread and is ended + by either that or the plugin's own `shutdown()`. - An explicitly configured `endpoint_url` (or `OPCUA_ENDPOINT_URL`) always wins; discovery then does nothing, so it never opens a second session on a PLC the plugin already polls. diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh index dc52d3823..5b807472b 100755 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh @@ -37,8 +37,8 @@ CONFIG_DIR=/tmp/discovery_race_config # The fallback the plugin keeps when a scan selects nothing (OpcuaClientConfig). FALLBACK_ENDPOINT="opc.tcp://localhost:4840" # Config-less naming: with no node map the component id is derived from the -# device. Before any server exists that can only be the fallback endpoint's host; -# after adoption it is the test server's DI nameplate (Manufacturer "SelfPatch +# device. Before any server exists that can only be the fallback endpoint's +# host. After adoption it is the test server's DI nameplate (Manufacturer "SelfPatch # Devices" + Model "SPX-1000"), slugified. FALLBACK_COMPONENT_ID="opcua-localhost" DEVICE_COMPONENT_ID="selfpatch_devices_spx_1000" @@ -62,8 +62,8 @@ fail() { exit 1 } -# x-plc-status of a named component (the node-map pass pins the id; the -# config-less pass has to look it up first). +# x-plc-status of a named component. The node-map pass pins the id, the +# config-less pass has to look it up first. status_json_for() { curl -sf "http://localhost:${GATEWAY_PORT}/api/v1/components/$1/x-plc-status" || echo '{}' } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index 4ad2da902..0e831a2d9 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -163,14 +163,19 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // Where one discovery pass reports to, plus the memory that keeps a repeated // identical pass quiet. A rescan runs every ``interval_s`` for the life of a - // disconnected process, so re-emitting the same scan line, per-server lines, - // summary and "no auto-connectable server" WARN each time buries every other - // message in the log. ``previous_outcome`` is owned by the caller (the plugin - // keeps one across rescans): when it is non-null and the pass reaches the same - // outcome as the pass before it, the whole report goes to ``debug`` instead. - // The first pass, and every pass whose outcome changed, is always reported at - // info/warn. A null ``previous_outcome`` (the startup scan, and tests that do - // not care) reports every pass. + // disconnected process, so re-emitting the same per-server lines, summary and + // "no auto-connectable server" WARN each time buries every other message in + // the log. ``previous_outcome`` is owned by the caller (the plugin keeps one + // across rescans): when it is non-null and the pass reaches the same outcome + // as the pass before it, the whole report goes to ``debug`` instead. The first + // pass, and every pass whose outcome changed, is always reported at info/warn. + // A null ``previous_outcome`` (tests that do not care) reports every pass. + // + // The "scanning [subnets]" announcement is NOT part of that report. It is sent + // before the sweep runs, because a wide subnet takes minutes and an operator + // watching start-up has to see the gateway working rather than hung. It says + // what the pass is about to do rather than what it found, so it carries the + // same first-pass / rescan levelling on its own. struct DiscoveryReporter { std::function info; std::function warn; @@ -258,44 +263,110 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, const OpcuaClient::DeviceInfo & info, const std::string & endpoint_url); - // Build the ClearFault request for one fault code. ``link_state`` marks a - // clear that only reports the OPC-UA link came back (the connect-time - // ``PLC_COMMS_LOST`` clear). Such a clear must not cascade: a correlation rule - // may name PLC_COMMS_LOST as the root cause of every symptom the outage - // produced, and the link returning is not an operator resolving those. An - // operator-driven clear (the SOVD DELETE route) leaves the flag off and keeps - // the cascade. Static so the wire field is assertable without a fault manager. + // Why a ClearFault is being sent. Two properties follow from it and nothing + // else does, so the origin travels instead of a pair of loose booleans: + // - whether the correlation cascade must be skipped + // (``clear_skips_correlation``), which goes on the wire, and + // - whether the clear is re-derivable (``clear_is_link_state``), which is + // what the pending buffer may give up first under pressure. + enum class ClearOrigin { + /// The device reported the condition inactive (an ``event_alarms`` / + /// ``auto_alarms`` condition, or a threshold rule going false). A one-shot + /// edge nothing will re-send, and a real resolution, so the cascade stands. + DeviceAlarm, + /// The OPC-UA session came back, so ``PLC_COMMS_LOST`` no longer holds. + /// Re-derived on the next reconnect if it is lost, and not an operator + /// resolving a root cause, so it must not cascade. + LinkState, + /// The SOVD per-entity ``DELETE /{entity}/faults/{code}`` route reached + /// FaultProvider::clear_fault. An operator scoped to one entity must not + /// cascade-clear symptoms reported by apps in other entities, which is the + /// same rule the gateway applies on its own (non-plugin) branch of that + /// route. One-shot: nothing re-derives an operator's decision. + ScopedOperator + }; + + // Whether this clear must leave the correlation engine's auto_clear_with_root + // cascade alone. True for everything except a device-reported clear. + static bool clear_skips_correlation(ClearOrigin origin) { + return origin != ClearOrigin::DeviceAlarm; + } + + // Whether this clear will be re-derived if it is dropped. Only the link-state + // clear will: the next reconnect sends it again. + static bool clear_is_link_state(ClearOrigin origin) { + return origin == ClearOrigin::LinkState; + } + + // Whether a discovery sweep must stop now, given the two independent stop + // signals. Static and pure so both inputs are testable: the member + // ``discovery_cancelled()`` only reads them off the process and hands them + // here, so this is the whole rule. + // + // - ``shutdown_requested`` is set by shutdown(), which the gateway calls + // after its executor returns. That ends a RESCAN sweep, which runs on the + // poll thread long after start-up. + // - ``rclcpp_ok`` is false once rclcpp's own SIGINT / SIGTERM handler has + // run. The START-UP sweep runs inside set_context(), during node + // construction and before the executor spins, so shutdown() cannot be + // reached while it is in progress and the signal is the only thing that + // can end it. + static bool discovery_cancelled_for(bool shutdown_requested, bool rclcpp_ok) { + return shutdown_requested || !rclcpp_ok; + } + + // Which kind of clear a fault-detection signal going inactive is. The poller + // emits the component-scoped ``PLC_COMMS_LOST`` clear through the same + // callback as every device alarm, and only that one is a link-state event. + static ClearOrigin clear_origin_for_signal(const std::string & fault_code) { + return fault_code == kCommsLostFaultCode ? ClearOrigin::LinkState : ClearOrigin::DeviceAlarm; + } + + // Build the ClearFault request for one fault code. + // ``skip_correlation_auto_clear`` goes on the wire verbatim (see ClearOrigin + // for who sets it and why). Static so the wire field is assertable without a + // fault manager. static ros2_medkit_msgs::srv::ClearFault::Request make_clear_fault_request(const std::string & fault_code, - bool link_state); + bool skip_correlation_auto_clear); // One entry in the bounded buffer of fault dispatches held while the // fault_manager service is unmatched. struct PendingFaultDispatch { enum class Kind { Report, Clear }; Kind kind{Kind::Report}; - std::string fault_code; ///< dedup key for a Clear; diagnostic for a Report + std::string fault_code; ///< dedup key for a Clear, diagnostic for a Report + /// Clear only: this dispatch is re-derivable (ClearOrigin::LinkState), so + /// the buffer may drop it before anything that is not. + bool link_state{false}; std::function dispatch; }; // What ``enqueue_pending_dispatch`` did, so the caller can log it. enum class PendingEnqueueOutcome { - Buffered, ///< appended, nothing lost - ReplacedClear, ///< superseded the pending clear for the same fault code - EvictedClear, ///< buffer was full: dropped a pending clear to make room - EvictedReport, ///< buffer was full of reports and a report arrived - Refused ///< buffer was full of reports and a clear arrived + Buffered, ///< appended, nothing lost + ReplacedClear, ///< superseded the pending clear for the same fault code + EvictedLinkStateClear, ///< buffer was full: dropped a re-derivable clear to make room + EvictedOldest, ///< buffer was full with nothing re-derivable in it: dropped the oldest entry + Refused ///< buffer was full with nothing re-derivable in it and the incoming + ///< dispatch was itself a re-derivable clear, so it was dropped instead }; // Enqueue policy for the bounded pending-dispatch buffer. // - // Reports outrank clears. A report is a one-shot edge from the PLC that - // nothing will re-send, while a clear is re-derivable: the link state is - // re-observed on the next reconnect. So at most ONE clear per fault code is - // ever pending (a newer one moves to the back, keeping report-then-clear - // order), a full buffer gives up its oldest pending clear first, and a clear - // arriving at a buffer full of reports is refused rather than evicting one. - // Without this a flapping link enqueued one connect-time clear per reconnect - // attempt and pushed real alarm reports out of the buffer. + // Only a link-state clear is re-derivable: the next reconnect sends it again. + // Everything else in the buffer is a one-shot edge nothing will re-send - a + // report, a device alarm going inactive, an operator's scoped clear - so those + // rank together and age out oldest-first, exactly as the buffer behaved before + // any of this. A link-state clear is what a full buffer gives up first, and an + // incoming one is refused rather than pushing a one-shot dispatch out. At most + // ONE clear per fault code is pending at a time (a newer one moves to the + // back, so an interleaved report-then-clear still flushes in that order). + // + // Without the link-state ranking a flapping link enqueued one connect-time + // clear per reconnect attempt and pushed real alarm reports out of the buffer. + // Without the "only link-state" part, a device alarm's inactive edge was + // evicted ahead of an older report and the flush replayed the raise with no + // clear behind it, leaving the fault standing while the device said inactive. static PendingEnqueueOutcome enqueue_pending_dispatch(std::vector & buffer, size_t max_size, PendingFaultDispatch entry); @@ -324,9 +395,9 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // Report/clear fault via ROS 2 service (private helpers, not the FaultProvider overrides) void send_report_fault(const std::string & entity_id, const std::string & fault_code, const std::string & severity_str, const std::string & message); - // ``link_state`` marks a clear that reports the OPC-UA link came back rather - // than an operator resolving a root cause; see make_clear_fault_request. - void send_clear_fault(const std::string & fault_code, bool link_state = false); + // ``origin`` says why the clear is being sent, which decides both the wire + // flag and how the pending buffer ranks it. See ClearOrigin. + void send_clear_fault(const std::string & fault_code, ClearOrigin origin = ClearOrigin::DeviceAlarm); // Clear PLC_COMMS_LOST after the initial connect in set_context() succeeded. // Unconditional on purpose: the fault manager keys faults by fault_code and @@ -398,6 +469,11 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // (null to report every pass in full). DiscoveryReporter discovery_reporter(std::string * previous_outcome) const; + // Abort predicate handed to a discovery sweep: reads the two stop signals off + // the process and applies ``discovery_cancelled_for``, which holds the rule + // and the reasoning behind it. + bool discovery_cancelled() const; + // Poll-thread hook bound into PollerConfig::rediscover_endpoint whenever // discovery runs without a configured endpoint. Called from the poller's // reconnect arm, so only while no session is up, and rate-limited to one scan diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 5010f2046..75ab2ef12 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -1091,10 +1091,10 @@ void OpcuaPlugin::on_alarm_change(const std::string & entity_id, } else { log_info("Alarm cleared: " + signal.fault_code + " on " + entity_id); // The poller's own comms-lost clear on a successful reconnect is the same - // link-state clear as the connect-time one, so it does not cascade either. - // Every other code is a real alarm going inactive on the device and keeps - // the default correlation behaviour. - send_clear_fault(signal.fault_code, /*link_state=*/signal.fault_code == kCommsLostFaultCode); + // link-state event as the connect-time one. Every other code here is the + // device reporting its condition inactive, which is a real resolution and a + // one-shot edge, so it keeps the cascade and the buffer treats it as such. + send_clear_fault(signal.fault_code, clear_origin_for_signal(signal.fault_code)); } } @@ -1273,7 +1273,11 @@ void OpcuaPlugin::on_event_alarm(const AlarmEventDelivery & delivery) { break; case AlarmAction::ClearFault: log_info("AlarmCondition CLEARED: " + delivery.fault_code); - send_clear_fault(delivery.fault_code); + // The device itself reported the condition cleared (Part 9 lifecycle), so + // this IS a resolution at the source and the correlation engine may act on + // it. DeviceAlarm is also the default, spelled out here because this is + // the one call site where the cascade is deliberately kept. + send_clear_fault(delivery.fault_code, ClearOrigin::DeviceAlarm); break; case AlarmAction::NoOp: break; @@ -1308,34 +1312,29 @@ void OpcuaPlugin::send_report_fault(const std::string & entity_id, const std::st request->severity = ros2_medkit_msgs::msg::Fault::SEVERITY_INFO; } - send_or_buffer({PendingFaultDispatch::Kind::Report, fault_code, [this, request]() { + send_or_buffer({PendingFaultDispatch::Kind::Report, fault_code, /*link_state=*/false, [this, request]() { fault_clients_->report->async_send_request(request); }}); } ros2_medkit_msgs::srv::ClearFault::Request OpcuaPlugin::make_clear_fault_request(const std::string & fault_code, - bool link_state) { + bool skip_correlation_auto_clear) { ros2_medkit_msgs::srv::ClearFault::Request request; request.fault_code = fault_code; - // A link-state clear reports that the OPC-UA session came back. It is not an - // operator resolving a root cause, so it must not trip the correlation - // engine's auto_clear_with_root cascade: a rule naming PLC_COMMS_LOST as the - // root cause would otherwise clear every symptom fault the outage produced, - // none of which this plugin has any evidence about. - request.skip_correlation_auto_clear = link_state; + request.skip_correlation_auto_clear = skip_correlation_auto_clear; return request; } -void OpcuaPlugin::send_clear_fault(const std::string & fault_code, bool link_state) { +void OpcuaPlugin::send_clear_fault(const std::string & fault_code, ClearOrigin origin) { if (!fault_clients_->clear) { log_warn("ClearFault service client not available"); return; } - auto request = - std::make_shared(make_clear_fault_request(fault_code, link_state)); + auto request = std::make_shared( + make_clear_fault_request(fault_code, clear_skips_correlation(origin))); - send_or_buffer({PendingFaultDispatch::Kind::Clear, fault_code, [this, request]() { + send_or_buffer({PendingFaultDispatch::Kind::Clear, fault_code, clear_is_link_state(origin), [this, request]() { fault_clients_->clear->async_send_request(request); }}); } @@ -1347,8 +1346,14 @@ void OpcuaPlugin::clear_comms_lost_on_connect() { // ClearFault is idempotent from this side: send_clear_fault is // fire-and-forget, so a "Fault not found" answer for a code that was never // raised costs nothing here and is the normal case on a healthy start. - log_info(std::string("OPC-UA connection established; clearing any standing ") + kCommsLostFaultCode); - send_clear_fault(kCommsLostFaultCode, /*link_state=*/true); + // + // LinkState: this says the session came back, not that an operator resolved + // anything, so a correlation rule naming PLC_COMMS_LOST as a root cause must + // not cascade-clear the symptoms the outage produced. It is also the one clear + // the next reconnect re-derives, so the pending buffer may drop it before + // anything one-shot. + log_info(std::string("OPC-UA connection established, clearing any standing ") + kCommsLostFaultCode); + send_clear_fault(kCommsLostFaultCode, ClearOrigin::LinkState); } OpcuaPlugin::PendingEnqueueOutcome OpcuaPlugin::enqueue_pending_dispatch(std::vector & buffer, @@ -1371,21 +1376,22 @@ OpcuaPlugin::PendingEnqueueOutcome OpcuaPlugin::enqueue_pending_dispatch(std::ve PendingEnqueueOutcome outcome = replaced ? PendingEnqueueOutcome::ReplacedClear : PendingEnqueueOutcome::Buffered; if (buffer.size() >= max_size) { - // A report is a one-shot edge from the PLC that nothing will re-send; a - // clear is re-derivable from the next reconnect. So a full buffer gives up a - // pending clear first, and refuses an incoming clear rather than evicting a - // report for it. - const auto oldest_clear = std::find_if(buffer.begin(), buffer.end(), [](const PendingFaultDispatch & pending) { - return pending.kind == PendingFaultDispatch::Kind::Clear; + // Only a link-state clear is re-derivable: the next reconnect sends it + // again. A full buffer gives that up first, and refuses an incoming one + // rather than pushing out a dispatch nothing will re-send. Everything else - + // reports, a device alarm's inactive edge, an operator's scoped clear - is + // one-shot and ages out oldest-first. + const auto oldest_link_state = std::find_if(buffer.begin(), buffer.end(), [](const PendingFaultDispatch & pending) { + return pending.kind == PendingFaultDispatch::Kind::Clear && pending.link_state; }); - if (oldest_clear != buffer.end()) { - buffer.erase(oldest_clear); - outcome = PendingEnqueueOutcome::EvictedClear; - } else if (is_clear) { + if (oldest_link_state != buffer.end()) { + buffer.erase(oldest_link_state); + outcome = PendingEnqueueOutcome::EvictedLinkStateClear; + } else if (is_clear && entry.link_state) { return PendingEnqueueOutcome::Refused; } else { buffer.erase(buffer.begin()); - outcome = PendingEnqueueOutcome::EvictedReport; + outcome = PendingEnqueueOutcome::EvictedOldest; } } @@ -1402,15 +1408,15 @@ void OpcuaPlugin::send_or_buffer(PendingFaultDispatch entry) { std::lock_guard lock(pending_reports_mutex_); outcome = enqueue_pending_dispatch(pending_reports_, kMaxPendingDispatches, std::move(entry)); } - if (outcome == PendingEnqueueOutcome::EvictedReport) { + if (outcome == PendingEnqueueOutcome::EvictedOldest) { log_warn("pending fault dispatch buffer full (" + std::to_string(kMaxPendingDispatches) + - "), dropping the oldest report"); - } else if (outcome == PendingEnqueueOutcome::EvictedClear) { + "), dropping the oldest dispatch"); + } else if (outcome == PendingEnqueueOutcome::EvictedLinkStateClear) { log_warn("pending fault dispatch buffer full (" + std::to_string(kMaxPendingDispatches) + - "), dropping the oldest pending clear"); + "), dropping a link-state clear the next reconnect re-derives"); } else if (outcome == PendingEnqueueOutcome::Refused) { - log_warn("pending fault dispatch buffer full of reports (" + std::to_string(kMaxPendingDispatches) + - "), dropping this clear instead of a report"); + log_warn("pending fault dispatch buffer full of one-shot dispatches (" + std::to_string(kMaxPendingDispatches) + + "), dropping this link-state clear instead"); } // Drains immediately (in order) if the sink is already matched. flush_pending_reports(); @@ -1506,7 +1512,7 @@ std::optional OpcuaPlugin::rederived_component_identity(const } void OpcuaPlugin::maybe_rederive_component_identity() { - // An explicit node map owns the component name; only the config-less path + // An explicit node map owns the component name. Only the config-less path // derives it from the device. if (!node_map_path_.empty() || !client_ || !client_->is_connected()) { return; @@ -1692,12 +1698,19 @@ std::optional OpcuaPlugin::rescan_step(int interval_s, if (now() - *last_scan_end < std::chrono::seconds(interval_s)) { return std::nullopt; } - const auto result = sweep(); // Stamp the END of the sweep: a legal /16 runs for minutes, and stamping its // start would make the next one due the moment this one returned - the poll // thread would sweep back to back and only attempt a reconnect once a sweep. - *last_scan_end = now(); - return result; + // A sweep that threw still consumed that time, so the stamp is owed either + // way, or the next poll iteration would start another one immediately. + try { + const auto result = sweep(); + *last_scan_end = now(); + return result; + } catch (...) { + *last_scan_end = now(); + throw; + } } std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryConfig & config, bool endpoint_configured, @@ -1751,7 +1764,7 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo const auto subnets = discovery.resolve_subnets(); if (subnets.empty()) { - warn_line("OPC-UA discovery: no subnet configured and could not derive a local /24; nothing to scan."); + warn_line("OPC-UA discovery: no subnet configured and could not derive a local /24, nothing to scan."); emit(); return std::nullopt; } @@ -1759,8 +1772,18 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo for (const auto & s : subnets) { subnet_list += (subnet_list.empty() ? "" : ", ") + s; } - info_line("OPC-UA discovery: read-only active scan of [" + subnet_list + "] on " + - std::to_string(config.ports.size()) + " port(s)..."); + // The announcement goes out NOW, not through the buffered report: a sweep of a + // wide subnet runs for minutes, and an operator watching start-up has to see + // that the gateway is scanning rather than hung. It says what the pass is + // about to do, not what it found, so it stays out of the outcome digest and is + // levelled on its own - the first pass announces at INFO, a rescan at DEBUG so + // the recurring sweep does not repeat it every interval. + const bool first_pass = reporter.previous_outcome == nullptr || reporter.previous_outcome->empty(); + const auto & announce_sink = first_pass ? reporter.info : reporter.debug; + if (announce_sink) { + announce_sink("OPC-UA discovery: read-only active scan of [" + subnet_list + "] on " + + std::to_string(config.ports.size()) + " port(s)..."); + } const std::vector found = discovery.run(cancelled); @@ -1797,7 +1820,7 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo const DiscoveredEndpoint * chosen = NetworkDiscovery::select_auto_endpoint(found, config.anonymous_none_only); if (chosen == nullptr) { warn_line( - "OPC-UA discovery: no auto-connectable None/Anonymous data server found; leaving the endpoint unchanged. " + "OPC-UA discovery: no auto-connectable None/Anonymous data server found, leaving the endpoint unchanged. " "Secured-only servers require operator credentials."); emit(); return std::nullopt; @@ -1814,16 +1837,19 @@ void OpcuaPlugin::run_startup_discovery() { } if (endpoint_configured_) { log_info("OPC-UA discovery enabled but endpoint_url is explicitly configured (" + client_config_.endpoint_url + - "); skipping auto-discovery to avoid a second session."); + "). Skipping auto-discovery to avoid a second session."); return; } - // The startup scan is always reported in full (no previous outcome to compare - // against) and always cancellable, so a shutdown during set_context does not - // wait out a whole sweep. + // The startup scan is the first pass, so its outcome digest is still empty and + // the report comes out in full. It is cancellable too, but not by shutdown(): + // this runs inside set_context(), i.e. during node construction and before the + // executor spins, so nothing can call shutdown() until this returns. What ends + // it is the SIGINT / SIGTERM that rclcpp's own handler turns into + // !rclcpp::ok() - see discovery_cancelled(). const auto chosen = discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, discovery_reporter(&last_discovery_outcome_), [this]() { - return shutdown_requested_.load(); + return discovery_cancelled(); }); // Stamp when the sweep FINISHED: the rescan cadence is measured from the end // of the previous sweep, so a long sweep is not immediately followed by @@ -1839,11 +1865,11 @@ void OpcuaPlugin::run_startup_discovery() { // which of the two they configured. const int startup_interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); if (startup_interval_s > 0) { - log_info("OPC-UA discovery: startup scan selected no endpoint; the reconnect loop rescans every " + + log_info("OPC-UA discovery: startup scan selected no endpoint. The reconnect loop rescans every " + std::to_string(startup_interval_s) + "s while down."); } else { log_warn( - "OPC-UA discovery: startup scan selected no endpoint and re-scanning is off (interval_s: 0); the endpoint " + "OPC-UA discovery: startup scan selected no endpoint and re-scanning is off (interval_s: 0). The endpoint " "stays at " + client_config_.endpoint_url + " until the plugin is restarted."); } @@ -1854,6 +1880,13 @@ void OpcuaPlugin::run_startup_discovery() { log_info("OPC-UA discovery: auto-selected endpoint " + *chosen + " - handing to the connect + introspect path."); } +bool OpcuaPlugin::discovery_cancelled() const { + // rclcpp::ok() only reads the default context's atomic shutdown flag, so it is + // safe to call from the set_context thread and the poll thread alike. The rule + // itself, and why both signals are needed, lives in discovery_cancelled_for. + return discovery_cancelled_for(shutdown_requested_.load(), rclcpp::ok()); +} + OpcuaPlugin::DiscoveryReporter OpcuaPlugin::discovery_reporter(std::string * previous_outcome) const { DiscoveryReporter reporter; reporter.info = [this](const std::string & m) { @@ -1873,7 +1906,7 @@ std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { // A sweep is a bounded but multi-second blocking call on the poll thread, and // stop() has to wait for whatever it is in the middle of. Do not start one the // shutdown is going to throw away. - if (shutdown_requested_.load()) { + if (discovery_cancelled()) { return std::nullopt; } const int interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); @@ -1887,7 +1920,7 @@ std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { [this]() { return discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, discovery_reporter(&last_discovery_outcome_), [this]() { - return shutdown_requested_.load(); + return discovery_cancelled(); }); }); // The live client config, not client_config_: this runs on the poll thread @@ -2440,7 +2473,13 @@ tl::expected OpcuaPlugin::clear_f return tl::make_unexpected(FaultProviderErrorInfo{FaultProviderError::Internal, "plugin not initialized", 503}); } - send_clear_fault(fault_code); + // This is the per-entity SOVD route DELETE /{entity}/faults/{code}. The + // gateway sets skip_correlation_auto_clear on its own branch of that route so + // an operator scoped to one entity cannot cascade-clear symptoms reported by + // apps in other entities, and a plugin-owned entity must not be the hole in + // that rule: the request goes through this provider instead, so the same flag + // has to be set here. + send_clear_fault(fault_code, ClearOrigin::ScopedOperator); return dto::FaultClearResult{ nlohmann::json{{"status", "cleared"}, {"fault_code", fault_code}, {"entity_id", entity_id}}}; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp index 96c32e339..7ef7c0e4f 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp @@ -408,8 +408,8 @@ TEST(NetworkDiscoveryRun, IdentifyFailureRecordedAsLead) { // run(cancelled): a shutdown must not wait out a whole sweep // --------------------------------------------------------------------------- // TEST(NetworkDiscoveryRun, CancelStopsTheSweepInsteadOfProbingEveryHost) { - // A /24 is 254 probes and a legal /16 is 65k; the caller runs them on the poll - // thread a shutdown has to join. With scan_concurrency 1 the sweep is + // A /24 is 254 probes and a legal /16 is 65k, and the caller runs them on the + // poll thread a shutdown has to join. With scan_concurrency 1 the sweep is // sequential, so the probe count is exactly what the cancel predicate allowed. std::atomic probes{0}; std::atomic stop{false}; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 93b84ee1f..f96daf0e3 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -45,7 +45,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -53,6 +55,8 @@ #include #include +#include +#include #include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" @@ -193,17 +197,28 @@ class AlarmServer { if (pipe(pipefd) != 0) { return false; } + int stdin_pipe[2]; + if (pipe(stdin_pipe) != 0) { + close(pipefd[0]); + close(pipefd[1]); + return false; + } pid_ = fork(); if (pid_ < 0) { close(pipefd[0]); close(pipefd[1]); + close(stdin_pipe[0]); + close(stdin_pipe[1]); return false; } if (pid_ == 0) { dup2(pipefd[1], STDOUT_FILENO); dup2(pipefd[1], STDERR_FILENO); + dup2(stdin_pipe[0], STDIN_FILENO); close(pipefd[0]); close(pipefd[1]); + close(stdin_pipe[0]); + close(stdin_pipe[1]); std::string port_str = std::to_string(port); std::vector argv_vec{binary.c_str(), "--port", port_str.c_str()}; for (const auto & arg : extra_args) { @@ -214,11 +229,27 @@ class AlarmServer { _exit(127); } close(pipefd[1]); + close(stdin_pipe[0]); read_fd_ = pipefd[0]; + write_fd_ = stdin_pipe[1]; return wait_for_ready(15000); } + // One CLI command ("fire Overpressure 750", "clear Overpressure", ...). The + // fixture reads them line by line off stdin. + bool send(const std::string & command) { + if (write_fd_ < 0) { + return false; + } + const std::string line = command + "\n"; + return write(write_fd_, line.c_str(), line.size()) == static_cast(line.size()); + } + void stop() { + if (write_fd_ >= 0) { + close(write_fd_); + write_fd_ = -1; + } if (pid_ > 0) { kill(pid_, SIGTERM); int status = 0; @@ -258,6 +289,7 @@ class AlarmServer { pid_t pid_{-1}; int read_fd_{-1}; + int write_fd_{-1}; }; std::string fixture_binary() { @@ -631,4 +663,225 @@ TEST_F(OpcuaIdentityE2ETest, SuccessfulConnectClearsCommsLostNeverRaisedHere) { << "comms-lost must not be raised while the connection is up"; } +namespace { + +// RAII rclcpp init/shutdown, tearing down only what it started. +struct ScopedRclcpp { + const bool owned_; + ScopedRclcpp() : owned_(!rclcpp::ok()) { + if (owned_) { + rclcpp::init(0, nullptr); + } + } + ~ScopedRclcpp() { + if (owned_ && rclcpp::ok()) { + rclcpp::shutdown(); + } + } + ScopedRclcpp(const ScopedRclcpp &) = delete; + ScopedRclcpp & operator=(const ScopedRclcpp &) = delete; +}; + +// The plugin only builds its fault-service clients when the context hands it a +// real node, which is what makes the ClearFault request observable on the wire. +class RealNodePluginContext : public FakePluginContext { + public: + explicit RealNodePluginContext(rclcpp::Node * node) : node_(node) { + } + rclcpp::Node * node() const override { + return node_; + } + + private: + rclcpp::Node * node_; +}; + +} // namespace + +// The connect-time clear, read off the wire. clear_comms_lost_on_connect() is +// only reachable through a connect that SUCCEEDS, so it needs the live fixture, +// and the flag it sets is only observable with a real fault-manager service on +// the other end. A correlation rule may name PLC_COMMS_LOST as the root cause of +// every symptom an outage produced, and the link coming back is not an operator +// resolving those, so this clear must not cascade. +TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_connect_clear"); + auto fault_manager = std::make_shared("opcua_identity_connect_clear_faultmgr"); + + std::mutex received_mutex; + std::vector cleared_requests; + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", [](const std::shared_ptr, + std::shared_ptr res) { + res->accepted = true; + }); + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", + [&cleared_requests, &received_mutex](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(received_mutex); + cleared_requests.push_back(*req); + } + res->success = true; + }); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + std::thread spin_thread([&executor]() { + executor.spin(); + }); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + // The connect inside set_context() succeeds against the fixture, which is the + // only way to reach the connect-time clear. + plugin.set_context(ctx); + + // The clear may be buffered until the stub service is DDS-matched. The poll + // thread drains the buffer on its next cycle. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); + bool delivered = false; + while (!delivered && std::chrono::steady_clock::now() < deadline) { + { + std::lock_guard lock(received_mutex); + delivered = !cleared_requests.empty(); + } + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + + executor.cancel(); + if (spin_thread.joinable()) { + spin_thread.join(); + } + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + std::lock_guard lock(received_mutex); + ASSERT_FALSE(cleared_requests.empty()) << "a successful connect sent no ClearFault at all"; + EXPECT_EQ(cleared_requests.front().fault_code, std::string(kCommsLostFaultCode)); + EXPECT_TRUE(cleared_requests.front().skip_correlation_auto_clear) + << "the connect-time clear cascade-cleared the symptoms of the outage it ended"; +} + +// The other side of the same rule, also on the wire: when the DEVICE reports its +// condition inactive, that IS a resolution at the source, so the correlation +// engine may act on it and the flag stays off. Only a live AlarmCondition +// lifecycle reaches on_event_alarm's ClearFault arm, so this drives the +// fixture's own CLI to fire and then clear a condition. +TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_device_clear"); + auto fault_manager = std::make_shared("opcua_identity_device_clear_faultmgr"); + + std::mutex received_mutex; + std::vector reported; + std::vector cleared_requests; + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", + [&reported, &received_mutex](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(received_mutex); + reported.push_back(req->fault_code); + } + res->accepted = true; + }); + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", + [&cleared_requests, &received_mutex](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(received_mutex); + cleared_requests.push_back(*req); + } + res->success = true; + }); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + std::thread spin_thread([&executor]() { + executor.spin(); + }); + + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["poll_interval_ms"] = 100; + // Zero-config native A&C on the Server EventNotifier, with auto_clear so the + // condition going inactive clears the fault without an operator ack/confirm. + config["auto_alarms"] = nlohmann::json{{"enabled", true}, {"auto_clear", true}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + + const auto reported_count = [&received_mutex, &reported]() { + std::lock_guard lock(received_mutex); + return reported.size(); + }; + // The connect-time PLC_COMMS_LOST clear also lands here (this connect + // succeeded), so a clear is looked up by the code it names. + const auto clear_for = [&received_mutex, &cleared_requests](const std::string & code) -> std::optional { + std::lock_guard lock(received_mutex); + for (const auto & req : cleared_requests) { + if (req.fault_code == code) { + return req.skip_correlation_auto_clear; + } + } + return std::nullopt; + }; + + // Fire until the event subscription is up and a report lands. The retry is the + // subscription handshake, not flakiness in the assertion: an event fired + // before the subscribe simply is not delivered. + const auto fire_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (reported_count() == 0 && std::chrono::steady_clock::now() < fire_deadline) { + ASSERT_TRUE(server_.send("fire Overpressure 750")); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + } + ASSERT_GT(reported_count(), 0u) << "the fixture's AlarmCondition never reached the fault manager"; + + std::string alarm_code; + { + std::lock_guard lock(received_mutex); + alarm_code = reported.front(); + } + ASSERT_TRUE(server_.send("clear Overpressure")); + const auto clear_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (!clear_for(alarm_code).has_value() && std::chrono::steady_clock::now() < clear_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + executor.cancel(); + if (spin_thread.joinable()) { + spin_thread.join(); + } + plugin.shutdown(); + + const auto device_clear_skips = clear_for(alarm_code); + ASSERT_TRUE(device_clear_skips.has_value()) + << "the device reporting condition " << alarm_code << " inactive sent no ClearFault"; + EXPECT_FALSE(*device_clear_skips) << "a clear the device itself reported must keep the correlation cascade"; + + // The connect-time clear travelled the same wire in the same test, and it is + // the opposite case: not an operator resolving anything, so it does not + // cascade. Having both here is what makes the flag above a decision rather + // than a constant. + const auto link_state_clear_skips = clear_for(kCommsLostFaultCode); + ASSERT_TRUE(link_state_clear_skips.has_value()) << "the connect-time clear never arrived"; + EXPECT_TRUE(*link_state_clear_skips); +} + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp index 84d17f99e..b50935eef 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp @@ -984,6 +984,39 @@ TEST(RescanStep, SpacesSweepsFromTheEndOfThePreviousOne) { EXPECT_EQ(sweeps, 2); } +TEST(RescanStep, AThrowingSweepStillStampsTheCadence) { + // A sweep that throws still consumed its minutes. If the stamp were owed only + // on the normal path, the next poll iteration would find the cadence due and + // start another sweep immediately, so a server that makes the identify throw + // would turn the poll thread into a continuous scanner. + const auto t0 = std::chrono::steady_clock::time_point{}; + auto clock_now = t0 + std::chrono::seconds(30); + const auto now = [&clock_now]() { + return clock_now; + }; + int sweeps = 0; + const auto throwing_sweep = [&sweeps, &clock_now]() -> std::optional { + ++sweeps; + clock_now += std::chrono::seconds(120); + throw std::runtime_error("identify blew up mid-sweep"); + }; + + auto last_end = t0; + EXPECT_THROW(OpcuaPlugin::rescan_step(30, now, &last_end, throwing_sweep), std::runtime_error); + EXPECT_EQ(sweeps, 1); + EXPECT_EQ(last_end, clock_now) << "a sweep that threw still has to stamp the cadence"; + + // Inside the interval after that failed sweep: not due, so no second sweep. + clock_now += std::chrono::seconds(29); + EXPECT_NO_THROW(OpcuaPlugin::rescan_step(30, now, &last_end, throwing_sweep)); + EXPECT_EQ(sweeps, 1) << "a failed sweep let the next one start inside the interval"; + + // Positive control: one full interval later it is due again (and throws again). + clock_now += std::chrono::seconds(1); + EXPECT_THROW(OpcuaPlugin::rescan_step(30, now, &last_end, throwing_sweep), std::runtime_error); + EXPECT_EQ(sweeps, 2); +} + TEST(RescanStep, DoesNothingWithoutACadence) { const auto t0 = std::chrono::steady_clock::time_point{}; auto last_end = t0; @@ -1091,6 +1124,87 @@ TEST(DiscoverEndpoint, AnUnchangedRescanReportsAtDebugInsteadOfRepeatingItself) EXPECT_GT(info.size(), first_info) << "a changed outcome must be reported at INFO"; } +TEST(DiscoverEndpoint, APredicateThatFlipsMidSweepEndsThePass) { + // What a stop signal does to a sweep in progress. This predicate is the + // test's own, standing in for the one the plugin passes: it flips after a + // handful of probes, as either of the plugin's two stop signals would + // mid-sweep. The plugin's own predicate is pinned separately, by + // DiscoveryCancelledFor. + std::atomic probes{0}; + std::atomic stop{false}; + auto stopping_scan = [&probes, &stop](const std::string & ip, uint16_t port, int) { + if (probes.fetch_add(1) + 1 >= 5) { + stop.store(true); + } + return ip == "192.168.1.10" && port == 4840; // the PLC IS there to be found + }; + + OpcuaDiscoveryConfig cfg = rescan_cfg(); + cfg.scan_concurrency = 1; // sequential, so the probe count is the predicate's doing + const auto chosen = OpcuaPlugin::discover_endpoint(cfg, /*endpoint_configured=*/false, stopping_scan, + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), + silent_reporter(), [&stop]() { + return stop.load(); + }); + + EXPECT_FALSE(chosen.has_value()) << "a cancelled pass must not hand back a partial result"; + EXPECT_LE(probes.load(), 6) << "the sweep ran on after the stop signal"; + + // Positive control on the same fakes: without the predicate the very same + // sweep visits all 254 hosts and selects the PLC. + probes.store(0); + stop.store(false); + const auto uncancelled = OpcuaPlugin::discover_endpoint( + cfg, /*endpoint_configured=*/false, + [&probes](const std::string & ip, uint16_t port, int) { + probes.fetch_add(1); + return ip == "192.168.1.10" && port == 4840; + }, + fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), silent_reporter()); + ASSERT_TRUE(uncancelled.has_value()); + EXPECT_EQ(*uncancelled, "opc.tcp://192.168.1.10:4840"); + EXPECT_EQ(probes.load(), 254); +} + +TEST(DiscoverEndpoint, TheScanIsAnnouncedBeforeTheSweepRuns) { + // A /16 sweep runs for minutes. If the announcement waited for the report at + // the end of the pass, start-up would log nothing while it swept and an + // operator would read that as a hung gateway. + std::vector info; + std::vector debug; + std::string announced_before_first_probe; + std::string outcome; + OpcuaPlugin::DiscoveryReporter reporter; + reporter.info = [&info](const std::string & m) { + info.push_back(m); + }; + reporter.warn = kSilent; + reporter.debug = [&debug](const std::string & m) { + debug.push_back(m); + }; + reporter.previous_outcome = &outcome; + + auto scan_recording_the_log = [&info, &announced_before_first_probe](const std::string &, uint16_t, int) { + if (announced_before_first_probe.empty() && !info.empty()) { + announced_before_first_probe = info.front(); + } + return false; + }; + OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, scan_recording_the_log, fake_identify({}), + reporter); + EXPECT_NE(announced_before_first_probe.find("read-only active scan of"), std::string::npos) + << "the sweep started before the operator was told anything (first INFO line: '" + << (info.empty() ? std::string("") : info.front()) << "')"; + + // On a rescan the announcement drops to DEBUG: the sweep repeats every + // interval_s for the life of the outage and must not narrate every pass. + const size_t info_after_first = info.size(); + OpcuaPlugin::discover_endpoint(rescan_cfg(), /*endpoint_configured=*/false, fake_scan({}), fake_identify({}), + reporter); + EXPECT_EQ(info.size(), info_after_first) << "the rescan announced itself at INFO again"; + EXPECT_FALSE(debug.empty()); +} + TEST(DiscoverEndpoint, WithNoRepeatMemoryEveryPassIsReported) { // Positive control for the test above: the same two identical passes with no // previous_outcome (the startup scan's own reporter) report in full twice, so @@ -1163,38 +1277,104 @@ TEST(RederivedComponentIdentity, KeepsTheIdentityWhenNothingChanged) { } // --------------------------------------------------------------------------- -// ClearFault: a link-state clear does not cascade +// The stop signals a discovery sweep watches // --------------------------------------------------------------------------- -TEST(MakeClearFaultRequest, LinkStateClearSkipsTheCorrelationCascade) { - // The connect-time PLC_COMMS_LOST clear says the link came back. A - // correlation rule may name PLC_COMMS_LOST as the root cause of every symptom - // the outage produced, and clearing those is an operator's call, not a link - // event's. - const auto link_state = OpcuaPlugin::make_clear_fault_request(kCommsLostFaultCode, /*link_state=*/true); - EXPECT_EQ(link_state.fault_code, kCommsLostFaultCode); - EXPECT_TRUE(link_state.skip_correlation_auto_clear); +TEST(DiscoveryCancelledFor, EitherStopSignalEndsASweep) { + // The plugin's own predicate is this rule applied to two values it reads off + // the process, so this is the whole of it. + EXPECT_FALSE(OpcuaPlugin::discovery_cancelled_for(/*shutdown_requested=*/false, /*rclcpp_ok=*/true)) + << "a running process must not cancel its own sweep"; + // shutdown() ends a rescan sweep on the poll thread. + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for(/*shutdown_requested=*/true, /*rclcpp_ok=*/true)); + // SIGINT / SIGTERM ends the start-up sweep, which runs during node + // construction where shutdown() cannot be reached at all. + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for(/*shutdown_requested=*/false, /*rclcpp_ok=*/false)) + << "a signal during the start-up sweep left it running"; + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for(true, false)); +} + +TEST(DiscoveryCancelledFor, RclcppOkIsTheSignalTheStartUpSweepWatches) { + // The second input is not hypothetical: rclcpp's shutdown is what a SIGTERM + // turns into, and it is observable exactly this way. A private context keeps + // the process-wide default one (which other tests here initialise) untouched. + auto context = std::make_shared(); + context->init(0, nullptr); + ASSERT_TRUE(rclcpp::ok(context)); + EXPECT_FALSE(OpcuaPlugin::discovery_cancelled_for(/*shutdown_requested=*/false, rclcpp::ok(context))); + + context->shutdown("simulated SIGTERM"); + ASSERT_FALSE(rclcpp::ok(context)) << "rclcpp::ok did not follow the shutdown a signal performs"; + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for(/*shutdown_requested=*/false, rclcpp::ok(context))); +} - // Positive control on the same request builder: an operator-driven clear (the - // SOVD DELETE route) leaves the cascade alone, so the flag above is the - // link-state rule and not a hardcoded true. - const auto operator_clear = OpcuaPlugin::make_clear_fault_request("PLC_TANK_HIGH", /*link_state=*/false); - EXPECT_EQ(operator_clear.fault_code, "PLC_TANK_HIGH"); - EXPECT_FALSE(operator_clear.skip_correlation_auto_clear); +// --------------------------------------------------------------------------- +// ClearFault: only a clear the device itself reported may cascade +// --------------------------------------------------------------------------- + +TEST(ClearOriginForSignal, OnlyTheCommsLostCodeIsALinkStateClear) { + // The poller emits its component-scoped comms-lost clear through the same + // callback as every device alarm going inactive, so the fault code is the only + // thing that tells the two apart on that path. + EXPECT_EQ(OpcuaPlugin::clear_origin_for_signal(kCommsLostFaultCode), OpcuaPlugin::ClearOrigin::LinkState); + EXPECT_EQ(OpcuaPlugin::clear_origin_for_signal("PLC_TANK_HIGH"), OpcuaPlugin::ClearOrigin::DeviceAlarm); + EXPECT_EQ(OpcuaPlugin::clear_origin_for_signal(std::string(kCommsLostFaultCode) + "_UPSTREAM"), + OpcuaPlugin::ClearOrigin::DeviceAlarm) + << "the match must be the exact code, not a prefix"; +} + +TEST(ClearOrigin, OnlyADeviceReportedClearKeepsTheCorrelationCascade) { + using Origin = OpcuaPlugin::ClearOrigin; + // The link coming back is not an operator resolving a root cause, and neither + // is an operator scoped to ONE entity: a correlation rule naming + // PLC_COMMS_LOST as the root cause would otherwise clear symptom faults + // reported by apps in entities that operator cannot even see. The gateway + // applies exactly this rule on its own branch of the same DELETE route. + EXPECT_TRUE(OpcuaPlugin::clear_skips_correlation(Origin::LinkState)); + EXPECT_TRUE(OpcuaPlugin::clear_skips_correlation(Origin::ScopedOperator)); + // Positive control on the same predicate: the device reporting its own + // condition inactive IS a resolution at the source, so that clear cascades. + // Without this case the rule above would be indistinguishable from a + // hardcoded true. + EXPECT_FALSE(OpcuaPlugin::clear_skips_correlation(Origin::DeviceAlarm)); + + // The buffer's ranking is a different question from the wire flag: only the + // link-state clear is re-derivable, the operator's scoped clear is as + // one-shot as an alarm report. + EXPECT_TRUE(OpcuaPlugin::clear_is_link_state(Origin::LinkState)); + EXPECT_FALSE(OpcuaPlugin::clear_is_link_state(Origin::ScopedOperator)); + EXPECT_FALSE(OpcuaPlugin::clear_is_link_state(Origin::DeviceAlarm)); +} + +TEST(MakeClearFaultRequest, CarriesTheSkipFlagAndCodeVerbatim) { + const auto skipping = OpcuaPlugin::make_clear_fault_request(kCommsLostFaultCode, true); + EXPECT_EQ(skipping.fault_code, kCommsLostFaultCode); + EXPECT_TRUE(skipping.skip_correlation_auto_clear); + + const auto cascading = OpcuaPlugin::make_clear_fault_request("PLC_TANK_HIGH", false); + EXPECT_EQ(cascading.fault_code, "PLC_TANK_HIGH"); + EXPECT_FALSE(cascading.skip_correlation_auto_clear); } // --------------------------------------------------------------------------- -// Pending fault dispatch buffer: reports outrank clears +// Pending fault dispatch buffer: only what is re-derivable may be dropped first // --------------------------------------------------------------------------- namespace { OpcuaPlugin::PendingFaultDispatch report_entry(const std::string & code) { - return {OpcuaPlugin::PendingFaultDispatch::Kind::Report, code, []() {}}; + return {OpcuaPlugin::PendingFaultDispatch::Kind::Report, code, /*link_state=*/false, []() {}}; } -OpcuaPlugin::PendingFaultDispatch clear_entry(const std::string & code) { - return {OpcuaPlugin::PendingFaultDispatch::Kind::Clear, code, []() {}}; +// A clear the next reconnect will send again (PLC_COMMS_LOST). +OpcuaPlugin::PendingFaultDispatch link_state_clear_entry(const std::string & code) { + return {OpcuaPlugin::PendingFaultDispatch::Kind::Clear, code, /*link_state=*/true, []() {}}; +} + +// A clear nothing will re-send: the device reported its condition inactive, or +// an operator cleared through the scoped SOVD route. +OpcuaPlugin::PendingFaultDispatch device_clear_entry(const std::string & code) { + return {OpcuaPlugin::PendingFaultDispatch::Kind::Clear, code, /*link_state=*/false, []() {}}; } size_t count_kind(const std::vector & buffer, @@ -1210,14 +1390,15 @@ size_t count_kind(const std::vector & buffer, TEST(EnqueuePendingDispatch, ReconnectClearsNeverEvictABufferedAlarmReport) { // A flapping link with no fault_manager: 300 reconnects, each enqueueing a // connect-time clear, while ten real alarm reports wait to be flushed. The - // reports are one-shot edges from the PLC; the clears are re-derivable. + // reports are one-shot edges from the PLC, the clears are re-derivable. std::vector buffer; for (int i = 0; i < 10; ++i) { OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_" + std::to_string(i))); } for (int i = 0; i < 300; ++i) { - OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry(kCommsLostFaultCode)); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + link_state_clear_entry(kCommsLostFaultCode)); } EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Report), 10u) @@ -1229,7 +1410,7 @@ TEST(EnqueuePendingDispatch, ReconnectClearsNeverEvictABufferedAlarmReport) { } } -TEST(EnqueuePendingDispatch, AFullReportBufferRefusesAClearInsteadOfDroppingAReport) { +TEST(EnqueuePendingDispatch, AFullOneShotBufferRefusesALinkStateClearInsteadOfDroppingOne) { std::vector buffer; for (size_t i = 0; i < OpcuaPlugin::kMaxPendingDispatches; ++i) { OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, @@ -1238,24 +1419,25 @@ TEST(EnqueuePendingDispatch, AFullReportBufferRefusesAClearInsteadOfDroppingARep ASSERT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, - clear_entry(kCommsLostFaultCode)), + link_state_clear_entry(kCommsLostFaultCode)), OpcuaPlugin::PendingEnqueueOutcome::Refused); EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Report), OpcuaPlugin::kMaxPendingDispatches); - EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_0") << "the oldest report must survive an incoming clear"; + EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_0") << "the oldest report must survive an incoming link-state clear"; - // A report arriving at a full buffer still drops the oldest one: reports do - // not outrank each other, so the bound still holds. + // A report arriving at the same full buffer still drops the oldest entry: + // one-shot dispatches do not outrank each other, so the bound still holds. EXPECT_EQ( OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_NEW")), - OpcuaPlugin::PendingEnqueueOutcome::EvictedReport); + OpcuaPlugin::PendingEnqueueOutcome::EvictedOldest); EXPECT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_1"); EXPECT_EQ(buffer.back().fault_code, "PLC_ALARM_NEW"); } -TEST(EnqueuePendingDispatch, AFullBufferGivesUpAPendingClearBeforeAReport) { +TEST(EnqueuePendingDispatch, AFullBufferGivesUpALinkStateClearBeforeAReport) { std::vector buffer; - OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_OLD_CLEAR")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + link_state_clear_entry(kCommsLostFaultCode)); for (size_t i = 1; i < OpcuaPlugin::kMaxPendingDispatches; ++i) { OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_" + std::to_string(i))); @@ -1264,18 +1446,55 @@ TEST(EnqueuePendingDispatch, AFullBufferGivesUpAPendingClearBeforeAReport) { EXPECT_EQ( OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_NEW")), - OpcuaPlugin::PendingEnqueueOutcome::EvictedClear); + OpcuaPlugin::PendingEnqueueOutcome::EvictedLinkStateClear); EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 0u); - EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_1") << "the clear went, not the oldest report"; + EXPECT_EQ(buffer.front().fault_code, "PLC_ALARM_1") << "the re-derivable clear went, not the oldest report"; +} + +TEST(EnqueuePendingDispatch, ADeviceAlarmClearIsNotEvictedAheadOfAnOlderReport) { + // The device says an alarm went inactive while the fault_manager is + // unreachable. That edge is as one-shot as the raise: drop it and the flush + // replays the raise with nothing behind it, so the fault stands while the + // device reports it clear. Only the link-state clear is re-derivable. + std::vector buffer; + for (int i = 0; i < 100; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + report_entry("PLC_ALARM_" + std::to_string(i))); + } + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_TANK_HIGH")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + device_clear_entry("PLC_TANK_HIGH")); + for (size_t i = buffer.size(); i < OpcuaPlugin::kMaxPendingDispatches; ++i) { + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + report_entry("PLC_FILLER_" + std::to_string(i))); + } + ASSERT_EQ(buffer.size(), OpcuaPlugin::kMaxPendingDispatches); + + EXPECT_EQ( + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_ALARM_NEW")), + OpcuaPlugin::PendingEnqueueOutcome::EvictedOldest); + EXPECT_NE(buffer.front().fault_code, "PLC_ALARM_0") << "the oldest entry is what ages out"; + const auto device_clear = + std::find_if(buffer.begin(), buffer.end(), [](const OpcuaPlugin::PendingFaultDispatch & entry) { + return entry.kind == OpcuaPlugin::PendingFaultDispatch::Kind::Clear && entry.fault_code == "PLC_TANK_HIGH"; + }); + ASSERT_NE(device_clear, buffer.end()) << "a device alarm's inactive edge was evicted ahead of an older report"; + // ... and it still flushes after the raise it supersedes. + const auto raise = std::find_if(buffer.begin(), buffer.end(), [](const OpcuaPlugin::PendingFaultDispatch & entry) { + return entry.kind == OpcuaPlugin::PendingFaultDispatch::Kind::Report && entry.fault_code == "PLC_TANK_HIGH"; + }); + ASSERT_NE(raise, buffer.end()); + EXPECT_LT(raise - buffer.begin(), device_clear - buffer.begin()); } TEST(EnqueuePendingDispatch, ARequeuedClearMovesToTheBackSoOrderStillHolds) { // Report-then-clear for one code must still flush in that order after the // clear is re-enqueued, or the flush would leave the fault standing. std::vector buffer; - OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_FLAP")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, link_state_clear_entry("PLC_FLAP")); OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_FLAP")); - EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_FLAP")), + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + link_state_clear_entry("PLC_FLAP")), OpcuaPlugin::PendingEnqueueOutcome::ReplacedClear); ASSERT_EQ(buffer.size(), 2u); @@ -1283,7 +1502,7 @@ TEST(EnqueuePendingDispatch, ARequeuedClearMovesToTheBackSoOrderStillHolds) { EXPECT_EQ(buffer[1].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear) << "the newest clear must flush after the report it supersedes"; // Clears for DIFFERENT codes are independent. - OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, clear_entry("PLC_OTHER")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, device_clear_entry("PLC_OTHER")); EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 2u); } @@ -1791,4 +2010,92 @@ component_id: race_runtime << "flush_pending_reports never dispatched - swap-vs-push path not covered"; } +// The SOVD per-entity route DELETE /{entity}/faults/{code} lands on +// FaultProvider::clear_fault for a plugin-owned entity, which is the branch the +// gateway takes INSTEAD of its own (where it sets skip_correlation_auto_clear +// itself). So the flag has to be set here or the documented guarantee - an +// operator scoped to one entity cannot cascade-clear symptoms reported by apps +// in other entities - has a hole exactly where a PLC is involved. This drives +// the real route entry point and reads the field off the wire. +TEST(OpcuaPluginScopedClear, SovdDeleteSkipsTheCorrelationCascade) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_scoped_clear_flag"); + auto fault_manager = std::make_shared("opcua_scoped_clear_faultmgr"); + + std::mutex received_mutex; + std::vector received; + auto report_srv = fault_manager->create_service( + "/fault_manager/report_fault", [](const std::shared_ptr, + std::shared_ptr res) { + res->accepted = true; + }); + auto clear_srv = fault_manager->create_service( + "/fault_manager/clear_fault", + [&received, &received_mutex](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(received_mutex); + received.push_back(*req); + } + res->success = true; + }); + + const std::string yaml_path = "/tmp/test_opcua_scoped_clear_nodemap.yaml"; + { + std::ofstream f(yaml_path); + f << R"( +area_id: scoped_plc +component_id: scoped_runtime +nodes: + - node_id: "ns=2;i=1" + entity_id: tank + data_name: level + data_type: float +)"; + } + + OpcuaPlugin plugin; + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["endpoint_url"] = "opc.tcp://127.0.0.1:1"; // nothing listening, the fault sink is the subject + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/scoped_plc", "/scoped_plc/scoped_runtime/tank"}; + plugin.set_context(ctx); + + ScopedExecutorSpin spinner({node, fault_manager}); + auto probe = node->create_client("/fault_manager/clear_fault"); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!probe->service_is_ready() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_TRUE(probe->service_is_ready()) << "stub ClearFault server never became discoverable"; + + // The route's own entry point, not a helper it happens to call. + const auto result = plugin.clear_fault("tank", "PLC_TANK_HIGH"); + ASSERT_TRUE(result.has_value()); + + const auto flush_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + bool delivered = false; + while (!delivered && std::chrono::steady_clock::now() < flush_deadline) { + { + std::lock_guard lock(received_mutex); + delivered = !received.empty(); + } + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + + spinner.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + std::lock_guard lock(received_mutex); + ASSERT_FALSE(received.empty()) << "the scoped DELETE never reached the fault manager"; + EXPECT_EQ(received.front().fault_code, "PLC_TANK_HIGH"); + EXPECT_TRUE(received.front().skip_correlation_auto_clear) + << "a per-entity DELETE served by the plugin cascade-cleared correlated symptoms"; +} + } // namespace ros2_medkit_gateway From 59457b506c0a3ebdc20e9b7e27942adc30e9cfac Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 18:15:12 +0200 Subject: [PATCH 06/25] fix(gateway): keep the gateway's own node in the discovered apps The app filter now removes only the helper nodes that the gateway runs in its own process: "_sub", "_fault_clients" and "_lifecycle_state_reader". The gateway's own node stays an App. SOVD serves its ROS parameters as that App's configurations, and two gateways that watch one graph have to agree on what is on it. The predicate becomes is_own_gateway_helper_node, and the peer count still skips the gateway's own node. A feature test launches a gateway with its real process node names. The suite's usual launch remaps all four nodes to one name. The test pins both halves: the gateway is listed, addressable and configurable, and each helper is on the graph and absent from the app list. --- .../ros2_medkit_gateway/gateway_node.hpp | 33 +-- src/ros2_medkit_gateway/src/gateway_node.cpp | 27 +-- .../test/test_gateway_node.cpp | 6 +- .../test/test_handler_context.cpp | 35 +-- .../test/features/test_own_node_apps.test.py | 201 ++++++++++++++++++ 5 files changed, 260 insertions(+), 42 deletions(-) create mode 100644 src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index 89a153d1b..5517b8f38 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -498,17 +498,24 @@ class GatewayNode : public rclcpp::Node { }; /** - * @brief Is this node FQN the gateway's own, rather than a diagnosable peer? + * @brief Is this node FQN one of the helper nodes the gateway runs in-process? * - * True for the gateway node itself and for the helper nodes it creates inside - * its own process: the subscription executor's `_sub`, the fault-service + * True for the subscription executor's `_sub`, the fault-service * transport's `_fault_clients`, and the lifecycle reader's * `_lifecycle_state_reader`. None of these begins with '_', so the ROS 2 * hidden-node convention does not cover them and the gateway would otherwise - * count them as peers and list them as diagnosable Apps - reporting on itself. + * list its own plumbing as diagnosable Apps. They carry no parameters and no + * services of their own, so there is nothing to diagnose on them. * - * A fault_manager node sharing the process is NOT ours: it is a separate, - * diagnosable component and stays visible. + * False for the gateway node itself. The gateway IS a diagnosable App: its ROS + * parameters are served as that App's configurations, and callers read and + * write them at `/apps//configurations`. Excluding it would remove the + * only entity carrying, for instance, `aggregation.peer_auth_header`, and would + * make two gateways watching one graph disagree about that graph, because each + * would hide a different node. + * + * A fault_manager node sharing the process is NOT ours either: it is a + * separate, diagnosable component and stays visible. * * Exact matches only. A prefix test would also claim a genuine peer named * `_monitor` or `2`, and dropping a real node is the worse error. @@ -516,7 +523,7 @@ class GatewayNode : public rclcpp::Node { * @param node_fqn Fully qualified node name to test ("/ns/node") * @param self_fqn The gateway node's own FQN. An empty value matches nothing */ -bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_fqn); +bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn); /** * @brief Filter ROS 2 internal nodes from an app list @@ -526,14 +533,16 @@ bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_ * before checking for the underscore prefix, using the routing table for precise * prefix detection. * - * Also removes local apps bound to one of the gateway's own nodes - * (is_own_gateway_node), which the underscore rule cannot see. The test is on - * the bound node FQN, and only for apps with no routing-table entry: a peer's - * helper nodes are the peer's business and are left to the peer's own filter. + * Also removes local apps bound to one of the gateway's in-process helper nodes + * (is_own_gateway_helper_node), which the underscore rule cannot see. The + * gateway's own node is NOT removed - it is a diagnosable App whose ROS + * parameters are served as its configurations. The test is on the bound node + * FQN, and only for apps with no routing-table entry: a peer's helper nodes are + * the peer's business and are left to the peer's own filter. * * @param apps App vector to filter in place * @param peer_routing_table Maps entity_id -> peer_name for remote entities - * @param self_fqn The gateway node's own FQN. Empty disables the self check + * @param self_fqn The gateway node's own FQN. Empty disables the helper check * @return Number of apps removed */ size_t filter_internal_node_apps(std::vector & apps, diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index 0d4e8c5ff..e2bcfd589 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -1597,13 +1597,10 @@ GatewayNode::GatewayNode(const rclcpp::NodeOptions & options) : Node("ros2_medki }); } -bool is_own_gateway_node(const std::string & node_fqn, const std::string & self_fqn) { +bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn) { if (self_fqn.empty() || node_fqn.empty()) { return false; } - if (node_fqn == self_fqn) { - return true; - } // The helper nodes the gateway creates inside its own process, each named // after this node plus a fixed suffix. Where each one is set: // "_sub" Ros2SubscriptionExecutor::Config @@ -1630,7 +1627,10 @@ size_t GatewayNode::count_peer_nodes(const std::vector & apps, if (original_id.size() > prefix.size() && original_id.compare(0, prefix.size(), prefix) == 0) { original_id = original_id.substr(prefix.size()); } - } else if (is_own_gateway_node(app.effective_fqn(), self_fqn)) { - // A local app bound to one of this gateway's own nodes. Those names do - // not start with '_' ("_sub", "_fault_clients", ...), - // so only the FQN test catches them, and without it the gateway - // advertises its own plumbing as diagnosable apps. Remote entities are - // skipped deliberately: a peer's helper nodes carry the same FQNs and are - // the peer's own filter's business. + } else if (is_own_gateway_helper_node(app.effective_fqn(), self_fqn)) { + // A local app bound to one of this gateway's own helper nodes. Those + // names do not start with '_' ("_sub", + // "_fault_clients", ...), so only the FQN test catches them, and + // without it the gateway advertises its own plumbing as diagnosable apps. + // The gateway's own node is deliberately not covered: its ROS parameters + // are served as that App's configurations, so callers reach them at + // /apps//configurations. Remote entities are skipped too - a + // peer's helper nodes carry the same FQNs and are the peer's own filter's + // business. return true; } // ROS 2 internal nodes use _ prefix convention diff --git a/src/ros2_medkit_gateway/test/test_gateway_node.cpp b/src/ros2_medkit_gateway/test/test_gateway_node.cpp index 557aef176..a92edcc4e 100644 --- a/src/ros2_medkit_gateway/test/test_gateway_node.cpp +++ b/src/ros2_medkit_gateway/test/test_gateway_node.cpp @@ -1058,9 +1058,9 @@ TEST(GatewayStartupSummary, CountPeerNodesExcludesOwnAndHidden) { } TEST(GatewayStartupSummary, CountPeerNodesZeroWhenOnlyOwnNodes) { - // Every helper the gateway creates inside its own process. A gateway alone on - // the graph must report zero peers, so each helper has to be recognized - - // including the lifecycle reader, which the list previously omitted. + // The gateway node plus every helper it creates inside its own process. A + // gateway alone on the graph must report zero peers, so each of these has to + // be recognized as not-a-peer, the lifecycle reader included. const std::vector> nodes = { {"ros2_medkit_gateway", "/"}, {"ros2_medkit_gateway_sub", "/"}, diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 573c356c0..dfadb271b 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -1024,11 +1024,12 @@ App bound_app(const std::string & id, const std::string & fqn) { } // namespace -TEST(FilterInternalNodeAppsTest, DropsTheGatewaysOwnHelperNodes) { +TEST(FilterInternalNodeAppsTest, DropsTheGatewaysOwnHelperNodesButKeepsItsOwnNode) { // The gateway creates "_sub", "_fault_clients" and // "_lifecycle_state_reader" in its own process. None starts with '_', // so runtime introspection returns them as ordinary apps and the gateway ends - // up listing its own plumbing as diagnosable. + // up listing its own plumbing as diagnosable. Its own node is a different + // case and stays: its ROS parameters are served as that App's configurations. const std::string self_fqn = "/ros2_medkit_gateway"; std::vector apps{ bound_app("ros2_medkit_gateway", self_fqn), @@ -1047,12 +1048,13 @@ TEST(FilterInternalNodeAppsTest, DropsTheGatewaysOwnHelperNodes) { std::unordered_map routing; auto removed = filter_internal_node_apps(apps, routing, self_fqn); - EXPECT_EQ(removed, 4u); + EXPECT_EQ(removed, 3u); std::set remaining; for (const auto & app : apps) { remaining.insert(app.id); } - EXPECT_EQ(remaining, (std::set{"other_gateway_sub", "ros2_medkit_gateway_monitor", "fault_manager"})); + EXPECT_EQ(remaining, (std::set{"ros2_medkit_gateway", "other_gateway_sub", "ros2_medkit_gateway_monitor", + "fault_manager"})); } TEST(FilterInternalNodeAppsTest, LeavesPeerHelperNodesToThePeer) { @@ -1070,22 +1072,25 @@ TEST(FilterInternalNodeAppsTest, LeavesPeerHelperNodesToThePeer) { ASSERT_EQ(apps.size(), 1u); } -TEST(IsOwnGatewayNodeTest, MatchesSelfAndHelpersExactlyAndNothingElse) { +TEST(IsOwnGatewayHelperNodeTest, MatchesTheThreeHelperSuffixesExactlyAndNothingElse) { const std::string self_fqn = "/ros2_medkit_gateway"; - EXPECT_TRUE(is_own_gateway_node(self_fqn, self_fqn)); - EXPECT_TRUE(is_own_gateway_node(self_fqn + "_sub", self_fqn)); - EXPECT_TRUE(is_own_gateway_node(self_fqn + "_fault_clients", self_fqn)); - EXPECT_TRUE(is_own_gateway_node(self_fqn + "_lifecycle_state_reader", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_sub", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_lifecycle_state_reader", self_fqn)); + + // The gateway's own node is not a helper: it carries the parameters SOVD + // serves as configurations, so it stays a diagnosable App. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, self_fqn)); // Prefix neighbours are genuine peers, not ours. - EXPECT_FALSE(is_own_gateway_node(self_fqn + "_monitor", self_fqn)); - EXPECT_FALSE(is_own_gateway_node(self_fqn + "2", self_fqn)); - EXPECT_FALSE(is_own_gateway_node("/other" + self_fqn + "_sub", self_fqn)); - EXPECT_FALSE(is_own_gateway_node("/fault_manager", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn + "_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn + "2", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/other" + self_fqn + "_sub", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/fault_manager", self_fqn)); // An unknown self FQN must claim nothing rather than everything. - EXPECT_FALSE(is_own_gateway_node(self_fqn, "")); - EXPECT_FALSE(is_own_gateway_node("", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, "")); + EXPECT_FALSE(is_own_gateway_helper_node("", self_fqn)); } // ============================================================================= diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py new file mode 100644 index 000000000..5cb0cb459 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""What of the gateway's own process shows up in ``/apps``. + +The gateway runs four ROS nodes in one process: itself, the subscription +executor's ``_sub``, the fault-service transport's +``_fault_clients`` and the lifecycle reader's +``_lifecycle_state_reader``. None of the three helper names begins +with ``_``, so the ROS 2 hidden-node convention leaves them in the graph and +runtime discovery would turn each into an App - the gateway advertising its own +plumbing as something an operator can diagnose. + +The gateway's own node is the opposite case and stays an App: its ROS +parameters are what SOVD serves as that App's configurations, so +``/apps//configurations`` is the only place a caller can read or write +``aggregation.peer_auth_header`` and its neighbours. Two gateways watching one +graph also have to agree on what is on it, which they cannot do if each hides +a different node. + +This fixture launches the gateway with no ``__node`` remap, the way ``ros2 run`` +and the container images start it. ``launch_ros``' ``name=`` applies +``-r __node:=`` to the whole process, which renames all four nodes to the +same string - so under the suite's usual launch the helper names do not exist +and nothing here could be observed. +""" + +import time +import unittest + +import launch +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NODE = 'ros2_medkit_gateway' +HELPER_NODES = ( + f'{GATEWAY_NODE}_sub', + f'{GATEWAY_NODE}_fault_clients', + f'{GATEWAY_NODE}_lifecycle_state_reader', +) + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one bare gateway, keeping its process's real node names.""" + gateway_node = create_gateway_node( + port=GATEWAY_PORT, + name=None, + extra_params={'server.host': '127.0.0.1', 'refresh_interval_ms': 1000}, + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +def _graph_node_fqns(awaited, timeout=30.0): + """Fully qualified node names on the graph, waiting for *awaited*. + + Runs on its own rclpy context so it cannot disturb anything else in the + process, and polls: a graph query reads the discovery database directly and + needs no executor. + """ + context = Context() + rclpy.init(context=context) + probe = Node('own_node_apps_graph_probe', context=context) + try: + deadline = time.monotonic() + timeout * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + finally: + probe.destroy_node() + rclpy.shutdown(context=context) + + +class TestOwnNodeApps(unittest.TestCase): + """The gateway is a diagnosable App; its in-process helpers are not.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway to answer, then read its app list once.""" + cls.session = requests.Session() + deadline = time.monotonic() + 60.0 * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + break + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + else: + raise AssertionError(f'gateway not ready within 60s (last: {last})') + + # The app list is served from the discovery cache, which the first + # refresh fills; poll until the gateway's own node is in it or the + # budget is out, so the absence assertions below read a settled list. + cls.apps = set() + app_deadline = time.monotonic() + 30.0 * get_time_scale() + while time.monotonic() < app_deadline: + body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() + cls.apps = {item['id'] for item in body.get('items', [])} + if GATEWAY_NODE in cls.apps: + break + time.sleep(0.5) + + @classmethod + def tearDownClass(cls): + cls.session.close() + + def test_the_gateways_own_node_is_an_app(self): + """The gateway's own ROS node is listed, addressable and configurable. + + @verifies REQ_INTEROP_003 + """ + self.assertIn( + GATEWAY_NODE, self.apps, + f'the gateway node must be a diagnosable App. Listed: {sorted(self.apps)}') + + detail = self.session.get(f'{BASE_URL}/apps/{GATEWAY_NODE}', timeout=10) + self.assertEqual(detail.status_code, 200, detail.text) + self.assertEqual(detail.json()['id'], GATEWAY_NODE) + + configurations = self.session.get( + f'{BASE_URL}/apps/{GATEWAY_NODE}/configurations', timeout=10) + self.assertEqual(configurations.status_code, 200, configurations.text) + items = configurations.json().get('items', []) + self.assertTrue( + items, + 'the gateway App must serve its own ROS parameters as configurations; ' + 'they are reachable nowhere else') + + def test_the_in_process_helper_nodes_are_not_apps(self): + """The gateway's own helper nodes stay out of the app list. + + @verifies REQ_INTEROP_003 + """ + for helper in HELPER_NODES: + self.assertNotIn( + helper, self.apps, + f'in-process helper "{helper}" must not be a diagnosable App. ' + f'Listed: {sorted(self.apps)}') + + # Anti-vacuity: each helper really is a node on this graph, so the + # assertions above are about the filter and not about names that were + # never there. Without this the test passes on a gateway that creates + # no helpers at all. + awaited = {f'/{helper}' for helper in HELPER_NODES} + graph_fqns = _graph_node_fqns(awaited) + self.assertTrue( + awaited <= graph_fqns, + f'helper nodes missing from the graph: {sorted(awaited - graph_fqns)}. ' + f'Nodes seen: {sorted(graph_fqns)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') From e3b3d32d90fe7e1b94c804fd724dbadb59f0ad7e Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 18:22:18 +0200 Subject: [PATCH 07/25] test(gateway): pin the stop response on the cancel contract and a running goal An accepted stop leaves the goal CANCELING or CANCELED. Which one a caller sees depends on how fast the server winds down. The body renders the status that the handler read. That status is at or before the one the test reads afterwards. A goal still CANCELING therefore pins the body exactly, and a goal already CANCELED allows either rendering. The test still rules out a goal that keeps running or completes. The fixture server succeeds at the requested sequence length, one element per 100 ms tick, so the order sets how long the goal stays cancellable. At 20 the goal finishes on its own in under two seconds. A loaded or instrumented runner can spend that time on the create and the cancel round trip, and the assertions then describe a completed goal. The test uses 50, the largest order the server accepts. The tracked-goal helper returns its optional, and the caller asserts, so a missing goal gives a readable failure. Dereferencing it after a non-fatal expectation would be undefined behaviour. --- .../test/test_operation_handlers.cpp | 44 ++++++++++++++++--- 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp index e5ec51125..1ed5aa2cd 100644 --- a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp @@ -524,10 +524,14 @@ class OperationHandlersFixtureTest : public ::testing::Test { return async_ptr->id; } - ActionGoalInfo get_tracked_goal_or_fail(const std::string & execution_id) { + // Returns the optional rather than dereferencing it: a non-fatal expectation + // followed by an unconditional `*goal_info` turns a missing goal into + // undefined behaviour instead of a failure anyone can read. The caller + // ASSERTs, which is what stops the test. + std::optional get_tracked_goal_or_fail(const std::string & execution_id) { auto goal_info = gateway_node_->get_operation_manager()->get_tracked_goal(execution_id); EXPECT_TRUE(goal_info.has_value()); - return *goal_info; + return goal_info; } CorsConfig cors_{}; @@ -978,7 +982,15 @@ TEST_F(OperationHandlersFixtureTest, GetOperationResolvesAQualifiedIdToItsMember } TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocation) { - const auto execution_id = create_action_execution(20); + // The fixture server pushes one sequence element per 100 ms tick and succeeds + // at the requested length, so the order sets how long the goal stays + // cancellable: 20 finishes on its own in under two seconds, which a loaded or + // instrumented runner can spend on the create plus the cancel round trip, and + // the goal is then SUCCEEDED with nothing left to stop. 50 is the largest + // order handle_goal accepts and buys about five seconds, which is the whole + // point of asking for it - the assertions below are about an accepted stop, + // not about how fast the machine is. + const auto execution_id = create_action_execution(50); ASSERT_FALSE(execution_id.empty()); auto raw_req = @@ -989,7 +1001,9 @@ TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocati body.capability = "stop"; auto result = handlers_->update_execution(typed, body); - auto goal_info = get_tracked_goal_or_fail(execution_id); + auto tracked = get_tracked_goal_or_fail(execution_id); + ASSERT_TRUE(tracked.has_value()); + const auto & goal_info = *tracked; if (result.has_value()) { const auto & exec = result.value().first.value; @@ -1007,8 +1021,26 @@ TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocati EXPECT_TRUE(has_location); ASSERT_TRUE(exec.id.has_value()); EXPECT_EQ(*exec.id, execution_id); - EXPECT_EQ(exec.status, "running"); - EXPECT_EQ(goal_info.status, ActionGoalStatus::CANCELING); + + // An accepted stop promises the goal is on its way out: CANCELING while + // the server winds down, CANCELED once it has. It never promises which of + // the two the caller observes, and a server that cancels within the + // round trip lands on CANCELED directly. What it does rule out is a goal + // that is still running (ACCEPTED, EXECUTING) or one that completed + // anyway (SUCCEEDED): those mean the stop did not take. + EXPECT_TRUE(goal_info.status == ActionGoalStatus::CANCELING || goal_info.status == ActionGoalStatus::CANCELED) + << "tracked status after an accepted stop: " << ros2_medkit_gateway::action_status_to_string(goal_info.status); + + // The body renders the status the handler read, which is at or before the + // one read above - a goal only moves CANCELING -> CANCELED, never back. So + // a goal still CANCELING here cannot have been CANCELED when the handler + // looked, which pins the body exactly; a goal already CANCELED admits + // either rendering. + if (goal_info.status == ActionGoalStatus::CANCELING) { + EXPECT_EQ(exec.status, "running"); + } else { + EXPECT_TRUE(exec.status == "running" || exec.status == "failed") << "execution status in body: " << exec.status; + } } else { // The fixture's action server always ACCEPTS cancels, so the only // realistic failure here is a lost/late CancelGoal response whose From 03c5e68a4e4c9d0e0362866ca70ea83bb40cd4e7 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 19:18:26 +0200 Subject: [PATCH 08/25] test(opcua): tear the executor spin down on every exit path The e2e tests that drive the plugin against a live fault-manager stub spin a MultiThreadedExecutor on a thread of their own. A gtest ASSERT_* returns from the middle of the test body. A std::thread destroyed while it is still joinable calls std::terminate. The run then ends in SIGABRT, the assertion message never reaches the report, and the alarm-server child outlives the test and holds its port. An RAII guard now owns the thread. stop() cancels the executor and then joins the thread. A cancel issued before spin() has begun is refused and lost, so the guard issues the cancel again until the spin function reports that it returned, and gives up with a message after 10 s. A cancel that throws is contained and issued again, and the join runs whatever the cancel did. stop() is idempotent. It sets its flag after the join, so a failed attempt can run again. The destructor catches what cancel() or join() throw and prints it, because an exception that escapes a destructor also ends in std::terminate. The test has recorded its verdict by then. The guard takes the cancel as a callable that defaults to the executor's own. rclcpp::Executor::cancel() is virtual on Jazzy and later but not on Humble. There, a derived executor's cancel() does not compile with override, and a call through a base reference would not reach it. A test passes a cancel that throws after it stops the spin, and the test reaches the end of its body. --- .../test/test_opcua_identity.cpp | 146 ++++++++++++++++-- 1 file changed, 132 insertions(+), 14 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index f96daf0e3..8c6ea478b 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -41,13 +41,18 @@ #include #include +#include #include #include #include +#include #include +#include +#include #include #include #include +#include #include #include #include @@ -682,6 +687,101 @@ struct ScopedRclcpp { ScopedRclcpp & operator=(const ScopedRclcpp &) = delete; }; +// Spins an executor on its own thread and guarantees cancel -> join on every +// exit path. A gtest ASSERT_* returns from the middle of the test body, so a +// bare std::thread would be destroyed while still joinable, and that calls +// std::terminate: the run ends in SIGABRT and the assertion message that says +// what actually failed never reaches the report. +class ScopedExecutorSpin { + public: + using CancelFn = std::function; + + // The cancel is injectable so a test can make it fail. A callable rather than + // a virtual override on a derived executor because rclcpp::Executor::cancel() + // is virtual on jazzy and later but NOT on humble, where a subclass's + // cancel() would neither compile with `override` nor be the one called + // through a base reference. + explicit ScopedExecutorSpin(rclcpp::executors::MultiThreadedExecutor & executor, CancelFn cancel = nullptr) + : executor_(executor) + , cancel_(cancel ? std::move(cancel) : CancelFn([this]() { + executor_.cancel(); + })) + , thread_([this]() { + executor_.spin(); + spin_returned_.store(true); + }) { + } + + ~ScopedExecutorSpin() { + // A destructor is implicitly noexcept, and both cancel() and join() can + // throw, so an escape here would be the std::terminate this class exists + // to prevent. Swallowing is right in a destructor: by this point the test + // has either passed or recorded its failure, and that verdict is what the + // run has to report. + try { + stop(); + } catch (const std::exception & e) { + std::cerr << "ScopedExecutorSpin teardown failed: " << e.what() << "\n"; + } catch (...) { + std::cerr << "ScopedExecutorSpin teardown failed\n"; + } + } + + // Idempotent, so a test can end the spin at the point it wants the executor + // quiet and still be covered on the paths that never get there. + // + // The join is unconditional on the cancel's outcome. cancel() throws if the + // guard condition cannot be triggered, and letting that skip the join would + // move the terminate from this class's destructor - where the catch above can + // report it - into the std::thread member's destructor, which no catch here + // can reach: ~std::thread calls std::terminate on a joinable thread. The + // flag goes last so a failed teardown leaves the object willing to try again. + void stop() { + if (stopped_) { + return; + } + // cancel() refuses with an exception while the executor is not spinning, + // and a cancel that lands before spin() has begun is simply lost - the + // thread then spins for good and the join below never returns. So the + // cancel is re-issued until the spin function has actually returned, which + // only the spin thread can report. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!spin_returned_.load()) { + try { + cancel_(); + } catch (const std::exception &) { + // Not spinning yet, or the guard condition could not be triggered. The + // next attempt is what resolves either case. + } + if (std::chrono::steady_clock::now() >= deadline) { + std::cerr << "ScopedExecutorSpin: executor still spinning after 10s of cancel attempts\n"; + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + if (thread_.joinable()) { + // Nothing is left to try if this throws: the thread stays joinable and + // its own destructor ends the process. That is a clean abort with a + // reason, which is the honest outcome - detaching instead would leave a + // live executor thread running against nodes about to be destroyed. + thread_.join(); + } + stopped_ = true; + } + + ScopedExecutorSpin(const ScopedExecutorSpin &) = delete; + ScopedExecutorSpin & operator=(const ScopedExecutorSpin &) = delete; + ScopedExecutorSpin(ScopedExecutorSpin &&) = delete; + ScopedExecutorSpin & operator=(ScopedExecutorSpin &&) = delete; + + private: + rclcpp::executors::MultiThreadedExecutor & executor_; + CancelFn cancel_; + std::atomic spin_returned_{false}; + std::thread thread_; + bool stopped_{false}; +}; + // The plugin only builds its fault-service clients when the context hands it a // real node, which is what makes the ClearFault request observable on the wire. class RealNodePluginContext : public FakePluginContext { @@ -698,6 +798,34 @@ class RealNodePluginContext : public FakePluginContext { } // namespace +// A cancel() that throws must not cost the join. If it does, the guard's thread +// member is destroyed while joinable and ~std::thread calls std::terminate, so +// this test does not fail - it takes the whole binary down with SIGABRT, which +// is why it asserts on having reached the end at all. +TEST(ScopedExecutorSpinTest, AThrowingCancelStillJoinsTheThread) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("scoped_spin_throwing_cancel"); + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + + { + // Fails the way rclcpp documents cancel() can - the guard condition cannot + // be triggered - after actually stopping the spin, so what is under test is + // the join and not a hang. Injected rather than overridden: cancel() is not + // virtual on every distro this builds on. + ScopedExecutorSpin spin(executor, [&executor]() { + executor.cancel(); + throw std::runtime_error("cancel failed"); + }); + // stop() contains the throw itself, so the explicit teardown a test does at + // the point it wants the executor quiet stays usable. + EXPECT_NO_THROW(spin.stop()); + } + + executor.remove_node(node); + SUCCEED() << "the guard joined its thread despite cancel() throwing"; +} + // The connect-time clear, read off the wire. clear_comms_lost_on_connect() is // only reachable through a connect that SUCCEEDS, so it needs the live fixture, // and the flag it sets is only observable with a real fault-manager service on @@ -730,9 +858,7 @@ TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade rclcpp::executors::MultiThreadedExecutor executor; executor.add_node(node); executor.add_node(fault_manager); - std::thread spin_thread([&executor]() { - executor.spin(); - }); + ScopedExecutorSpin spin(executor); const std::string yaml_path = write_minimal_node_map(); OpcuaPlugin plugin; @@ -760,10 +886,7 @@ TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade std::this_thread::sleep_for(std::chrono::milliseconds(50)); } - executor.cancel(); - if (spin_thread.joinable()) { - spin_thread.join(); - } + spin.stop(); plugin.shutdown(); std::remove(yaml_path.c_str()); @@ -811,9 +934,7 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) rclcpp::executors::MultiThreadedExecutor executor; executor.add_node(node); executor.add_node(fault_manager); - std::thread spin_thread([&executor]() { - executor.spin(); - }); + ScopedExecutorSpin spin(executor); OpcuaPlugin plugin; nlohmann::json config; @@ -864,10 +985,7 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) std::this_thread::sleep_for(std::chrono::milliseconds(100)); } - executor.cancel(); - if (spin_thread.joinable()) { - spin_thread.join(); - } + spin.stop(); plugin.shutdown(); const auto device_clear_skips = clear_for(alarm_code); From cd80c13253ebbd29d61437953d375029e72de6de Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 19:49:23 +0200 Subject: [PATCH 09/25] test(opcua): keep every alarm-server call on the server's own thread The fixture's open62541 is built with UA_MULTITHREADING 0, so only one thread may use the server. The command handler ran on a second thread. It wrote condition fields and triggered condition events there while UA_Server_run iterated on the main thread. The mutex it held was taken nowhere else, so it protected nothing against the server. The reader thread now only queues command lines. The main thread drives the server with UA_Server_run_startup, run_iterate and run_shutdown, and it runs each queued command between iterations. The address space and the Alarms & Conditions subsystem have a single user. The reader ends on EOF. The harness produces it by closing its write end of the pipe before it signals the server. The quit branch does not close the process's own stdin, because close(2) does not unblock a reader that already waits in read(2). Startup and shutdown each report their own bad status on exit. The return value of run_iterate is how long the server may idle, so it bounds the pause between iterations. --- .../test_alarm_server/test_alarm_server.cpp | 94 +++++++++++++++---- 1 file changed, 75 insertions(+), 19 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp index 7085543c2..e49b9a365 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp @@ -28,10 +28,13 @@ #include #include +#include #include +#include #include #include #include +#include #include #include #include @@ -41,6 +44,7 @@ #include #include #include +#include namespace { @@ -69,7 +73,14 @@ void log_state(const Condition & c) { } std::map g_conditions; -std::mutex g_mutex; +// Guards the command queue between the stdin reader and the server thread. +// Every UA_Server_* call in this process runs on the server thread: this +// open62541 is built with UA_MULTITHREADING 0, so a second thread touching the +// address space while the server iterates is an unsynchronised use of a +// single-threaded library - and the subsystem that breaks first is the one +// under test, Alarms & Conditions. +std::mutex g_commands_mutex; +std::deque g_commands; std::atomic g_running{true}; void stop_handler(int) { @@ -605,18 +616,33 @@ void add_di_nameplate(UA_Server * server, const std::string & serial) { UA_NODEID_NUMERIC(0, UA_NS0ID_PROPERTYTYPE), va_order, nullptr, nullptr); } -void cli_loop(UA_Server * server, UA_UInt16 ns) { +// Reads command lines and queues them. Deliberately free of UA_Server_* calls: +// this thread blocks in getline for most of the fixture's life and must not +// reach into the server while it is iterating. +void stdin_reader_loop() { std::string line; while (g_running && std::getline(std::cin, line)) { + std::lock_guard guard(g_commands_mutex); + g_commands.push_back(line); + } +} + +// Runs one queued command. Called from the server thread only. +void execute_command(UA_Server * server, UA_UInt16 ns, const std::string & line) { + { std::istringstream iss(line); std::string cmd, name; iss >> cmd >> name; if (cmd == "quit") { g_running = false; std::cout << "OK quit" << std::endl; - break; + // Only the server loop stops here. What ends the reader is EOF, which + // the writer produces by closing its end of the pipe - close(2) on this + // process's own descriptor does not release a reader already parked in + // read(2). So the join below waits for the caller to let go of stdin, + // which AlarmServer::stop() does before it signals the server. + return; } - std::lock_guard guard(g_mutex); // ``set `` writes a polled Int32 variable (StatusWord / // FaultCode). Handled before the condition lookup because these nodes are // plain variables, not AlarmCondition instances in g_conditions. @@ -624,7 +650,7 @@ void cli_loop(UA_Server * server, UA_UInt16 ns) { long val = 0; if (!(iss >> val)) { std::cout << "ERR set_missing_value:" << name << std::endl; - continue; + return; } UA_Int32 v32 = static_cast(val); UA_Variant var; @@ -637,7 +663,7 @@ void cli_loop(UA_Server * server, UA_UInt16 ns) { } else { std::cout << "ERR " << name << ":" << UA_StatusCode_name(rc) << std::endl; } - continue; + return; } // ``sysevent`` fires a non-condition BaseEventType on the Server object // (i=2253); it has no and is not in g_conditions, so handle it @@ -649,12 +675,12 @@ void cli_loop(UA_Server * server, UA_UInt16 ns) { } else { std::cout << "ERR sysevent:" << UA_StatusCode_name(rc) << std::endl; } - continue; + return; } auto it = g_conditions.find(name); if (cmd != "quit" && it == g_conditions.end()) { std::cout << "ERR unknown_condition:" << name << std::endl; - continue; + return; } Condition & cref = it->second; UA_StatusCode rc = UA_STATUSCODE_BADNOTSUPPORTED; @@ -713,7 +739,7 @@ void cli_loop(UA_Server * server, UA_UInt16 ns) { } } else { std::cout << "ERR unknown_cmd:" << cmd << std::endl; - continue; + return; } if (rc == UA_STATUSCODE_GOOD) { std::cout << "OK " << name << std::endl; @@ -822,18 +848,48 @@ int main(int argc, char ** argv) { } std::cout << "READY port=" << port << " namespace=" << ns << " secure=" << (secure ? "true" : "false") << std::endl; - std::thread cli(cli_loop, server, ns); - - UA_StatusCode rc = UA_Server_run(server, reinterpret_cast(&g_running)); - g_running = false; - // The exit code is 1 for every bad status, so print the status itself: - // otherwise a server that ends on its own leaves the driving test with a - // closed stdin pipe and no reason anywhere. + std::thread reader(stdin_reader_loop); + + // The server is driven by hand rather than by UA_Server_run so that queued + // commands execute between iterations, on this thread. run_iterate is called + // with waitInternal false and the loop paced by a short sleep: blocking + // inside the server would hold a command back until the next scheduled + // callback, and a command is how a test makes the alarm it is waiting for + // happen. + UA_StatusCode rc = UA_Server_run_startup(server); if (rc != UA_STATUSCODE_GOOD) { - std::cout << "EXIT UA_Server_run rc=" << UA_StatusCode_name(rc) << std::endl; + std::cout << "EXIT UA_Server_run_startup rc=" << UA_StatusCode_name(rc) << std::endl; + } + while (rc == UA_STATUSCODE_GOOD && g_running) { + // The return is how long the server may idle until its next scheduled + // callback, not a status: open62541 reports no mid-run failure through it, + // and UA_Server_run did not either - its own return is run_shutdown's. It + // does bound the pause below, which is otherwise kept small so a queued + // command is never held back by a server that has nothing to do. + const UA_UInt16 idle_ms = UA_Server_run_iterate(server, false); + for (;;) { + std::string line; + { + std::lock_guard guard(g_commands_mutex); + if (g_commands.empty()) { + break; + } + line = std::move(g_commands.front()); + g_commands.pop_front(); + } + execute_command(server, ns, line); + } + std::this_thread::sleep_for(std::chrono::milliseconds(std::clamp(idle_ms, 1, 2))); } - if (cli.joinable()) { - cli.join(); + if (rc == UA_STATUSCODE_GOOD) { + rc = UA_Server_run_shutdown(server); + if (rc != UA_STATUSCODE_GOOD) { + std::cout << "EXIT UA_Server_run_shutdown rc=" << UA_StatusCode_name(rc) << std::endl; + } + } + g_running = false; + if (reader.joinable()) { + reader.join(); } UA_Server_delete(server); return rc == UA_STATUSCODE_GOOD ? 0 : 1; From 0a7d3188d3b306ee48801f3c5e8d67781292c202 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 21:16:51 +0200 Subject: [PATCH 10/25] fix(gateway): recognise a namespaced gateway's helper nodes The three helper FQNs are not built the same way. The subscription executor passes the gateway's own namespace, so its node follows the gateway. The fault-client and lifecycle-reader nodes are built from the gateway's node name alone and take the process default namespace. A remap that names only the gateway, "-r :__ns:=/x", moves the gateway and the subscription node and leaves the other two behind. The gateway then served both of them as diagnosable apps and counted them as peers. The predicate derives each helper's FQN from its own creation site. It matches both spellings for the two nodes that do not follow the namespace. A declared app that is bound to a helper node is dropped with a warning that names the app and the node. An app is declared when its source is the manifest, the inventory or a plugin. --- .../ros2_medkit_gateway/gateway_node.hpp | 15 +++- src/ros2_medkit_gateway/src/gateway_node.cpp | 68 ++++++++++++++--- .../test/test_handler_context.cpp | 73 +++++++++++++++++++ 3 files changed, 143 insertions(+), 13 deletions(-) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index 5517b8f38..e636d9381 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -520,6 +520,14 @@ class GatewayNode : public rclcpp::Node { * Exact matches only. A prefix test would also claim a genuine peer named * `_monitor` or `2`, and dropping a real node is the worse error. * + * Two FQN spellings are recognised, because the three creation sites do not + * agree on the namespace: the subscription node is created with the gateway's + * own namespace, while the fault-client and lifecycle-reader nodes are created + * from the gateway's node NAME alone and so take the process default. A + * node-specific namespace remap on the gateway (`-r :__ns:=/x`) moves + * the gateway and the subscription node and leaves the other two behind, which + * is why those two are also matched as `/`. + * * @param node_fqn Fully qualified node name to test ("/ns/node") * @param self_fqn The gateway node's own FQN. An empty value matches nothing */ @@ -543,10 +551,15 @@ bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string * @param apps App vector to filter in place * @param peer_routing_table Maps entity_id -> peer_name for remote entities * @param self_fqn The gateway node's own FQN. Empty disables the helper check + * @param dropped_declared_apps Optional sink for " -> " of + * every app removed by the helper rule whose source is not runtime + * discovery. Removing a declared entity silently would override the + * manifest without saying so, and this function has no logger * @return Number of apps removed */ size_t filter_internal_node_apps(std::vector & apps, const std::unordered_map & peer_routing_table, - const std::string & self_fqn); + const std::string & self_fqn, + std::vector * dropped_declared_apps = nullptr); } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index e2bcfd589..c4fd9ea55 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -32,6 +32,7 @@ #include "ros2_medkit_gateway/core/aggregation/network_utils.hpp" #include "ros2_medkit_gateway/core/data/topic_data_provider.hpp" +#include "ros2_medkit_gateway/core/discovery/merge_types.hpp" #include "ros2_medkit_gateway/core/discovery/refresh_debounce.hpp" #include "ros2_medkit_gateway/core/entity_validation.hpp" #include "ros2_medkit_gateway/core/faults/fault_scope.hpp" @@ -1601,17 +1602,40 @@ bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string if (self_fqn.empty() || node_fqn.empty()) { return false; } - // The helper nodes the gateway creates inside its own process, each named - // after this node plus a fixed suffix. Where each one is set: - // "_sub" Ros2SubscriptionExecutor::Config - // (subscription_node_name_suffix) - // "_fault_clients" Ros2FaultServiceTransport - // "_lifecycle_state_reader" Ros2LifecycleStateReader + // The helper nodes the gateway creates inside its own process. Each one's FQN + // is fixed by how its creation site builds the node, which is not the same + // for all three: + // "_sub" Ros2SubscriptionExecutor passes the gateway's + // own namespace (ros2_subscription_executor.cpp), + // so this one always shares it. + // "_fault_clients" Ros2FaultServiceTransport and + // "_lifecycle_state_reader" Ros2LifecycleStateReader build their node from + // the gateway's node NAME alone, so they take + // whatever namespace the process defaults to. + // Usually that is the gateway's namespace too and all three spellings + // coincide. They come apart when only the gateway is moved - a node-specific + // remap, `-r :__ns:=/x` - which leaves the last two where the + // process default put them. Both spellings are then ours, so both are + // matched for those two; `_sub` is matched only in the gateway's namespace, + // because a root-namespace `_sub` provably belongs to another process. + struct HelperNode { + const char * suffix; + bool follows_gateway_namespace; + }; + static constexpr std::array kHelperNodes{{ + {"_sub", true}, + {"_fault_clients", false}, + {"_lifecycle_state_reader", false}, + }}; + const auto last_slash = self_fqn.rfind('/'); + const std::string bare_name = last_slash == std::string::npos ? self_fqn : self_fqn.substr(last_slash + 1); // Exact matches only: a prefix test would also claim a genuine peer named // "_monitor" or "2", and hiding a real node is the worse error. - static constexpr std::array kHelperSuffixes{"_sub", "_fault_clients", "_lifecycle_state_reader"}; - return std::any_of(kHelperSuffixes.begin(), kHelperSuffixes.end(), [&](const char * suffix) { - return node_fqn == self_fqn + suffix; + return std::any_of(kHelperNodes.begin(), kHelperNodes.end(), [&](const HelperNode & helper) { + if (node_fqn == self_fqn + helper.suffix) { + return true; + } + return !helper.follows_gateway_namespace && !bare_name.empty() && node_fqn == "/" + bare_name + helper.suffix; }); } @@ -2482,10 +2506,19 @@ void GatewayNode::refresh_cache() { // Covers local heuristic apps (which bypass the merge pipeline orphan filter // in runtime_only mode) and any peer apps that slipped through fetch_entities. if (filter_internal_nodes_) { - auto removed = filter_internal_node_apps(apps, peer_routing_table, get_fully_qualified_name()); + std::vector dropped_declared_apps; + auto removed = + filter_internal_node_apps(apps, peer_routing_table, get_fully_qualified_name(), &dropped_declared_apps); if (removed > 0) { RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix or own helper node)", removed); } + for (const auto & dropped : dropped_declared_apps) { + RCLCPP_WARN(get_logger(), + "Declared app '%s' is bound to one of this gateway's own in-process helper nodes and is not " + "served. Those nodes carry no parameters, services or actions to diagnose. Bind the app to the " + "node you meant, or drop the declaration.", + dropped.c_str()); + } } // Capture sizes for logging @@ -2582,9 +2615,9 @@ void GatewayNode::stop_rest_server() { size_t filter_internal_node_apps(std::vector & apps, const std::unordered_map & peer_routing_table, - const std::string & self_fqn) { + const std::string & self_fqn, std::vector * dropped_declared_apps) { auto before = apps.size(); - auto end = std::remove_if(apps.begin(), apps.end(), [&peer_routing_table, &self_fqn](const App & app) { + auto end = std::remove_if(apps.begin(), apps.end(), [&](const App & app) { std::string original_id = app.id; auto rt_it = peer_routing_table.find(app.id); if (rt_it != peer_routing_table.end()) { @@ -2604,6 +2637,17 @@ size_t filter_internal_node_apps(std::vector & apps, // /apps//configurations. Remote entities are skipped too - a // peer's helper nodes carry the same FQNs and are the peer's own filter's // business. + // + // Dropping a runtime-discovered app is the whole point and stays quiet. + // Dropping one somebody DECLARED is different: in manifest and hybrid + // mode the manifest is the source of truth, so removing an entry from it + // without a word is a silent override. The declared sources are the ones + // the merge pipeline already protects from orphan suppression, so the + // same predicate decides it here; those are reported to the caller, which + // owns the logger. + if (dropped_declared_apps != nullptr && discovery::is_protected_source(app.source)) { + dropped_declared_apps->push_back(app.id + " -> " + app.effective_fqn()); + } return true; } // ROS 2 internal nodes use _ prefix convention diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index dfadb271b..7d1fc1aae 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -1093,6 +1093,79 @@ TEST(IsOwnGatewayHelperNodeTest, MatchesTheThreeHelperSuffixesExactlyAndNothingE EXPECT_FALSE(is_own_gateway_helper_node("", self_fqn)); } +TEST(IsOwnGatewayHelperNodeTest, RecognizesHelpersOfANamespacedGateway) { + // A gateway moved on its own - `-r :__ns:=/subsystem_b` - keeps the + // subscription node with it, because the executor passes the gateway's + // namespace, while the fault-client and lifecycle-reader nodes are built from + // the node name alone and stay in the process default namespace. All three + // are still this gateway's plumbing. + const std::string self_fqn = "/subsystem_b/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_sub", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // A process-wide namespace remap moves all four together, so the in-namespace + // spelling has to keep working for the same two. + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // The subscription node always shares the gateway's namespace, so a + // root-namespace one belongs to a different gateway process and is that + // gateway's own filter's business. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_sub", self_fqn)); + + // Still nothing else: a peer in either namespace, and the gateway itself. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_b/fault_manager", self_fqn)); +} + +TEST(FilterInternalNodeAppsTest, ReportsOnlyDeclaredAppsItDropsAsHelpers) { + // A runtime-discovered helper app is what this filter exists to remove, and + // saying so on every refresh would be noise. A DECLARED one is a manifest or + // plugin entity being overridden, which the caller has to be able to log. + const std::string self_fqn = "/ros2_medkit_gateway"; + App runtime_helper = bound_app("ros2_medkit_gateway_sub", self_fqn + "_sub"); + runtime_helper.source = "heuristic"; + App declared_helper = bound_app("plc_bridge", self_fqn + "_fault_clients"); + declared_helper.source = "manifest"; + std::vector apps{runtime_helper, declared_helper}; + + std::unordered_map routing; + std::vector dropped_declared; + auto removed = filter_internal_node_apps(apps, routing, self_fqn, &dropped_declared); + + EXPECT_EQ(removed, 2u); + EXPECT_TRUE(apps.empty()); + ASSERT_EQ(dropped_declared.size(), 1u); + EXPECT_EQ(dropped_declared[0], "plc_bridge -> /ros2_medkit_gateway_fault_clients"); +} + +TEST(FilterInternalNodeAppsTest, DropsTheHelpersOfANamespacedGateway) { + // The same split at the level callers see: two of the three helpers of a + // namespaced gateway live in the root namespace, and without them being + // recognised the gateway lists its own plumbing. + const std::string self_fqn = "/subsystem_b/ros2_medkit_gateway"; + std::vector apps{ + bound_app("ros2_medkit_gateway", self_fqn), + bound_app("ros2_medkit_gateway_sub", "/subsystem_b/ros2_medkit_gateway_sub"), + bound_app("ros2_medkit_gateway_fault_clients", "/ros2_medkit_gateway_fault_clients"), + bound_app("ros2_medkit_gateway_lifecycle_state_reader", "/ros2_medkit_gateway_lifecycle_state_reader"), + bound_app("other_gateway", "/subsystem_c/other_gateway"), + }; + + std::unordered_map routing; + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 3u); + std::set remaining; + for (const auto & app : apps) { + remaining.insert(app.id); + } + EXPECT_EQ(remaining, (std::set{"ros2_medkit_gateway", "other_gateway"})); +} + // ============================================================================= // Area fault/log aggregation handler tests (via REST API) // ============================================================================= From 54e74b5ecf7ba622a6c714556d239a7ce17a5052 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 21:17:12 +0200 Subject: [PATCH 11/25] fix(gateway): stop reporting the gateway's own helper nodes as undeclared The undeclared-node warning on GET /health is an instruction: "Declare them in the manifest". It named the three nodes that the gateway runs in its own process. A manifest app declared for one of them is then removed by the app filter, so the two halves contradicted each other. The orphan scan in RuntimeLinker skips the helper nodes with the same predicate that the app filter uses. A linker built without a node has an empty self FQN, which matches nothing. A feature test drives a hybrid gateway with "unmanifested_nodes: error" and pins both halves: the report names an undeclared node of the test's own and none of the helpers. --- .../src/discovery/manifest/runtime_linker.cpp | 15 +- .../features/test_own_node_undeclared.test.py | 246 ++++++++++++++++++ 2 files changed, 260 insertions(+), 1 deletion(-) create mode 100644 src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py diff --git a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp index 565a23736..0923e77f7 100644 --- a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp +++ b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp @@ -16,6 +16,7 @@ #include "ros2_medkit_gateway/core/discovery/merge_types.hpp" #include "ros2_medkit_gateway/core/http/warning_codes.hpp" +#include "ros2_medkit_gateway/gateway_node.hpp" #include @@ -182,9 +183,21 @@ LinkingResult RuntimeLinker::link(const std::vector & manifest_apps, const result.linked_apps.push_back(linked_app); } - // Find orphan nodes (runtime apps not matching any manifest app) + // Find orphan nodes (runtime apps not matching any manifest app). + // + // The gateway's own in-process helper nodes are skipped. They are not + // entities and the app filter removes them, so reporting them here would + // tell the operator to declare, in the manifest, nodes that can never + // become apps - and the 'error' policy words that report as an instruction + // ("Declare them in the manifest"). An empty self FQN, which is what a + // linker constructed without a node has, matches nothing and leaves this + // behaviour exactly as it was. + const std::string self_fqn = node_ != nullptr ? node_->get_fully_qualified_name() : std::string(); for (const auto & rt_app : runtime_apps) { if (rt_app.bound_fqn.has_value() && matched_nodes.find(rt_app.bound_fqn.value()) == matched_nodes.end()) { + if (is_own_gateway_helper_node(rt_app.bound_fqn.value(), self_fqn)) { + continue; + } result.orphan_nodes.push_back(rt_app.bound_fqn.value()); } } diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py new file mode 100644 index 000000000..e56f754ae --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The gateway does not ask the operator to declare its own plumbing. + +In hybrid mode with ``unmanifested_nodes: error``, every running node that no +manifest app binds is reported on ``GET /health`` as an ``unmanifested_nodes`` +warning, and the message is an instruction: "Declare them in the manifest". + +The gateway's three in-process helper nodes must not be in that list. They can +never become apps - the app filter removes them - so declaring them would +silence the warning and produce manifest entities the gateway then deletes. The +two halves have to agree: what is never an App is never something to declare. + +Launched without a ``__node`` remap for the same reason as +``test_own_node_apps.test.py``: ``launch_ros``' ``name=`` renames every node in +the process, so under the suite's usual launch the helper names do not exist +and nothing here could be observed. +""" + +import os +import tempfile +import time +import unittest + +import launch +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NODE = 'ros2_medkit_gateway' +HELPER_FQNS = ( + f'/{GATEWAY_NODE}_sub', + f'/{GATEWAY_NODE}_fault_clients', + f'/{GATEWAY_NODE}_lifecycle_state_reader', +) +# An undeclared node of the test's own, so the warning below is known to be +# listing things rather than empty for an unrelated reason. +WITNESS_NODE = 'own_node_undeclared_witness' + +WARN_UNMANIFESTED_NODES = 'unmanifested_nodes' + +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +WARNING_BUDGET = 25.0 + +_MANIFEST_DIR = tempfile.mkdtemp(prefix='medkit-own-node-undeclared-') +_MANIFEST_PATH = os.path.join(_MANIFEST_DIR, 'manifest.yaml') + +# One app bound to a node this launch never starts: the manifest is valid and +# loaded, nothing links, so every running node is undeclared and the policy has +# something to report. +with open(_MANIFEST_PATH, 'w') as _manifest: + _manifest.write("""\ +manifest_version: "1.0" +metadata: + name: "Own node undeclared test vehicle" + version: "1.0.0" +config: + unmanifested_nodes: "error" +areas: + - id: test_area + name: "Test Area" +components: + - id: test_ecu + name: "Test ECU" + area: test_area +apps: + - id: absent_app + name: "An app whose node is not running" + is_located_on: test_ecu + ros_binding: + node_name: absent_node + namespace: /nowhere +""") + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one hybrid gateway keeping its process's real node names.""" + gateway_node = create_gateway_node( + port=GATEWAY_PORT, + name=None, + extra_params={ + 'server.host': '127.0.0.1', + 'refresh_interval_ms': 1000, + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': _MANIFEST_PATH, + }, + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +class TestOwnNodeUndeclared(unittest.TestCase): + """The undeclared-node warning and the app filter agree on the helpers.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway, then settle the warning against the helpers.""" + cls.session = requests.Session() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node('own_node_undeclared_probe', context=cls.context) + cls.witness = None + cls.graph_fqns = set() + cls.warning = None + + cls._wait_for_health() + cls.graph_fqns = cls._graph_fqns_until(set(HELPER_FQNS), HELPERS_ON_GRAPH_BUDGET) + if not set(HELPER_FQNS) <= cls.graph_fqns: + return + + # Created after the helpers, and undeclared like them. Waiting for the + # warning to name it dates the report: a report that has seen the + # witness has seen the helpers, which appeared earlier. + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.warning = cls._warning_until_lists(f'/{WITNESS_NODE}', WARNING_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + return + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') + + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Node FQNs on the graph, polled until *awaited* is a subset.""" + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + + @classmethod + def _warning_until_lists(cls, node_fqn, budget): + """Poll /health until the unmanifested_nodes warning names *node_fqn*.""" + deadline = time.monotonic() + budget * get_time_scale() + warning = None + while True: + body = cls.session.get(f'{BASE_URL}/health', timeout=10).json() + for candidate in body.get('warnings', []): + if candidate.get('code') == WARN_UNMANIFESTED_NODES: + warning = candidate + break + if warning is not None and node_fqn in warning.get('ros_node_fqns', []): + return warning + if time.monotonic() >= deadline: + return warning + time.sleep(0.5) + + def test_the_undeclared_warning_is_reporting(self): + """The policy is active and the warning lists the nodes it found. + + Without this the absence check below would pass on a gateway that + reported nothing at all. + """ + self.assertTrue( + set(HELPER_FQNS) <= self.graph_fqns, + f'helper nodes missing from the graph: ' + f'{sorted(set(HELPER_FQNS) - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + + self.assertIsNotNone( + self.warning, + f'no "{WARN_UNMANIFESTED_NODES}" warning on /health, so this file ' + f'cannot tell a filtered helper from an unreported one') + self.assertIn( + f'/{WITNESS_NODE}', self.warning.get('ros_node_fqns', []), + f'the warning never named "{WITNESS_NODE}", so no report is known ' + f'to post-date the helper nodes. Listed: ' + f'{sorted(self.warning.get("ros_node_fqns", []))}') + + def test_the_helper_nodes_are_not_reported_as_undeclared(self): + """The gateway's own helper nodes are not something to declare.""" + self.assertIsNotNone(self.warning, 'no unmanifested_nodes warning to read') + reported = set(self.warning.get('ros_node_fqns', [])) + self.assertFalse( + reported & set(HELPER_FQNS), + f'the gateway asked the operator to declare its own in-process ' + f'helper nodes: {sorted(reported & set(HELPER_FQNS))}. Declaring ' + f'them produces entities the app filter then removes. Reported: ' + f'{sorted(reported)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') From 89aa909e2741c88f48c7769fbb9b27a071e96452 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 21:17:23 +0200 Subject: [PATCH 12/25] test(gateway): date the app-list snapshot against the helper nodes The gateway creates its helper nodes at three points of start-up. Two of them come after the first refresh and after /health starts to answer. A settle gate that waits for the gateway's own node can therefore read a list built from a graph with no helper at all. The absence assertions then pass whatever the filter does. The settle sequence now proves its own order. First all three helpers are on the graph. Then the test creates a witness node. Then it reads an /apps snapshot that lists the witness, so that snapshot came from a graph that held the helpers too. The three stage budgets add up to well inside the file's ctest timeout, so a failure reports its assertion before ctest kills the test. The config reference documents the rule. It lists the filtered nodes and says why the gateway's own node stays an App. It explains the namespaces of the helper nodes and what the switch exposes again when it is off. It says that the helpers stay out of the unmanifested_nodes report, and that a declared App bound to one is dropped with a warning. --- docs/config/discovery-options.rst | 46 ++++- .../test/features/test_own_node_apps.test.py | 158 ++++++++++++------ 2 files changed, 149 insertions(+), 55 deletions(-) diff --git a/docs/config/discovery-options.rst b/docs/config/discovery-options.rst index c47613dc0..5f13049be 100644 --- a/docs/config/discovery-options.rst +++ b/docs/config/discovery-options.rst @@ -85,8 +85,52 @@ filters out ROS 2 internal infrastructure nodes such as ``_ros2cli_*``, SOVD entities. The filter applies to both locally discovered Apps and peer-discovered Apps (after stripping the peer prefix). +The same switch also excludes the helper nodes the gateway runs inside its own +process. Three of them exist in every deployment, named after the gateway node: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Node + - What it is + * - ``_sub`` + - The subscription executor that serves ``/data`` reads and cyclic + subscriptions. + * - ``_fault_clients`` + - The service clients that talk to the fault manager. + * - ``_lifecycle_state_reader`` + - The client that reads managed nodes' lifecycle state. + +None of these names begins with an underscore, so the convention above does not +cover them, and without this rule the gateway would list its own plumbing as +diagnosable Apps. They carry no parameters, services or actions of their own, so +there is nothing on them to diagnose. They are also left out of the +``unmanifested_nodes`` report on ``GET /health``: a node that can never be an +App is not a node to declare in a manifest. + +**The gateway's own node stays an App.** Its ROS parameters are what the +gateway serves as that App's configurations, so +``/apps//configurations`` is where a client reads and writes them - +there is no other entity carrying, for instance, +``aggregation.peer_auth_header``. Two gateways watching one graph also have to +agree about what is on it, which they cannot do if each hides a different node. + +Namespaces: the subscription node is created with the gateway's own namespace, +while the other two are created from the gateway's node name alone and take the +process default namespace. A process-wide remap (``-r __ns:=/line_a``) moves all +four together; a remap naming the gateway alone +(``-r ros2_medkit_gateway:__ns:=/line_a``) moves the gateway and the +subscription node and leaves the other two in the default namespace. Both +spellings are recognised, so the rule holds either way. + +A manifest- or plugin-declared App bound to one of these nodes is not served, +and the gateway logs a warning naming it: bind the App to the node you meant, +or drop the declaration. + Set to ``false`` if you need to expose all ROS 2 nodes regardless of naming -convention. +convention. That re-exposes the three helper nodes as well as the underscore +ones. Function Entities from Namespaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py index 5cb0cb459..3739d4c0c 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py @@ -35,6 +35,16 @@ ``-r __node:=`` to the whole process, which renames all four nodes to the same string - so under the suite's usual launch the helper names do not exist and nothing here could be observed. + +An absence assertion is only worth reading if the thing could have been there. +The gateway's helper nodes are created at three different points of start-up, +two of them after the first ``refresh_cache()`` and after ``/health`` starts +answering, so a naive "wait until /apps is non-empty" can read a list built +from a graph that did not yet contain them - and then the absences below hold +whatever the filter does. The settle sequence therefore proves the order it +needs: all three helpers on the graph FIRST, then a witness node created after +them, then an ``/apps`` snapshot that contains the witness. Such a snapshot was +built from a graph that held the helpers too. """ import time @@ -66,6 +76,15 @@ f'{GATEWAY_NODE}_fault_clients', f'{GATEWAY_NODE}_lifecycle_state_reader', ) +WITNESS_NODE = 'own_node_apps_refresh_witness' + +# The three stages of the settle sequence. They run one after another, so their +# sum plus launch and teardown has to stay inside this file's ctest TIMEOUT +# (the feature default, 120 s): a run killed by ctest reports a timeout and no +# test name, which hides whichever assertion actually failed. +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +REFRESH_WITNESS_BUDGET = 25.0 @pytest.mark.launch_test @@ -83,67 +102,91 @@ def generate_test_description(): ]), {'gateway_node': gateway_node} -def _graph_node_fqns(awaited, timeout=30.0): - """Fully qualified node names on the graph, waiting for *awaited*. - - Runs on its own rclpy context so it cannot disturb anything else in the - process, and polls: a graph query reads the discovery database directly and - needs no executor. - """ - context = Context() - rclpy.init(context=context) - probe = Node('own_node_apps_graph_probe', context=context) - try: - deadline = time.monotonic() + timeout * get_time_scale() - while True: - fqns = { - (namespace.rstrip('/') + '/' + name) - for name, namespace in probe.get_node_names_and_namespaces() - } - if awaited <= fqns or time.monotonic() >= deadline: - return fqns - time.sleep(0.2) - finally: - probe.destroy_node() - rclpy.shutdown(context=context) - - class TestOwnNodeApps(unittest.TestCase): """The gateway is a diagnosable App; its in-process helpers are not.""" @classmethod def setUpClass(cls): - """Wait for the gateway to answer, then read its app list once.""" + """Wait for the gateway, then settle /apps against the helper nodes.""" cls.session = requests.Session() - deadline = time.monotonic() + 60.0 * get_time_scale() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node('own_node_apps_graph_probe', context=cls.context) + cls.witness = None + cls.apps = set() + cls.graph_fqns = set() + cls.witness_seen = False + + cls._wait_for_health() + # Stage 1: every helper the absences below are about must be on the + # graph before anything reads /apps. + cls.graph_fqns = cls._graph_fqns_until( + {f'/{helper}' for helper in HELPER_NODES}, HELPERS_ON_GRAPH_BUDGET) + if not {f'/{helper}' for helper in HELPER_NODES} <= cls.graph_fqns: + return + + # Stage 2: a node created strictly after the helpers appeared. Its own + # arrival in /apps dates the snapshot: the gateway cannot have seen the + # witness without having seen the helpers. + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.witness_seen, cls.apps = cls._apps_until_contains( + WITNESS_NODE, REFRESH_WITNESS_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() last = None while time.monotonic() < deadline: try: response = cls.session.get(f'{BASE_URL}/health', timeout=5) if response.status_code == 200: - break + return last = response.status_code except requests.RequestException as exc: last = str(exc) time.sleep(0.5) - else: - raise AssertionError(f'gateway not ready within 60s (last: {last})') + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') - # The app list is served from the discovery cache, which the first - # refresh fills; poll until the gateway's own node is in it or the - # budget is out, so the absence assertions below read a settled list. - cls.apps = set() - app_deadline = time.monotonic() + 30.0 * get_time_scale() - while time.monotonic() < app_deadline: - body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() - cls.apps = {item['id'] for item in body.get('items', [])} - if GATEWAY_NODE in cls.apps: - break - time.sleep(0.5) + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Node FQNs on the graph, polled until *awaited* is a subset. + + The graph query reads the discovery database directly, so this polls + rather than spinning an executor. Returns the last set seen even on + timeout - the caller asserts on it, so a timeout cannot pass silently. + """ + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) @classmethod - def tearDownClass(cls): - cls.session.close() + def _apps_until_contains(cls, app_id, budget): + """Poll /apps until *app_id* is listed. Returns (seen, last snapshot).""" + deadline = time.monotonic() + budget * get_time_scale() + apps = set() + while True: + body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() + apps = {item['id'] for item in body.get('items', [])} + if app_id in apps: + return True, apps + if time.monotonic() >= deadline: + return False, apps + time.sleep(0.5) def test_the_gateways_own_node_is_an_app(self): """The gateway's own ROS node is listed, addressable and configurable. @@ -172,23 +215,30 @@ def test_the_in_process_helper_nodes_are_not_apps(self): @verifies REQ_INTEROP_003 """ + # Anti-vacuity, first half: each helper really is a node on this graph, + # so the absences below are about the filter and not about names that + # were never there. + awaited = {f'/{helper}' for helper in HELPER_NODES} + self.assertTrue( + awaited <= self.graph_fqns, + f'helper nodes missing from the graph: {sorted(awaited - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + + # Anti-vacuity, second half: the snapshot was rebuilt after they + # appeared. Without this the assertions below can be read from a list + # the gateway built before it could have listed a helper at all. + self.assertTrue( + self.witness_seen, + f'/apps never listed "{WITNESS_NODE}", so no snapshot is known to ' + f'post-date the helper nodes and these absences prove nothing. ' + f'Listed: {sorted(self.apps)}') + for helper in HELPER_NODES: self.assertNotIn( helper, self.apps, f'in-process helper "{helper}" must not be a diagnosable App. ' f'Listed: {sorted(self.apps)}') - # Anti-vacuity: each helper really is a node on this graph, so the - # assertions above are about the filter and not about names that were - # never there. Without this the test passes on a gateway that creates - # no helpers at all. - awaited = {f'/{helper}' for helper in HELPER_NODES} - graph_fqns = _graph_node_fqns(awaited) - self.assertTrue( - awaited <= graph_fqns, - f'helper nodes missing from the graph: {sorted(awaited - graph_fqns)}. ' - f'Nodes seen: {sorted(graph_fqns)}') - @launch_testing.post_shutdown_test() class TestShutdown(unittest.TestCase): From f5213afaa6a0e63fd5e33c686ea639bf790bff51 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 22:29:51 +0200 Subject: [PATCH 13/25] fix(gateway): report a helper-bound declared app when the set changes An app declared against one of the gateway's helper nodes is a static misconfiguration. refresh_cache() runs on every graph event and on the backstop cadence, which the integration fixtures set to one second, so the warning repeated for the life of the process. It now fires only when the set of offending apps changes, through remember_dropped_declared_apps. The entity-cache warning in the same function fires once per run. This one also fires again when the set changes. The predicate's doc comment and the config reference describe what the root-namespace spelling costs. A helper-named node at the root counts as plumbing whichever gateway created it. Two gateways that keep the default node name and differ only in namespace build the same fully qualified name for it. The comment in the /health handler states that the orphan list excludes the helper nodes. Each app-filter declaration sits under its own doc block, so doxygen attaches every block to the right function. --- docs/config/discovery-options.rst | 14 +++++ .../ros2_medkit_gateway/gateway_node.hpp | 31 ++++++++++ src/ros2_medkit_gateway/src/gateway_node.cpp | 31 ++++++++-- .../src/http/handlers/health_handlers.cpp | 17 +++--- .../test/test_handler_context.cpp | 56 +++++++++++++++++++ 5 files changed, 136 insertions(+), 13 deletions(-) diff --git a/docs/config/discovery-options.rst b/docs/config/discovery-options.rst index 5f13049be..3205376bc 100644 --- a/docs/config/discovery-options.rst +++ b/docs/config/discovery-options.rst @@ -124,6 +124,20 @@ four together; a remap naming the gateway alone subscription node and leaves the other two in the default namespace. Both spellings are recognised, so the rule holds either way. +.. note:: + + A node named ``_fault_clients`` or + ``_lifecycle_state_reader`` in the **root** namespace is treated as + plumbing whichever gateway created it. Two gateways that keep the default + node name and differ only in namespace produce the same fully qualified name + for those two nodes, so the name cannot say whose they are, and each gateway + will filter the other's. They carry nothing to diagnose in either process, + and the alternative is that a namespaced gateway lists and counts its own + plumbing. Give each gateway its own node name (``-r __node:=gateway_a``, as + :doc:`../tutorials/multi-instance` does) when several run on one ROS graph + and you want each one's nodes distinguishable. The subscription node is not + affected: it always follows its gateway's namespace. + A manifest- or plugin-declared App bound to one of these nodes is not served, and the gateway logs a warning naming it: bind the App to the node you meant, or drop the declaration. diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index e636d9381..b935605ed 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -440,6 +441,9 @@ class GatewayNode : public rclcpp::Node { // One-shot WARN when entity_cache capacity is exceeded (grew on first refresh after reserve). // Cleared only at construction time; never reset so the WARN fires at most once per run. bool warned_cache_grow_{false}; + /// Declared apps the helper-node rule last warned about, so a static + /// misconfiguration is reported when it appears or changes, not every refresh. + std::set warned_helper_bound_apps_; // Graph-change-driven discovery refresh. // @@ -528,6 +532,18 @@ class GatewayNode : public rclcpp::Node { * the gateway and the subscription node and leaves the other two behind, which * is why those two are also matched as `/`. * + * CONTRACT, and the cost of that second spelling: a helper-named node in the + * root namespace is treated as plumbing whichever gateway created it. Two + * gateways that keep the default node name and differ only in namespace build + * the same literal `/_fault_clients` and + * `/_lifecycle_state_reader`, so the name cannot say whose it is, and + * each will claim the other's. What it is does not depend on who owns it - + * those nodes carry nothing to diagnose in either process - and the + * alternative is that every namespaced gateway serves and counts its own + * plumbing. The subscription node is exempt: it always follows its gateway's + * namespace, so a root-namespace one is provably another process's and stays + * visible. + * * @param node_fqn Fully qualified node name to test ("/ns/node") * @param self_fqn The gateway node's own FQN. An empty value matches nothing */ @@ -562,4 +578,19 @@ size_t filter_internal_node_apps(std::vector & apps, const std::string & self_fqn, std::vector * dropped_declared_apps = nullptr); +/** + * @brief Remember which declared apps were dropped, and say whether that changed + * + * The condition this gates is a static misconfiguration, while the caller runs + * on every graph event and on the refresh cadence, so warning per call would + * repeat the same line for the life of the process. Returns true only when the + * set differs from the remembered one and is not empty; the remembered set is + * updated either way, so a condition that clears and returns is reported again. + * + * @param dropped App ids (with their bound FQNs) dropped by the helper rule + * @param remembered In/out: the set the caller last warned about + * @return true when the caller should warn + */ +bool remember_dropped_declared_apps(const std::vector & dropped, std::set & remembered); + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index c4fd9ea55..8da9a4aa9 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -2512,12 +2512,20 @@ void GatewayNode::refresh_cache() { if (removed > 0) { RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix or own helper node)", removed); } - for (const auto & dropped : dropped_declared_apps) { - RCLCPP_WARN(get_logger(), - "Declared app '%s' is bound to one of this gateway's own in-process helper nodes and is not " - "served. Those nodes carry no parameters, services or actions to diagnose. Bind the app to the " - "node you meant, or drop the declaration.", - dropped.c_str()); + // Warn on the set, not on the refresh. The condition is a static + // misconfiguration - an app declared against a helper node stays declared + // - while this function runs on every graph event and again on the + // backstop cadence, which the integration fixtures set to one second. The + // same one-shot discipline as the entity-cache WARN below, widened to + // re-fire when the set of offending apps actually changes. + if (remember_dropped_declared_apps(dropped_declared_apps, warned_helper_bound_apps_)) { + for (const auto & dropped : dropped_declared_apps) { + RCLCPP_WARN(get_logger(), + "Declared app '%s' is bound to one of this gateway's own in-process helper nodes and is not " + "served. Those nodes carry no parameters, services or actions to diagnose. Bind the app to the " + "node you meant, or drop the declaration.", + dropped.c_str()); + } } } @@ -2613,6 +2621,17 @@ void GatewayNode::stop_rest_server() { } } +bool remember_dropped_declared_apps(const std::vector & dropped, std::set & remembered) { + std::set current(dropped.begin(), dropped.end()); + if (current == remembered) { + return false; + } + remembered = std::move(current); + // An empty set is remembered so the condition clearing and coming back warns + // again, but there is nothing to say about no apps at all. + return !remembered.empty(); +} + size_t filter_internal_node_apps(std::vector & apps, const std::unordered_map & peer_routing_table, const std::string & self_fqn, std::vector * dropped_declared_apps) { diff --git a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp index fcc0948d0..84c640e11 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp @@ -104,13 +104,16 @@ http::Result HealthHandlers::get_health(const http::TypedRequest & "policy to 'warn' or 'ignore'."; // The subjects of this warning are ROS nodes, so they go in // ros_node_fqns and entity_ids stays empty. Not a formality: the - // orphan list is taken from the UNFILTERED runtime app list - // (runtime_layer.cpp), before gap-fill, the namespace filters and - // the policy filter, so some of these nodes have no SOVD entity at - // all - and for those that do, the App id is derived from the bare - // node name and only becomes namespace-qualified when some other - // node collides with it (ros2_runtime_introspection.cpp), so it can - // change between two polls while the FQN cannot. + // orphan list is taken from the runtime app list (runtime_layer.cpp) + // before gap-fill, the namespace filters and the policy filter, so + // some of these nodes have no SOVD entity at all - and for those that + // do, the App id is derived from the bare node name and only becomes + // namespace-qualified when some other node collides with it + // (ros2_runtime_introspection.cpp), so it can change between two + // polls while the FQN cannot. The one thing already taken out of it + // is the gateway's own in-process helper nodes (runtime_linker.cpp): + // they can never become apps, so naming them here would be an + // instruction to declare something the app filter then removes. warning.ros_node_fqns = linking->orphan_nodes; warnings.push_back(std::move(warning)); } diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 7d1fc1aae..95da3bb54 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -1142,6 +1142,62 @@ TEST(FilterInternalNodeAppsTest, ReportsOnlyDeclaredAppsItDropsAsHelpers) { EXPECT_EQ(dropped_declared[0], "plc_bridge -> /ros2_medkit_gateway_fault_clients"); } +TEST(IsOwnGatewayHelperNodeTest, RecognizesAForeignRootHelperAsPlumbing) { + // Two gateways, both left at the default node name, one moved into a + // namespace: the namespaced one's root-dwelling helpers and the root one's + // are the SAME literal FQN, because both are built from the same node name in + // the same namespace. Nothing in the name can tell them apart, so a + // helper-named node in the root namespace is treated as plumbing whichever + // process created it. That is the deliberate trade: the alternative is + // serving and counting one's own helper nodes in every namespaced + // deployment, and a node named "_fault_clients" is plumbing in + // either case - what it is does not depend on who owns it. + const std::string self_fqn = "/subsystem_a/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // The subscription node is the one case the name DOES settle: it always + // follows its gateway's namespace, so a root-namespace one is another + // process's and stays visible. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_sub", self_fqn)); + + // A foreign gateway's own node is never plumbing, in any namespace. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_c/ros2_medkit_gateway", self_fqn)); + + // A differently NAMED gateway's helpers are not ours: the bare name has to + // match, so the rule does not reach across to a peer that was renamed. + EXPECT_FALSE(is_own_gateway_helper_node("/other_gateway_fault_clients", self_fqn)); +} + +TEST(RememberDroppedDeclaredAppsTest, ReportsTheSetOnceUntilItChanges) { + // The condition is a static misconfiguration and the caller runs on every + // refresh, so the same set must be reported once, not once per second. + std::set remembered; + + const std::vector one{"plc_bridge -> /ros2_medkit_gateway_fault_clients"}; + EXPECT_TRUE(remember_dropped_declared_apps(one, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(one, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(one, remembered)); + + // A changed set is news again. + const std::vector two{"plc_bridge -> /ros2_medkit_gateway_fault_clients", + "aux_bridge -> /ros2_medkit_gateway_sub"}; + EXPECT_TRUE(remember_dropped_declared_apps(two, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(two, remembered)); + + // Order is not a change: the caller builds this list from a vector whose + // order follows discovery, which is not stable between refreshes. + const std::vector two_reordered{"aux_bridge -> /ros2_medkit_gateway_sub", + "plc_bridge -> /ros2_medkit_gateway_fault_clients"}; + EXPECT_FALSE(remember_dropped_declared_apps(two_reordered, remembered)); + + // Clearing says nothing, but it is remembered, so the same set coming back + // is reported again rather than staying silent for the life of the process. + EXPECT_FALSE(remember_dropped_declared_apps({}, remembered)); + EXPECT_TRUE(remember_dropped_declared_apps(one, remembered)); +} + TEST(FilterInternalNodeAppsTest, DropsTheHelpersOfANamespacedGateway) { // The same split at the level callers see: two of the three helpers of a // namespaced gateway live in the root namespace, and without them being From e12f15caad2636ad17daf36de636b6871d52da7c Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Sat, 12 Sep 2026 22:30:33 +0200 Subject: [PATCH 14/25] test(gateway): drive the split-namespace helper case against a real gateway The three helper node names take two shapes. A remap that names only the gateway moves the gateway and the subscription node into a namespace. The fault-client and lifecycle-reader nodes stay at the root. The predicate builds that second shape from the gateway's name. A new feature test launches exactly that invocation. It asserts that the split is on the graph. It pins the app list to the gateway plus the test's own two nodes. --- .../test_own_node_apps_namespaced.test.py | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py new file mode 100644 index 000000000..afbb86139 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The helper-node filter holds when the gateway alone is moved to a namespace. + +``test_own_node_apps.test.py`` covers the case where all four of the gateway's +nodes share one namespace, where the three helper names are simply the +gateway's plus a suffix. That is not the only shape they take. The subscription +node is created with the gateway's namespace, but the fault-client and +lifecycle-reader nodes are created from the gateway's node NAME alone, so they +take the process default namespace - and a remap naming the gateway alone, +``-r ros2_medkit_gateway:__ns:=/subsystem_b``, moves the gateway and the +subscription node while leaving those two in the root namespace. + +The gateway must still recognise all three. This launch is the split +invocation, with one gateway on the graph so nothing else can supply a node by +those names. +""" + +import time +import unittest + +import launch +import launch_ros.actions +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.coverage import get_coverage_env + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NAME = 'ros2_medkit_gateway' +GATEWAY_NS = '/subsystem_b' +GATEWAY_FQN = f'{GATEWAY_NS}/{GATEWAY_NAME}' +# The split: one helper follows the gateway, two stay at the root. +NAMESPACED_HELPER_FQN = f'{GATEWAY_NS}/{GATEWAY_NAME}_sub' +ROOT_HELPER_FQNS = ( + f'/{GATEWAY_NAME}_fault_clients', + f'/{GATEWAY_NAME}_lifecycle_state_reader', +) +ALL_HELPER_FQNS = (NAMESPACED_HELPER_FQN,) + ROOT_HELPER_FQNS +WITNESS_NODE = 'own_node_apps_ns_witness' +PROBE_NODE = 'own_node_apps_ns_probe' +# This file's own two nodes are ordinary ROS nodes, so the gateway lists them +# as apps like any other. Naming them keeps the assertion an exact set rather +# than a membership test that would not notice a helper slipping back in. +TEST_OWN_NODES = {WITNESS_NODE, PROBE_NODE} + +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +REFRESH_WITNESS_BUDGET = 25.0 + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one gateway with only its own node moved into a namespace. + + Built here rather than through ``create_gateway_node`` because the remap has + to name the gateway node: passing a namespace to the launch action instead + would move all four nodes together, which is the case the sibling file + already covers. + """ + gateway_node = launch_ros.actions.Node( + package='ros2_medkit_gateway', + executable='gateway_node', + name=None, + output='screen', + parameters=[{ + 'server.host': '127.0.0.1', + 'server.port': GATEWAY_PORT, + 'refresh_interval_ms': 1000, + }], + ros_arguments=['-r', f'{GATEWAY_NAME}:__ns:={GATEWAY_NS}'], + additional_env=dict(get_coverage_env()), + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +class TestOwnNodeAppsNamespaced(unittest.TestCase): + """A gateway moved on its own still filters all three of its helpers.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway, then settle /apps against the helper nodes.""" + cls.session = requests.Session() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node(PROBE_NODE, context=cls.context) + cls.witness = None + cls.apps = set() + cls.graph_fqns = set() + cls.witness_seen = False + + cls._wait_for_health() + cls.graph_fqns = cls._graph_fqns_until(set(ALL_HELPER_FQNS), HELPERS_ON_GRAPH_BUDGET) + if not set(ALL_HELPER_FQNS) <= cls.graph_fqns: + return + + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.witness_seen, cls.apps = cls._apps_until_contains( + WITNESS_NODE, REFRESH_WITNESS_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + return + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') + + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Poll the graph until *awaited* is a subset of the node FQNs.""" + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + + @classmethod + def _apps_until_contains(cls, app_id, budget): + """Poll /apps until *app_id* is listed. Returns (seen, last snapshot).""" + deadline = time.monotonic() + budget * get_time_scale() + apps = set() + while True: + body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() + apps = {item['id'] for item in body.get('items', [])} + if app_id in apps: + return True, apps + if time.monotonic() >= deadline: + return False, apps + time.sleep(0.5) + + def test_the_split_helper_nodes_are_all_on_the_graph(self): + """The remap really does split them, so the absences below mean something. + + Without this the next test would pass on a launch where the namespace + remap silently did nothing and the helpers never existed under these + names. + """ + self.assertTrue( + set(ALL_HELPER_FQNS) <= self.graph_fqns, + f'expected the gateway and its subscription node in {GATEWAY_NS} and the ' + f'other two helpers at the root, missing: ' + f'{sorted(set(ALL_HELPER_FQNS) - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + self.assertIn( + GATEWAY_FQN, self.graph_fqns, + f'the gateway node is not in {GATEWAY_NS}, so the remap did not apply') + + def test_apps_lists_the_gateway_and_none_of_its_helpers(self): + """A namespaced gateway serves itself and none of its plumbing. + + @verifies REQ_INTEROP_003 + """ + self.assertTrue( + self.witness_seen, + f'/apps never listed "{WITNESS_NODE}", so no snapshot is known to ' + f'post-date the helper nodes. Listed: {sorted(self.apps)}') + + self.assertEqual( + self.apps, {GATEWAY_NAME} | TEST_OWN_NODES, + f'a gateway alone on the graph must serve its own node and nothing ' + f"of its plumbing, beside this test's own two nodes. " + f'Listed: {sorted(self.apps)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') From 02d72a4e487900bee2649592ab8220ea224ce813 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 15 Sep 2026 14:51:55 +0200 Subject: [PATCH 15/25] gateway: skip the helper nodes in the orphan report only while the app filter is on RuntimeLinker takes discovery.runtime.filter_internal_nodes. While it is true, the gateway's own helper nodes stay out of orphan_nodes, which feeds linking.orphan_count and the unmanifested_nodes warning on /health. While it is false, the helpers are served as Apps and counted like any other undeclared node. discovery-options.rst and the comment in the health handler describe this coupling. A unit test links the same runtime apps with the setting on and off and checks a helper node against a real unmanifested node. Comments in fault_handlers.cpp and in six gateway and integration-test files are reworded to state what the code does. --- docs/config/discovery-options.rst | 18 +++-- .../discovery/manifest/runtime_linker.hpp | 12 +++- .../src/discovery/discovery_manager.cpp | 3 +- .../src/discovery/manifest/runtime_linker.cpp | 21 +++--- .../src/http/handlers/fault_handlers.cpp | 5 +- .../src/http/handlers/health_handlers.cpp | 11 +-- .../test/test_entity_freeze_frame_capture.cpp | 10 +-- .../test/test_handler_context.cpp | 6 +- .../test/test_operation_handlers.cpp | 8 +-- .../test/test_runtime_linker.cpp | 72 +++++++++++++++++++ .../test/features/test_own_node_apps.test.py | 6 +- .../test_own_node_apps_namespaced.test.py | 7 +- .../features/test_own_node_undeclared.test.py | 4 +- 13 files changed, 141 insertions(+), 42 deletions(-) diff --git a/docs/config/discovery-options.rst b/docs/config/discovery-options.rst index 3205376bc..4df39077e 100644 --- a/docs/config/discovery-options.rst +++ b/docs/config/discovery-options.rst @@ -105,9 +105,15 @@ process. Three of them exist in every deployment, named after the gateway node: None of these names begins with an underscore, so the convention above does not cover them, and without this rule the gateway would list its own plumbing as diagnosable Apps. They carry no parameters, services or actions of their own, so -there is nothing on them to diagnose. They are also left out of the -``unmanifested_nodes`` report on ``GET /health``: a node that can never be an -App is not a node to declare in a manifest. +there is nothing on them to diagnose. + +While this setting is ``true`` they are also left out of the linking report on +``GET /health`` - ``discovery.linking.orphan_count``, and the +``ros_node_fqns`` of the ``unmanifested_nodes`` warning raised under +``unmanifested_nodes: error``: a node that can never be an App is not a node to +declare in a manifest. That report exists only in ``hybrid`` mode, where +manifest apps are linked to runtime nodes; ``runtime_only`` has no linking block +at all. **The gateway's own node stays an App.** Its ROS parameters are what the gateway serves as that App's configurations, so @@ -144,7 +150,11 @@ or drop the declaration. Set to ``false`` if you need to expose all ROS 2 nodes regardless of naming convention. That re-exposes the three helper nodes as well as the underscore -ones. +ones, and in ``hybrid`` mode the linking report follows: a helper node the +gateway serves as an App is also an undeclared one, so it is counted in +``discovery.linking.orphan_count`` and, under ``unmanifested_nodes: error``, +named in the ``ros_node_fqns`` of the ``unmanifested_nodes`` warning. With +``true`` it appears in neither. Function Entities from Namespaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp index 56769b7a4..6a14fa4d9 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp @@ -95,8 +95,16 @@ class RuntimeLinker { /** * @brief Constructor * @param node ROS node for logging (can be nullptr for testing) + * @param filter_internal_nodes The effective + * `discovery.runtime.filter_internal_nodes` setting. It decides + * whether this gateway's own in-process helper nodes are left out of + * `orphan_nodes`, because it is the same setting that decides whether + * they are filtered out of the served apps. With it off the helpers + * ARE served as apps, so reporting them as unmanifested is the truth; + * omitting them there would hide from `/health` exactly the nodes the + * operator turned the filter off to see. */ - explicit RuntimeLinker(rclcpp::Node * node = nullptr); + explicit RuntimeLinker(rclcpp::Node * node = nullptr, bool filter_internal_nodes = true); /** * @brief Link manifest apps to runtime apps (nodes) @@ -190,6 +198,8 @@ class RuntimeLinker { void log_error(const std::string & msg) const; rclcpp::Node * node_; + /// See the constructor: the helper-node skip below follows the app filter. + bool filter_internal_nodes_{true}; LinkingResult last_result_; }; diff --git a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp index e8c1bf49f..03e4af988 100644 --- a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp +++ b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp @@ -213,7 +213,8 @@ void DiscoveryManager::build_pipeline() { // RuntimeLinker only makes sense when runtime is enabled. if (config_.runtime_enabled) { - pipeline->set_linker(std::make_unique(node_), get_manifest_config()); + pipeline->set_linker(std::make_unique(node_, config_.runtime.filter_internal_nodes), + get_manifest_config()); } pipeline_ = std::move(pipeline); diff --git a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp index 0923e77f7..d2ed63566 100644 --- a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp +++ b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp @@ -69,7 +69,8 @@ bool topic_path_matches(const std::string & topic, const std::string & topic_nam } // namespace -RuntimeLinker::RuntimeLinker(rclcpp::Node * node) : node_(node) { +RuntimeLinker::RuntimeLinker(rclcpp::Node * node, bool filter_internal_nodes) + : node_(node), filter_internal_nodes_(filter_internal_nodes) { } LinkingResult RuntimeLinker::link(const std::vector & manifest_apps, const std::vector & runtime_apps, @@ -185,17 +186,19 @@ LinkingResult RuntimeLinker::link(const std::vector & manifest_apps, const // Find orphan nodes (runtime apps not matching any manifest app). // - // The gateway's own in-process helper nodes are skipped. They are not - // entities and the app filter removes them, so reporting them here would - // tell the operator to declare, in the manifest, nodes that can never - // become apps - and the 'error' policy words that report as an instruction - // ("Declare them in the manifest"). An empty self FQN, which is what a - // linker constructed without a node has, matches nothing and leaves this - // behaviour exactly as it was. + // The gateway's own in-process helper nodes are skipped WHILE the app filter + // removes them - the two are one setting, discovery.runtime.filter_internal_nodes. + // With it on they are not entities, so reporting them here would tell the + // operator to declare, in the manifest, nodes that can never become apps, and + // the 'error' policy words that report as an instruction ("Declare them in + // the manifest"). With it off they ARE served as apps, and an unmanifested + // served app is precisely what orphan_nodes and /health's orphan_count are + // for. An empty self FQN, which is what a linker constructed without a node + // has, matches nothing either way. const std::string self_fqn = node_ != nullptr ? node_->get_fully_qualified_name() : std::string(); for (const auto & rt_app : runtime_apps) { if (rt_app.bound_fqn.has_value() && matched_nodes.find(rt_app.bound_fqn.value()) == matched_nodes.end()) { - if (is_own_gateway_helper_node(rt_app.bound_fqn.value(), self_fqn)) { + if (filter_internal_nodes_ && is_own_gateway_helper_node(rt_app.bound_fqn.value(), self_fqn)) { continue; } result.orphan_nodes.push_back(rt_app.bound_fqn.value()); diff --git a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp index c7282fd7b..0df7419b9 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp @@ -352,8 +352,9 @@ dto::FaultDetail FaultHandlers::build_sovd_fault_response(const json & fault_jso } // Entity-frame provenance (merge_entity_freeze_frames), only when known. // "source" names the capture path (a plugin DataProvider or the - // plugin's x-plc-data route). A consumer reads it instead of the - // empty topic/message_type an entity frame necessarily carries. + // plugin's x-plc-data route). It is where a consumer reads the + // provenance from: an entity frame necessarily carries an empty + // topic/message_type. if (s.contains("source") && s["source"].is_string()) { snap["x-medkit"]["source"] = s["source"]; } diff --git a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp index 84c640e11..cbe232f20 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp @@ -110,10 +110,13 @@ http::Result HealthHandlers::get_health(const http::TypedRequest & // do, the App id is derived from the bare node name and only becomes // namespace-qualified when some other node collides with it // (ros2_runtime_introspection.cpp), so it can change between two - // polls while the FQN cannot. The one thing already taken out of it - // is the gateway's own in-process helper nodes (runtime_linker.cpp): - // they can never become apps, so naming them here would be an - // instruction to declare something the app filter then removes. + // polls while the FQN cannot. One thing is taken out of it while + // discovery.runtime.filter_internal_nodes is on (runtime_linker.cpp): + // the gateway's own in-process helper nodes, which the app filter + // then removes, so naming them here would be an instruction to + // declare something that can never become an App. With the filter off + // they are served as Apps and appear here like any other undeclared + // node. warning.ros_node_fqns = linking->orphan_nodes; warnings.push_back(std::move(warning)); } diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index b97372530..da14202d2 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -578,9 +578,9 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedEntityWithLastKnownValuesIsCapt // DataProvider flavour of the same case asserts the other constant, which is // what stops the two from being swapped at their call sites unnoticed. EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceXPlcDataRoute); - // The wire value itself, not just the symbol: swapping what the two constants - // hold is an API break for every consumer of x-medkit.source, and comparing - // symbol against symbol would not see it. + // The wire value itself, because comparing symbol against symbol cannot see + // the two constants' contents being swapped, and that swap is an API break + // for every consumer of x-medkit.source. EXPECT_EQ(frames[0].source, "plugin_x_plc_data_route"); } @@ -823,8 +823,8 @@ TEST(MergeEntityFreezeFrames, OmitsSourceForAFrameThatNamesNoPath) { // A merge-helper contract, not a control for the capture tests: both capture // paths always name themselves (asserted from real captures in // Disconnected{Entity,DataProvider}WithLastKnownValuesIsCaptured), so this - // frame is one only a caller can build. The helper must then leave the key - // out rather than invent a provenance the wire consumer would trust. + // frame is one only a caller can build. The helper leaves the key out, so + // the wire carries no provenance a consumer would trust. json env_data = {{"snapshots", json::array()}}; EntityFreezeFrameCapture::Frame frame; frame.entity_id = "plc_app"; diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 95da3bb54..f1dbbcd38 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -1088,7 +1088,7 @@ TEST(IsOwnGatewayHelperNodeTest, MatchesTheThreeHelperSuffixesExactlyAndNothingE EXPECT_FALSE(is_own_gateway_helper_node("/other" + self_fqn + "_sub", self_fqn)); EXPECT_FALSE(is_own_gateway_helper_node("/fault_manager", self_fqn)); - // An unknown self FQN must claim nothing rather than everything. + // An unknown self FQN claims nothing. EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, "")); EXPECT_FALSE(is_own_gateway_helper_node("", self_fqn)); } @@ -1192,8 +1192,8 @@ TEST(RememberDroppedDeclaredAppsTest, ReportsTheSetOnceUntilItChanges) { "plc_bridge -> /ros2_medkit_gateway_fault_clients"}; EXPECT_FALSE(remember_dropped_declared_apps(two_reordered, remembered)); - // Clearing says nothing, but it is remembered, so the same set coming back - // is reported again rather than staying silent for the life of the process. + // Clearing says nothing, and it is remembered, so the same set coming back + // is reported again. EXPECT_FALSE(remember_dropped_declared_apps({}, remembered)); EXPECT_TRUE(remember_dropped_declared_apps(one, remembered)); } diff --git a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp index 1ed5aa2cd..f95e73995 100644 --- a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp @@ -524,10 +524,10 @@ class OperationHandlersFixtureTest : public ::testing::Test { return async_ptr->id; } - // Returns the optional rather than dereferencing it: a non-fatal expectation - // followed by an unconditional `*goal_info` turns a missing goal into - // undefined behaviour instead of a failure anyone can read. The caller - // ASSERTs, which is what stops the test. + // Returns the optional, so a missing goal reaches the caller as an empty + // value and the caller's ASSERT stops the test with a message. A non-fatal + // expectation followed by an unconditional `*goal_info` here would be + // undefined behaviour. std::optional get_tracked_goal_or_fail(const std::string & execution_id) { auto goal_info = gateway_node_->get_operation_manager()->get_tracked_goal(execution_id); EXPECT_TRUE(goal_info.has_value()); diff --git a/src/ros2_medkit_gateway/test/test_runtime_linker.cpp b/src/ros2_medkit_gateway/test/test_runtime_linker.cpp index 2fc04afdc..f8ceef122 100644 --- a/src/ros2_medkit_gateway/test/test_runtime_linker.cpp +++ b/src/ros2_medkit_gateway/test/test_runtime_linker.cpp @@ -14,6 +14,13 @@ #include +#include +#include +#include +#include + +#include + #include "ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp" using namespace ros2_medkit_gateway::discovery; @@ -804,6 +811,71 @@ TEST_F(RuntimeLinkerTest, MergedInput_PreservesOrphanRuntimeApps) { EXPECT_EQ(result.orphan_nodes[0], "/nav/planner"); } +// ============================================================================= +// The gateway's own helper nodes: skipped by the same setting that filters them +// out of the served apps +// ============================================================================= + +TEST(RuntimeLinkerHelperNodes, TheOrphanSkipFollowsTheAppFilterSetting) { + // discovery.runtime.filter_internal_nodes decides whether the gateway's own + // in-process helper nodes are served as apps. While it is on they are not + // entities, so listing them as unmanifested would tell the operator to declare + // nodes that can never become apps. Turn it off - which the config docs say + // re-exposes them - and they ARE served, so leaving them out of orphan_nodes + // hides from /health's unmanifested_nodes exactly the nodes the operator + // turned the filter off in order to see. + const bool owned_rclcpp = !rclcpp::ok(); + if (owned_rclcpp) { + rclcpp::init(0, nullptr); + } + auto node = std::make_shared("runtime_linker_helper_gateway"); + const std::string self_fqn = node->get_fully_qualified_name(); + // The subscription helper node the gateway creates in its own namespace. + const std::string helper_fqn = self_fqn + "_sub"; + + App helper; + helper.id = "helper"; + helper.name = "helper"; + helper.source = "heuristic"; + helper.is_online = true; + helper.bound_fqn = helper_fqn; + + App foreign; + foreign.id = "planner"; + foreign.name = "planner"; + foreign.source = "heuristic"; + foreign.is_online = true; + foreign.bound_fqn = "/nav/planner"; + + const std::vector runtime_apps{helper, foreign}; + ManifestConfig config; + + const auto contains = [](const std::vector & orphans, const std::string & fqn) { + return std::find(orphans.begin(), orphans.end(), fqn) != orphans.end(); + }; + + { + RuntimeLinker filtering(node.get(), /*filter_internal_nodes=*/true); + const auto result = filtering.link({}, runtime_apps, config); + EXPECT_FALSE(contains(result.orphan_nodes, helper_fqn)) + << "a helper node the app filter removes was reported as unmanifested"; + EXPECT_TRUE(contains(result.orphan_nodes, foreign.bound_fqn.value())) + << "the control: a real unmanifested node is still reported"; + } + { + RuntimeLinker serving(node.get(), /*filter_internal_nodes=*/false); + const auto result = serving.link({}, runtime_apps, config); + EXPECT_TRUE(contains(result.orphan_nodes, helper_fqn)) + << "with the app filter off the helper is served as an app, so it is unmanifested and must be counted"; + EXPECT_TRUE(contains(result.orphan_nodes, foreign.bound_fqn.value())); + } + + node.reset(); + if (owned_rclcpp) { + rclcpp::shutdown(); + } +} + int main(int argc, char ** argv) { testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py index 3739d4c0c..7ace8ac09 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py @@ -160,9 +160,9 @@ def _wait_for_health(cls): def _graph_fqns_until(cls, awaited, budget): """Node FQNs on the graph, polled until *awaited* is a subset. - The graph query reads the discovery database directly, so this polls - rather than spinning an executor. Returns the last set seen even on - timeout - the caller asserts on it, so a timeout cannot pass silently. + The graph query reads the discovery database directly, so polling is + what this needs. Returns the last set seen even on timeout - the caller + asserts on it, so a timeout cannot pass silently. """ deadline = time.monotonic() + budget * get_time_scale() while True: diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py index afbb86139..941dbe3ec 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py @@ -79,10 +79,9 @@ def generate_test_description(): """Launch one gateway with only its own node moved into a namespace. - Built here rather than through ``create_gateway_node`` because the remap has - to name the gateway node: passing a namespace to the launch action instead - would move all four nodes together, which is the case the sibling file - already covers. + Built here because the remap has to name the gateway node, which + ``create_gateway_node`` cannot express: a namespace on the launch action + moves all four nodes together, and that case is the sibling file's. """ gateway_node = launch_ros.actions.Node( package='ros2_medkit_gateway', diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py index e56f754ae..aafd08ac9 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py @@ -61,8 +61,8 @@ f'/{GATEWAY_NODE}_fault_clients', f'/{GATEWAY_NODE}_lifecycle_state_reader', ) -# An undeclared node of the test's own, so the warning below is known to be -# listing things rather than empty for an unrelated reason. +# An undeclared node of the test's own, so the warning below is known to have +# something to list. WITNESS_NODE = 'own_node_undeclared_witness' WARN_UNMANIFESTED_NODES = 'unmanifested_nodes' From 276c374f76526051fa7cd6059a4b16004f3b43d1 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 15 Sep 2026 14:51:57 +0200 Subject: [PATCH 16/25] opcua: decide the comms-lost clear per bridge, bind to one server, keep the PLC's order Comms-lost clear. The plugin decides the PLC_COMMS_LOST clear on the poll thread, and only while the link is up. It asks the fault manager who reported the standing row. It clears the row only when every source is an id that this bridge reports under: the current component id, the stand-ins it assigned and the NodeMap default. A row that two bridges hold stays for an operator. - One fault-store read is outstanding at a time. It is stamped with the probe and the connection generation. - A read that the store does not answer within fault_service_timeout_ms is dropped and taken again. - An answer from before a link drop is discarded, and the decision is owed again. - A link-state clear that the bounded buffer could not keep is owed. It is decided once the sink is ready. Order. The pending buffer merges a clear into an earlier pending clear for the same code only when nothing was reported for that code in between. So report, clear, report, clear flushes in that order. Identity in config-less mode: - The connect hook derives the component name from the device again before the session subscribes, so replayed conditions land on the new entity. - A nameplate replaces a stand-in. Nothing replaces a nameplate. - An empty read renames a stand-in only when the endpoint changed. - Reads are budgeted per session. - Conditions that the poller already pinned move to the new entity. - The event path reads a copy of the alarm routing that the poller owns. Binding. The plugin remembers the ApplicationUri of the server a session reached. A rescan while disconnected looks for that server at any address and refuses every other server. A different server at the bound address is dropped at connect. The first adoption, a configured endpoint_url and a server without an ApplicationUri have no binding constraint. The README says so. A rescan that throws is caught on the poll thread. The cancel predicate reads the node's own context. Tests. End-to-end tests against the fixture cover: - a row that two bridges hold, left standing - an owed clear, sent after the buffer drains - no decision while the link is down - an unanswered probe, dropped and taken again - a stale answer after a probe timeout and after an outage - a fault raised under a stand-in, healed once the device names itself - the binding across an address change and a swap - the connect hook, run before the event routing is copied Unit tests cover the clear gate, the probe conditions, the identity rules and read budget, the buffer order, bound selection, the guarded rescan, the context-aware cancel, and the poller's routing copy and repin. The fixture takes --app-uri, so two fixtures can stand for two PLCs. --- .../ros2_medkit_opcua/README.md | 116 +- .../ros2_medkit_opcua/device_identity.hpp | 7 + .../ros2_medkit_opcua/network_discovery.hpp | 25 +- .../include/ros2_medkit_opcua/node_map.hpp | 7 +- .../ros2_medkit_opcua/opcua_client.hpp | 7 + .../ros2_medkit_opcua/opcua_plugin.hpp | 390 ++++++- .../ros2_medkit_opcua/opcua_poller.hpp | 80 ++ .../ros2_medkit_opcua/src/device_identity.cpp | 12 +- .../src/network_discovery.cpp | 17 +- .../ros2_medkit_opcua/src/opcua_client.cpp | 21 + .../ros2_medkit_opcua/src/opcua_plugin.cpp | 668 +++++++++-- .../ros2_medkit_opcua/src/opcua_poller.cpp | 120 +- .../test_alarm_server/test_alarm_server.cpp | 15 +- .../test/test_network_discovery.cpp | 42 + .../test/test_opcua_identity.cpp | 1022 +++++++++++++++-- .../test/test_opcua_plugin.cpp | 408 ++++++- .../test/test_opcua_poller.cpp | 97 ++ 17 files changed, 2785 insertions(+), 269 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 8cb6920bc..030f080df 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -657,8 +657,9 @@ ros2_medkit_gateway: | `condition_replay_strategy` | `auto` | Active-condition replay on reconnect: `method`, `read`, `auto`, `off` (see below) | | `require_confirm_for_clear` | `true` | Require both Acknowledge AND Confirm before a native alarm auto-clears. Set `false` for Confirm-less servers (e.g. Siemens S7-1500) so alarms clear on Acknowledge alone (see below) | | `comms_lost_fault_enabled` | `true` | Raise a component-scoped `PLC_COMMS_LOST` fault when the connection stays down, and clear it on every successful connect (issue #496) | -| `comms_lost_debounce_ms` | `5000` | Continuous down time before `PLC_COMMS_LOST` is raised (debounces reconnect blips; clamped to [0, 3600000] ms) | +| `comms_lost_debounce_ms` | `5000` | Continuous down time before `PLC_COMMS_LOST` is raised (debounces reconnect blips; clamped to [0, 3600000] ms, with a warning) | | `comms_lost_severity` | `ERROR` | SOVD severity bucket for the `PLC_COMMS_LOST` fault | +| `fault_service_timeout_ms` | `5000` | How long a fault-store read may stay outstanding before the `PLC_COMMS_LOST` decision is owed again (out of range [100, 600000] ms is refused with a warning and the previous value is kept) | | `discovery.enabled` | `false` | Opt-in read-only PLC network discovery (auto endpoint). See below | ### OPC-UA client security (SecurityPolicy, certificates, user auth) @@ -741,18 +742,78 @@ How it works: EndpointUrl, which a server may report as a non-resolvable hostname. 5. While no session is established, the reconnect loop scans again every `interval_s` (default 30 s), measured from the END of the previous sweep, and - adopts a newly found server for its next connect attempt, logging the swap at - INFO. The re-scan is consulted once per reconnect attempt, and those are + adopts the server it is **bound** to at whatever address it now answers on, + logging the swap at INFO. The re-scan is consulted once per reconnect attempt, and those are spaced by an exponential backoff, so the backoff ceiling is capped at - `interval_s` while discovery is re-scanning - otherwise the real cadence - would be `max(interval_s, backoff)` rather than the stated one. This covers - the common race where the gateway and the PLC boot together: the startup scan - finds nothing because the PLC is still coming up, and without a re-scan the - plugin would retry the fallback endpoint until someone restarted it. + `interval_s` while discovery is re-scanning, which is what makes the stated + cadence the real one: uncapped it would be `max(interval_s, backoff)`. This + covers the common race where the gateway and the PLC boot together: the + startup scan finds nothing because the PLC is still coming up, and without a + re-scan the plugin would retry the fallback endpoint until someone restarted + it. 6. On the first session after such an adoption, a config-less deployment (no node map) re-derives the SOVD component identity from the device itself, so the component stops being served under the provisional `opcua-` name it - got when nothing answered. The change is logged at INFO. + got when nothing answered. The change is logged at INFO. A rename the connect + hook makes happens before the session subscribes, so the conditions the + server replays are hosted on the new entity; a rename a later poll makes + happens after, which is the residual described below. + + The rename is ruled by what the read produced. A device nameplate takes over + from an `opcua-` stand-in; a name the device gave us keeps its place. A + read without a nameplate moves one stand-in to another, which is the adoption + case where the endpoint changed, and leaves a device-given name alone: the + first device-info read of a fresh session can come back empty while the PLC's + address space is still filling in. + + The read budget is: **one read at connect** (in `set_context`, for the first + session), then per session **up to four in the connect hook** - the first + plus three more, 250 ms apart - and **up to five poll reads**. A first + session therefore costs at most 1 + 4 + 5 = 10 reads, and every later one at + most 9. After that the stand-in stands for the session, and the question is + asked again on the next one. A shutdown ends the connect burst without + waiting it out and without renaming: the residual is up to one 250 ms pause + plus one device-info read already in flight. Plugin start-up pays the burst + once, and only when the device answers the first read without a nameplate. + + The stand-ins a fault can have been reported under are the **first eight + distinct** ones a process assigns; a stand-in re-adopted later keeps its + place. + + When the rename happens, the alarm routing the event path uses is refreshed + and the conditions the poller already pinned are moved to the new + `_alarms` entity. One residual: a condition that changes state + between its pin and the rename is reported once under the old entity. + +### What the plugin is bound to + +The **binding** is the OPC-UA `ApplicationUri` of the server the plugin actually +held a session with, read off that session (the `ServerArray`, whose first entry +is that URI). It is not persisted, so it exists only for the life of the process. + +- A re-scan looks for **that server and no other**, at any address: the bound + `ApplicationUri` is an input to the selection, so a foreign server does not win + by sorting lower. A sweep that finds no hit carrying it selects nothing, the + endpoint stands and the `PLC_COMMS_LOST` fault stands with it. +- A **different server at the bound address** is caught when the session comes + up: the live `ApplicationUri` is read, the mismatch is logged at WARN, the + session is dropped, no link-state clear is sent, and the reconnect loop keeps + trying. Both reports are once per distinct URI per outage, and the list is + cleared as soon as the bound server is reached again. +- A PLC that **moved** - new address, same `ApplicationUri` - is re-adopted, + which is what the re-scan is for. +- **Replacing a PLC is a recommissioning**: restart the plugin against the new + one. Adopting a different PLC silently would re-point every SOVD entity at + hardware nobody asked for and would clear the outage as if the link had healed. + +Three cases have no binding, and in each the next adoption is unconstrained: + +1. Nothing has been connected yet - the gateway that started before its PLC, and + every process after a restart, because the binding is never persisted. +2. An operator-configured `endpoint_url`, which runs no discovery at all. +3. A server that publishes **no `ApplicationUri`**: there is nothing to bind to, + so after a drop a re-scan accepts whichever server answers. This is logged at + WARN when such a server is adopted. Re-scanning stops as soon as a session is up, and never starts at all when an `endpoint_url` is configured. @@ -774,8 +835,8 @@ Safety / OT posture: plugin already polls. - Secured-only servers (no None/Anonymous endpoint) are surfaced in the log as leads requiring operator credentials - never auto-connected or probed. A - re-scan whose outcome has not changed reports at DEBUG instead of repeating - the whole report, so a recurring sweep does not bury the rest of the log. + re-scan whose outcome has not changed reports at DEBUG, so a recurring sweep + does not bury the rest of the log under the same report. - The scan is bounded (short connect timeout, capped concurrency) and CIDRs wider than /16 are rejected to prevent an accidental broad sweep. @@ -842,13 +903,30 @@ When the OPC-UA connection stays down for `comms_lost_debounce_ms` continuously, the plugin raises one component-scoped `PLC_COMMS_LOST` fault (a shorter blip during a normal reconnect does not flap it). -The fault is cleared on **every** successful connect, both the initial one and -every later reconnect, whether or not this process was the one that raised it. -The fault manager keys faults by fault code and persists them, so a fault raised -before a gateway restart is still standing while the new process has no memory -of it. Clearing only what the running process remembered left exactly that fault -CONFIRMED for good. The clear is fire-and-forget, so a clear for a fault that is -not there is harmless. +Every successful connect - the initial one and every later reconnect - puts a +decision about the standing `PLC_COMMS_LOST` on the poll thread. The fault +manager keys faults by fault code and persists them, which cuts both ways: a +fault raised before a gateway restart is still standing while the new process +has no memory of it, so the decision cannot rest on this process's own memory; +and a gateway that also loads another field-bus bridge shares the one +`PLC_COMMS_LOST` code with it, so this link coming back says nothing about the +other bridge's link. + +So the plugin asks the fault manager who reported the standing fault, and clears +it only when **every** reporting source is an id this process could have +reported under: the component id it serves, the last few `opcua-` stand-ins +it assigned in this process, and the node-map default a YAML without a +`component_id` reports under. `ClearFault` carries no source and clears the whole +row, so a fault whose sources include a foreign one is left standing and the +decision is logged at INFO. A `PLC_COMMS_LOST` that two bridges raised is cleared +by an operator; the fault manager has no per-source de-assert. + +The decision is driven from the poll thread and only while the session is up, so +a clear is never sent against a link that is down; an answer that arrives after +the link dropped is held until the next connect. A store that cannot be reached, +a probe that goes unanswered within `fault_service_timeout_ms`, and a clear the +bounded pending-dispatch buffer had to give up under load all leave the decision +owed, and the next poll takes it again. Node map entries also support an optional `ros2_topic` field to override the auto-generated ROS 2 topic name for the PLC value bridge: @@ -886,7 +964,7 @@ Write operations use the `set_` prefix convention: | `OPCUA_CONDITION_REPLAY` | `method` / `read` / `auto` / `off` | | `OPCUA_REQUIRE_CONFIRM_FOR_CLEAR` | `0`/`false`/`no`/`off` to clear native alarms on Acknowledge alone (Confirm-less servers) | | `OPCUA_COMMS_LOST_ENABLED` | `0`/`false`/`no`/`off` to disable the `PLC_COMMS_LOST` fault | -| `OPCUA_COMMS_LOST_DEBOUNCE_MS` | Continuous down time (ms) before `PLC_COMMS_LOST` is raised (clamped to [0, 3600000] ms; non-numeric / out-of-range keeps the existing value) | +| `OPCUA_COMMS_LOST_DEBOUNCE_MS` | Continuous down time (ms) before `PLC_COMMS_LOST` is raised. A value above 3600000 ms is clamped, with a warning, as on the JSON path; a non-numeric or negative value is refused with a warning and the existing value is kept | ## Hardware Deployment diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/device_identity.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/device_identity.hpp index 5093dac4e..a6b90fce7 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/device_identity.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/device_identity.hpp @@ -53,6 +53,13 @@ struct ComponentIdentity { /// if ``endpoint_url`` is unparseable and no identity fields are present. ComponentIdentity derive_component_identity(const OpcuaClient::DeviceInfo & info, const std::string & endpoint_url); +/// True when ``info`` names the device itself - a DI nameplate or a BuildInfo +/// manufacturer/product. ``derive_component_identity`` then returns a +/// device-derived id (cases 1-2). False for an empty or unusable read, which is +/// what a server answers on the first read of a session whose address space is +/// not up yet. +bool component_identity_has_nameplate(const OpcuaClient::DeviceInfo & info); + /// Whether a live nameplate read over this connection may outrank the /// operator-authored manifest in the identity merge. /// diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp index 9f73d21d2..51076717a 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp @@ -113,9 +113,9 @@ struct OpcuaDiscoveryConfig { /// stop the recurring sweep (see OpcuaPlugin::effective_rescan_interval_s). /// The startup scan always runs once. The cadence only governs how often the /// disconnected reconnect loop scans again, so a gateway that started before - /// its PLC finished booting adopts the PLC when it appears instead of retrying - /// the fallback endpoint forever. Never used once an endpoint is configured - /// explicitly, and never while a session is up. + /// its PLC finished booting adopts the PLC when it appears, at that cadence, + /// for as long as it stays disconnected. Never used once an endpoint is + /// configured explicitly, and never while a session is up. std::optional interval_s; /// Only auto-register endpoints that expose a None + Anonymous endpoint (what @@ -176,13 +176,20 @@ class NetworkDiscovery { /// derived local /24. Exposed for logging / tests. std::vector resolve_subnets() const; - /// Pick the best endpoint for single-endpoint "auto endpoint" mode: an - /// OPC-UA data server (not an LDS) that identified cleanly and, when - /// ``anonymous_none_only``, offers a None + Anonymous endpoint. Deterministic - /// (lowest ip:port). Returns nullptr when no candidate qualifies. Pure / - /// static so the selection policy is unit tested without a network. + /// Pick the endpoint for single-endpoint "auto endpoint" mode: an OPC-UA data + /// server (not an LDS) that identified cleanly and, when + /// ``anonymous_none_only``, offers a None + Anonymous endpoint. + /// + /// ``bound_application_uri`` is the identity of the server the caller already + /// holds a session with. When it is set, the only candidate is the hit + /// carrying that ApplicationUri, at whatever address it answers on, and + /// nullptr means that server was not found. When it is empty the choice is + /// the deterministic lowest ip:port. Returns nullptr when no candidate + /// qualifies. Pure / static so the selection policy is unit tested without a + /// network. static const DiscoveredEndpoint * select_auto_endpoint(const std::vector & eps, - bool anonymous_none_only); + bool anonymous_none_only, + const std::string & bound_application_uri = {}); private: OpcuaDiscoveryConfig cfg_; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/node_map.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/node_map.hpp index 6e2ef92ba..02261a29c 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/node_map.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/node_map.hpp @@ -426,6 +426,11 @@ class NodeMap { const std::string & component_id() const { return component_id_; } + + /// The component id a NodeMap carries before anything overrides it: neither + /// a device nameplate nor an operator's YAML, so a fault reported under it + /// belongs to this plugin as much as any later id does. + static constexpr const char * kDefaultComponentId = "opcua_device"; const std::string & component_name() const { return component_name_; } @@ -516,7 +521,7 @@ class NodeMap { // these from the device's own read identity (set_component_identity); an // explicit node-map YAML overrides them from ``component_id`` / // ``component_name``. No device-specific product string is ever baked in. - std::string component_id_ = "opcua_device"; + std::string component_id_ = kDefaultComponentId; std::string component_name_ = "OPC UA Device"; AutoBrowseConfig auto_browse_config_; }; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp index 9888798ca..9031f39a4 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp @@ -132,6 +132,13 @@ class OpcuaClient { /// Get the endpoint URL (for status reporting) std::string endpoint_url() const; + /// The connected server's own ApplicationUri, read from the ServerArray + /// (ns=0, i=2254), whose first entry is that URI per OPC-UA Part 5. One read + /// on the open session, so a caller can tell which server it reached + /// independently of the address it reached it at. Empty when the session is + /// down, when the read fails, or when the server publishes no ServerArray. + std::string read_server_application_uri(); + /// Get the current config (for reconnection) OpcuaClientConfig current_config() const; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index 0e831a2d9..619c5d5e5 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -22,6 +22,7 @@ #include "ros2_medkit_opcua/opcua_poller.hpp" #include +#include #include #include @@ -44,6 +45,7 @@ #include #include #include +#include #include #include @@ -128,6 +130,40 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // false for the Component itself, which has no entity_defs entry. bool has_operations(const std::string & entity_id) const override; + /// How many times the fault store has ANSWERED the link-state clear gate. + /// Read by tests that have to tell "the store said this fault belongs to + /// another bridge" apart from "the store was never asked" - an absent clear + /// looks identical either way, and no log or REST response separates them. + uint64_t comms_lost_probe_count_for_test() const { + std::lock_guard lock(comms_lost_probe_state_->mutex); + return comms_lost_probe_state_->answers; + } + + /// Substitute the discovery sweep and identify before configure(). + /// configure() keeps whatever is set here and installs the real POSIX / + /// open62541pp probes otherwise, so a test can drive the auto-endpoint and + /// rescan paths without a network while the endpoints it names are real. Not + /// part of the plugin's supported surface. + void set_discovery_io_for_test(PortScanFn scan, IdentifyFn identify) { + discovery_scan_fn_ = std::move(scan); + discovery_identify_fn_ = std::move(identify); + } + + /// How many sweeps found no hit carrying the binding while one was held (see + /// select_auto_endpoint). Read by tests that have to tell "the sweep saw a + /// foreign server and selected nothing" apart from "the sweep found nothing + /// at all", which an unchanged endpoint looks identical for. + uint64_t rescan_refused_count_for_test() const { + return rescan_refusals_.load(); + } + + /// How many sessions have been dropped for reaching a server other than the + /// one this bridge is bound to. Read by tests that have to tell that drop + /// apart from a connect that simply failed. + uint64_t binding_mismatch_count_for_test() const { + return binding_mismatch_disconnects_.load(); + } + /// The address-space walk configuration after configure() has merged the /// node map and the ROS parameters. Read by tests that need to see which /// source supplied a setting, which no REST response exposes. @@ -173,9 +209,9 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // // The "scanning [subnets]" announcement is NOT part of that report. It is sent // before the sweep runs, because a wide subnet takes minutes and an operator - // watching start-up has to see the gateway working rather than hung. It says - // what the pass is about to do rather than what it found, so it carries the - // same first-pass / rescan levelling on its own. + // watching start-up has to see the gateway working. It says what the pass is + // about to do, which is known before the outcome is, so it carries the same + // first-pass / rescan levelling on its own. struct DiscoveryReporter { std::function info; std::function warn; @@ -201,10 +237,19 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // @param reporter operator-visible log sinks + repeat-suppression memory // @param cancelled abort predicate handed to NetworkDiscovery::run, so a // shutdown does not have to wait out a full sweep + // @param application_uri when non-null, receives the selected server's + // ApplicationUri - the identity the binding is made of + // @param bound_application_uri the identity of the server the caller already + // holds a session with. Set, it is what selection looks for, at any + // address; empty, selection falls back to the lowest ip:port. A sweep + // that finds no hit carrying it selects nothing: the server this + // bridge is bound to is not on the network right now. static std::optional discover_endpoint(const OpcuaDiscoveryConfig & config, bool endpoint_configured, const PortScanFn & scan, const IdentifyFn & identify, const DiscoveryReporter & reporter, - const std::function & cancelled = {}); + const std::function & cancelled = {}, + std::string * application_uri = nullptr, + const std::string & bound_application_uri = {}); // Seconds between reconnect rescans, or 0 when the reconnect loop must never // rescan. That is the answer when discovery is disabled, when an endpoint was @@ -212,7 +257,7 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // means "keep discovery on but leave the startup scan one-shot". An UNSET // interval is the config-less case - it cannot name a cadence and is the one // that most needs its PLC adopted once it finishes booting - so it takes the - // built-in default instead of never rescanning. + // built-in default and keeps rescanning. static int effective_rescan_interval_s(const OpcuaDiscoveryConfig & config, bool endpoint_configured); // Default reconnect rescan cadence, in seconds, when discovery is enabled with @@ -237,6 +282,23 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, std::chrono::steady_clock::time_point * last_scan_end, const std::function()> & sweep); + // ``rescan_step`` with nothing escaping it. + // + // The sweep runs on the poll thread, which has no try of its own and is not + // the thread that owns the process's error handling: ``std::system_error`` + // from a thread the parallel scan could not create, or ``bad_alloc`` from a + // wide target list, leaves the whole gateway to ``std::terminate``. A sweep + // that failed is a sweep that found nothing, so anything thrown is logged at + // WARN and answered with nullopt; the cadence stamp ``rescan_step`` takes + // still holds, so the next sweep is an interval away. The handlers allocate + // nothing, because a bad_alloc is one of the throws they answer. This is the + // callable the plugin's rescan hook runs, so a test that drives it drives + // what the poll thread drives. + static std::optional rescan_guarded(int interval_s, + const std::function & now, + std::chrono::steady_clock::time_point * last_scan_end, + const std::function()> & sweep); + // Ceiling for the poller's exponential reconnect backoff. // // Without discovery this is ``default_ceiling`` (60 s). While the reconnect @@ -259,12 +321,57 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // the real server, the device can finally name itself, and the SOVD component // must stop serving the placeholder. Pure / static so the rule is testable // without a server. - static std::optional rederived_component_identity(const std::string & current_id, + // What the caller knows about the identity it is serving, so the rule below + // can be pure. The plugin owns the stand-in list these fields summarise. + struct ComponentIdentityState { + std::string current_id; ///< the id being served + bool current_is_placeholder{false}; ///< current_id came from a device read with no nameplate + }; + + static std::optional rederived_component_identity(const ComponentIdentityState & state, const OpcuaClient::DeviceInfo & info, const std::string & endpoint_url); + // Extra reads the connect hook spends before it lets the session subscribe, + // and the pause between them. The address space of a PLC that has just + // accepted a session fills in over the following second, and this is the only + // window in which a rename still precedes the conditions the server replays. + // Plugin start-up pays this burst once, and only when the device answers the + // first read without a nameplate. + static constexpr int kIdentityConnectRetries = 3; + static constexpr std::chrono::milliseconds kIdentityConnectRetryPause{250}; + + // Device-info reads the POLL path may spend on one session, counted apart + // from the connect burst above. A server that names itself settles on its + // first read; one that has still not named itself after the connect read, the + // burst and these keeps its stand-in for the session, so a nameplate-less + // device costs a bounded number of reads per session. + static constexpr int kMaxIdentityPollReads = 5; + + // What one call of the read loop did. + struct IdentityReadOutcome { + OpcuaClient::DeviceInfo info; ///< the last read; empty when no read was made + int reads_spent{0}; ///< the caller's budget after this call + bool read_made{false}; ///< false when the budget was already spent + bool interrupted{false}; ///< a pause asked the loop to stop + }; + + // Read device info until the device names itself, the budget runs out, or a + // pause is refused. + // + // One read is made per attempt, up to ``extra_reads`` further attempts, each + // preceded by ``pause_fn``. ``pause_fn`` returns false to stop the loop, which + // is how a shutdown ends the connect burst without waiting it out; its + // argument is the pause the caller asked for. ``reads_spent`` in and out is + // the caller's own budget against ``max_reads``, so the connect burst and the + // poll path can hold separate ones. Injected read and pause so the bound and + // the cadence are testable without a server and without sleeping. + static IdentityReadOutcome identity_reads(int extra_reads, int reads_spent, int max_reads, + const std::function & read_fn, + const std::function & pause_fn); + // Why a ClearFault is being sent. Two properties follow from it and nothing - // else does, so the origin travels instead of a pair of loose booleans: + // else does, so the origin is what travels and both are derived from it: // - whether the correlation cascade must be skipped // (``clear_skips_correlation``), which goes on the wire, and // - whether the clear is re-derivable (``clear_is_link_state``), which is @@ -298,6 +405,71 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return origin == ClearOrigin::LinkState; } + // Whether a link-state clear may be sent, given what the fault store answered + // when asked for that fault code. + // + // ClearFault carries no source: both storages clear the whole row. One + // gateway may load two field-bus bridges (``plugins: ["opcua", + // "beckhoff_ads"]``) that both raise ``PLC_COMMS_LOST``, and the fault + // manager keys faults by code alone, so a fault whose reporting sources + // include one this process never reported under is a fault another bridge + // still holds. The clear goes out only when EVERY reporting source is one of + // ``my_ids``: the component id being served, every placeholder id this + // process assigned, and the NodeMap default. A fault that is absent, carries + // no sources, or names any foreign source is left standing. A shared + // ``PLC_COMMS_LOST`` is healed by an operator clear; the fault manager has no + // per-source de-assert. Pure + static so the rule is testable without a fault + // manager. + static bool link_state_clear_permitted(bool fault_found, const std::vector & reporting_sources, + const std::unordered_set & my_ids); + + // Whether a parked answer belongs to the probe the poll thread is waiting for. + // + // Each probe carries a number, stamped when the request goes out. The timeout + // branch moves that number on before it re-owes the decision, because rclcpp + // takes a pending entry out of the client before it invokes the callback and + // outside its own mutex: a callback that won that race parks its answer while + // the poll thread has already given the probe up. Pure + static so the rule is + // testable without a store. + static bool comms_lost_generation_current(uint64_t answered_generation, uint64_t current_generation) { + return answered_generation == current_generation; + } + + // Whether a link-state decision may be probed for on this poll. + // + // A probe costs a service round trip and its answer decides whether a fault + // is cleared, so it is sent only when all four hold: a decision is owed, the + // session is up (a clear while the link is down would clear a fault that is + // genuinely standing), no probe is outstanding, and the store is reachable. + // Pure + static so the conjunction is testable without a session. + static bool comms_lost_probe_due(bool owed, bool connected, bool probe_in_flight, bool store_ready) { + return owed && connected && !probe_in_flight && store_ready; + } + + // Whether an answer the store parked may be acted on. + // + // The answer describes the store at the moment the probe was served, and the + // link can die between the probe and the answer. "The link is up now" is not + // enough to tell that it never went down: the decision is driven from + // publish_values, which the poll loop reaches only while connected, so an + // answer parked just before a drop is next looked at on the first tick AFTER + // the reconnect - when the link is up again and the store's answer describes + // an outage that has since been reported. The session the probe was sent on + // is what settles it: ``probe_session`` is the connection generation stamped + // when the request went out and ``live_session`` the one in force now. + // Clearing on an answer from an earlier session would clear a fault that is + // standing for a reason. Pure + static so the conjunction is testable without + // a session. + static bool comms_lost_answer_applicable(bool connected, bool has_answer, uint64_t probe_session, + uint64_t live_session) { + return connected && has_answer && probe_session == live_session; + } + + // How long a fault-store probe may stay outstanding before it is dropped and + // the decision is owed again. A fire-and-forget request whose answer never + // comes would otherwise hold the decision for the life of the process. + static constexpr std::chrono::milliseconds kDefaultFaultServiceTimeout{5000}; + // Whether a discovery sweep must stop now, given the two independent stop // signals. Static and pure so both inputs are testable: the member // ``discovery_cancelled()`` only reads them off the process and hands them @@ -315,6 +487,18 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return shutdown_requested || !rclcpp_ok; } + // ``discovery_cancelled_for`` reading the second signal off the node's OWN + // context. A host that builds its GatewayNode on a private context + // (``NodeOptions().context(...)``) never initialises the default one, and + // ``rclcpp::ok()`` with no argument is then false for the life of the + // process: every sweep cancels before it starts and the PLC is never found. A + // null ``context`` - the plugin before set_context() has handed it one - + // reads the default context, which is what a gateway built the ordinary way + // runs on. + static bool discovery_cancelled_for_context(bool shutdown_requested, const rclcpp::Context::SharedPtr & context) { + return discovery_cancelled_for(shutdown_requested, context ? rclcpp::ok(context) : rclcpp::ok()); + } + // Which kind of clear a fault-detection signal going inactive is. The poller // emits the component-scoped ``PLC_COMMS_LOST`` clear through the same // callback as every device alarm, and only that one is a link-state event. @@ -358,9 +542,10 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // report, a device alarm going inactive, an operator's scoped clear - so those // rank together and age out oldest-first, exactly as the buffer behaved before // any of this. A link-state clear is what a full buffer gives up first, and an - // incoming one is refused rather than pushing a one-shot dispatch out. At most - // ONE clear per fault code is pending at a time (a newer one moves to the - // back, so an interleaved report-then-clear still flushes in that order). + // incoming one is refused so that every one-shot dispatch in the buffer + // survives. At most ONE clear per fault code is pending at a time (a newer one + // moves to the back, so an interleaved report-then-clear still flushes in that + // order). // // Without the link-state ranking a flapping link enqueued one connect-time // clear per reconnect attempt and pushed real alarm reports out of the buffer. @@ -399,13 +584,51 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // flag and how the pending buffer ranks it. See ClearOrigin. void send_clear_fault(const std::string & fault_code, ClearOrigin origin = ClearOrigin::DeviceAlarm); - // Clear PLC_COMMS_LOST after the initial connect in set_context() succeeded. - // Unconditional on purpose: the fault manager keys faults by fault_code and - // persists them, so a comms-lost fault raised before a gateway restart is - // still standing in the store while this process has no memory of raising it. - // The poller's own reconnect clear can never reach that case, because a - // successful first connect means the reconnect arm is never entered. - void clear_comms_lost_on_connect(); + // Record that a link-state decision about PLC_COMMS_LOST is due. + // + // Called from every point that learns the link state changed or that a + // decision was lost: a successful connect (set_context and the poller's + // reconnect arm, the latter through on_alarm_change), a link-state clear the + // bounded buffer evicted or refused, and a probe that timed out. Setting a + // flag is all it does; the decision itself belongs to the poll thread. + void owe_comms_lost_decision(); + + // Drive the owed link-state decision one step. Poll thread only (called from + // publish_values), which is the thread that performs the config-less rename, + // so component ids are read here without a lock. + // + // One step is: consume an answer the executor thread parked, or drop a probe + // that outlived the store timeout, or send a probe when one is due + // (comms_lost_probe_due). The store is asked asynchronously because the first + // decision is owed from set_context, where the gateway's executor is not + // spinning yet and a blocking wait would never be answered. + void drive_comms_lost_decision(); + + // Apply the gate to an answer the store gave and send the clear it permits. + // Poll thread only. + void apply_comms_lost_answer(bool found, const std::vector & reporting_sources); + + // The ids a fault reported by THIS process can carry: the component id being + // served, every placeholder this process assigned, and the NodeMap default. + // Poll thread only (see drive_comms_lost_decision). + std::unordered_set reporting_ids_of_this_process() const; + + // Check the session that just came up against the binding, and establish the + // binding when there is none. Returns false when the session reached a + // different server, in which case it has been dropped and the caller must + // treat the connect as not having happened. Called from every path that + // observes a successful connect. + bool bind_or_drop_session(); + + // Record an ApplicationUri this bridge has reported as not its own, keeping + // the last kMaxRefusedApplicationUris. Returns true the first time a given + // URI is seen since the last adoption, which is what makes the report once + // per outage per server. + bool note_refused_application_uri(const std::string & uri); + + // Record an id assigned from a read with no nameplate, keeping the last + // kMaxPlaceholderIds. Poll thread / set_context thread. + void remember_placeholder_id(const std::string & id); // Dispatch now if the fault_manager service is matched, else buffer the // dispatch (bounded, order-preserving) to be flushed once it appears. @@ -432,15 +655,29 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // node_map_mutex_. No-op (never called) when auto_browse is disabled. void run_auto_browse(); - // Poll-thread hook (from publish_values): re-derive the SOVD component - // identity from the device once a NEW session is up, in config-less mode only + // Re-derive the SOVD component identity from the device once a NEW session is + // up, in config-less mode only. Called from PollerConfig::on_connected, which + // the poller fires right after a session comes up and BEFORE it + // (re)subscribes, and again from publish_values on every poll until the + // identity is settled (a first device-info read that came back empty leaves + // it unsettled, and the poll retries within the same session) // (an explicit node map owns the name). This is what stops a gateway that // started before its PLC from serving the ``opcua-`` // placeholder for the life of the process after discovery adopted the real // server. Logs the change at INFO and rebuilds every derived reference (the // ``_alarms`` entity, entity_defs) under the node-map lock. // No-op when the identity is unchanged. - void maybe_rederive_component_identity(); + // + // Running before the (re)subscribe is what keeps the rename ahead of the + // conditions the server replays on the adopted session: apply_condition_state + // pins an entity id at the FIRST observation of a ConditionId, so a + // ConditionRefresh burst delivered before the rename would pin every replayed + // fault under the old ``_alarms`` entity, which the rename then drops. + // ``extra_reads`` is how many further device-info reads this call may spend, + // kIdentityConnectRetryPause apart, when the first comes back without a + // nameplate. The connect hook spends kIdentityConnectRetries; the poll-thread + // retry spends none and draws on kMaxIdentityPollReads for the session. + void maybe_rederive_component_identity(int extra_reads); // Poll-thread hook (from publish_values): re-run auto_browse when the client // has established a new session since the last walk. Covers the field case @@ -470,8 +707,8 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, DiscoveryReporter discovery_reporter(std::string * previous_outcome) const; // Abort predicate handed to a discovery sweep: reads the two stop signals off - // the process and applies ``discovery_cancelled_for``, which holds the rule - // and the reasoning behind it. + // the process and applies ``discovery_cancelled_for_context``, which holds the + // rule and the reasoning behind it. bool discovery_cancelled() const; // Poll-thread hook bound into PollerConfig::rediscover_endpoint whenever @@ -487,6 +724,11 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, std::atomic shutdown_requested_{false}; ros2_medkit_gateway::RosPluginContext * ctx_{nullptr}; + // The context the gateway node was built on. A discovery sweep reads the + // shutdown signal of THAT context. Null until set_context() (the plugin is + // constructed before it has a node), which discovery_cancelled_for_context + // treats as "use the default". + rclcpp::Context::SharedPtr context_; OpcuaClientConfig client_config_; PollerConfig poller_config_; std::string node_map_path_; @@ -508,13 +750,36 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, PortScanFn discovery_scan_fn_; IdentifyFn discovery_identify_fn_; // When the last discovery pass FINISHED, so the reconnect rescan honours the - // cadence instead of sweeping the subnet on every reconnect attempt. Stamped - // by the startup scan, then only ever read/written on the poll thread. See - // rescan_step for why the end of the sweep is the reference point. + // cadence and sweeps the subnet at most once per interval, whatever the + // reconnect attempts do. Stamped by the startup scan, then only ever + // read/written on the poll thread. See rescan_step for why the end of the + // sweep is the reference point. std::chrono::steady_clock::time_point last_discovery_scan_end_{}; + // The ApplicationUri of the server this plugin is bound to: the identity of + // the server a session was actually established with, read off that session + // (OpcuaClient::read_server_application_uri). It is what discovery looks for + // on every later sweep and what a fresh session is checked against. + // + // Empty until a session has been held: a process that has never connected is + // bound to nothing, so its first adoption is unconstrained. It is also empty + // for a server that publishes no ApplicationUri, which therefore cannot be + // bound to. A restart clears it, because it is never persisted. Written on + // the set_context thread, then the poll thread. + std::string bound_application_uri_; + // ApplicationUris already reported as not this bridge's, so one outage does + // not log the same foreign server every interval_s. Bounded, and cleared on + // every successful adoption so a later outage reports again. Poll thread only. + std::vector refused_application_uris_; + static constexpr size_t kMaxRefusedApplicationUris = 8; + // Sweeps that found no hit carrying the bound identity, and sessions dropped + // for reaching a different server (see the *_for_test accessors). + std::atomic rescan_refusals_{0}; + std::atomic binding_mismatch_disconnects_{0}; + // Outcome digest of the previous discovery pass, so an unchanged rescan - // reports at DEBUG instead of repeating the whole report every interval_s. - // Poll thread only (the startup scan runs before the poller exists). + // reports at DEBUG and the whole report goes out once per change, not once + // per interval_s. Poll thread only (the startup scan runs before the poller + // exists). std::string last_discovery_outcome_; std::unique_ptr client_; @@ -550,6 +815,9 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // differs, mirroring device_identity_generation_. Written on the set_context // thread (happens-before the poller starts) then only on the poll thread. uint64_t component_identity_generation_{0}; + // Session the poll-read counter below belongs to, so a new session gets its + // full kMaxIdentityPollReads budget. + uint64_t component_identity_read_generation_{0}; // ROS 2 service clients for fault reporting struct FaultClients; @@ -572,6 +840,74 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, std::mutex pending_reports_mutex_; std::vector pending_reports_; + // Where the fault store's answer is parked between the executor thread that + // receives it and the poll thread that acts on it. + // + // It is a separately owned block, not plugin members, because the GetFault + // callback outlives nothing it can check: rclcpp hands a callback to an + // executor thread that may already be inside it when the plugin is destroyed, + // and a mutex the plugin owns cannot make its own destruction safe. The + // callback holds a weak_ptr and does nothing when the lock fails. + // + // ``generation`` counts probes. The poll thread stamps it when it sends one; + // the callback carries the stamp it was created with and stores an answer + // only while it still matches, so an answer to a probe that already timed out + // cannot be read as the answer to the one that replaced it. + struct CommsLostProbeState { + std::mutex mutex; + uint64_t generation{0}; ///< the probe the poll thread is waiting for + uint64_t answered_generation{0}; ///< the probe the parked answer belongs to + bool answered{false}; + bool found{false}; + std::vector sources; + uint64_t answers{0}; ///< answers ever parked (comms_lost_probe_count_for_test) + }; + + // Link-state decision state. ``owed`` is set by whoever learns a decision is + // due (set_context, the poll thread via on_alarm_change, the REST thread via + // the pending buffer) and taken by the poll thread with exchange(false) + // BEFORE it sends a probe, so an owe raised while the send is in progress + // survives it. Everything else belongs to the poll thread, except the answer + // slot inside the state block. + // + // The buffer gives a link-state clear up first because the next reconnect + // re-derives it, which holds for the poller's reconnect clear and not for the + // one a successful connect queued: a gateway restarting with a persisted + // PLC_COMMS_LOST behind 256 alarm edges has no reconnect coming, so the debt + // is recorded here instead. + std::atomic comms_lost_decision_owed_{false}; + bool comms_lost_probe_in_flight_{false}; + std::chrono::steady_clock::time_point comms_lost_probe_sent_{}; + int64_t comms_lost_probe_request_id_{0}; + // OpcuaClient::connection_generation at the moment the probe went out. An + // answer is acted on only while that session is still the live one. + uint64_t comms_lost_probe_session_{0}; + std::shared_ptr comms_lost_probe_state_{std::make_shared()}; + // A PLC_COMMS_LOST raise this process sent since the last decision. The store + // can answer a probe before it has processed a ReportFault that was already + // on the wire, so one further decision follows any that had a raise behind + // it. Set on the poll thread, taken on the poll thread. + std::atomic comms_lost_raise_since_decision_{false}; + // How long a probe may stay outstanding (kDefaultFaultServiceTimeout unless + // ``fault_service_timeout_ms`` says otherwise). + std::chrono::milliseconds fault_service_timeout_{kDefaultFaultServiceTimeout}; + + // Component ids this process assigned from a device read with no nameplate, + // i.e. ``opcua-`` for some endpoint it targeted, oldest first. A fault + // reported while one of these was being served carries it as its source, and + // healing that fault after the device names itself is the whole point of + // keeping them. One entry per adopted endpoint: the FIRST kMaxPlaceholderIds + // distinct stand-ins of a process are kept, and a re-adopted one keeps its + // place, so the id set the gate builds stays small on a link that is + // re-discovered many times. Written on the set_context thread then the poll + // thread. + std::vector placeholder_ids_; + static constexpr size_t kMaxPlaceholderIds = 8; + // Poll-path device-info reads spent on this session's identity, bounded by + // kMaxIdentityPollReads. The connect burst carries a budget of its own and + // does not draw on this one. + int component_identity_poll_reads_{0}; + // Tracks which non-numeric nodes have already been warned about (avoids log spam). // Instance member instead of static to survive plugin reload (dlclose/dlopen). std::unordered_set warned_non_numeric_; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp index 5e3c183fd..717292306 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_poller.hpp @@ -148,6 +148,25 @@ struct PollerConfig { /// current one. Without it the reconnect loop retries the same endpoint /// forever, which strands a gateway that scanned before its PLC had booted. std::function()> rediscover_endpoint; + /// Optional hook fired once per session, immediately after a connect + /// succeeds and BEFORE the link-state edge and before the poller + /// (re)subscribes to data changes and alarm events. Bound by the plugin to + /// whatever must be settled before the server replays anything - the + /// identity check and the config-less component rename, in particular. + /// + /// Returns false to reject the session: the owner has already dropped it (it + /// reached a server the owner is not willing to poll), so the poller treats + /// the connect as not having happened - no link-state clear, no subscribe, + /// back to the reconnect backoff. + /// + /// Position matters: ``apply_condition_state`` pins a fault's entity id at + /// the FIRST observation of a ConditionId, and the ConditionRefresh burst + /// that follows ``setup_event_subscriptions()`` is that first observation for + /// every condition the device had standing. Renaming after it would file + /// those faults under an entity the rename then drops, orphaning them and + /// every later clear. Runs on the poll thread (or, for the first session, on + /// the thread that called ``start()``). + std::function on_connected; }; /// Fault code of the component-scoped OPC-UA connection fault the poller raises @@ -301,6 +320,61 @@ class OpcuaPoller { /// distinct). Pure and static so it is unit-testable without a server. static bool node_ids_equivalent(const std::string & a, const std::string & b); + /// The alarm configuration the event path reads, copied out of the NodeMap. + /// + /// ``on_event`` runs on ``event_pump_thread_`` while the poll thread may be + /// rewriting the node map underneath it: the config-less rename clears and + /// reassigns ``auto_alarms.entity_id`` and rebuilds entity_defs under the + /// plugin's node-map lock, which the event path does not hold and cannot + /// take (it must not block a subscription callback on an HTTP read). A + /// ConditionRefresh burst on the first adopted session lands on exactly that + /// window. So the poller keeps its own copy and the event path reads nothing + /// else. + struct AlarmRouting { + std::vector event_alarms; + AutoAlarmsConfig auto_alarms; + /// node-id string -> hosting entity id. That entity id is the whole of + /// what the event path needs ``NodeMap::find_by_node_id`` for, so the copy + /// carries the answer itself. + std::unordered_map entity_by_node_id; + }; + + /// Re-copy the alarm routing from the node map. Called at subscribe time (so + /// the copy and the monitored items are taken from one state of the map) and + /// callable by the owner after it has renamed anything the routing carries. + /// Poll thread / start() thread only - the event path never writes it. Not + /// part of the poller's supported surface (see alarm_routing()). + void refresh_alarm_routing(); + + /// The routing the event path reads. A shared_ptr snapshot, so a refresh + /// swaps in a new one and a callback already holding the old one finishes + /// against a consistent copy whose strings stay put. + /// + /// This and ``refresh_alarm_routing`` exist for the owner and its tests; they + /// are not part of the poller's supported surface and carry no compatibility + /// promise. + std::shared_ptr alarm_routing() const; + + /// Move every tracked condition hosted on ``old_entity_id`` to + /// ``new_entity_id``, under the lock ``apply_condition_state`` pins them + /// with. A condition's entity is pinned at the first sighting of its + /// ConditionId, so the config-less rename has to reach the ones already + /// pinned as well as the routing new ones are derived with; every later + /// report and clear for those ConditionIds then carries the new entity. A + /// condition that changes state between the pin and this call is reported + /// once under the old entity. No-op for an empty or unchanged id. + void repin_auto_alarms_entity(const std::string & old_entity_id, const std::string & new_entity_id); + + /// ``apply_condition_state`` reached without a server. Drives the pin, the + /// state machine and the dispatch exactly as a delivered event does, so the + /// pinning rules are testable without an address space. Not part of the + /// poller's supported surface (see alarm_routing()). + void apply_condition_state_for_test(const AlarmEventConfig & cfg, const opcua::NodeId & condition_id, + const AlarmEventInput & input, uint16_t severity, const std::string & message, + const opcua::ByteString * event_id, bool require_confirm_for_clear) { + apply_condition_state(cfg, condition_id, input, severity, message, event_id, require_confirm_for_clear); + } + /// True only for a real OPC-UA Condition event. Per Part 9 §5.5.2.13 the /// ConditionId SAO resolves to a non-null NodeId only for AlarmConditionType /// (and subtype) instances; a plain BaseEvent / SystemEvent notification - @@ -428,6 +502,12 @@ class OpcuaPoller { std::atomic event_subscription_id_{0}; std::vector event_monitored_item_ids_; + // The event path's own copy of the alarm configuration (see AlarmRouting). + // Written only from the poll thread / start() thread, read from the event + // pump thread; the mutex is held just long enough to swap the pointer. + mutable std::mutex alarm_routing_mutex_; + std::shared_ptr alarm_routing_; + mutable std::shared_mutex conditions_mutex_; std::unordered_map conditions_; // ConditionId stringForm -> runtime diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/device_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/device_identity.cpp index de2196449..def402688 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/device_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/device_identity.cpp @@ -21,6 +21,9 @@ namespace ros2_medkit_gateway { namespace { constexpr const char * kOpcuaSource = "opcua"; +/// Prefix of the neutral endpoint-derived component id. +constexpr const char * kEndpointIdPrefix = "opcua-"; + /// Lowercase underscore slug: alnum kept, every other run collapsed to a /// single '_', leading/trailing '_' trimmed. Yields a URL-safe SOVD id. std::string slugify(const std::string & in) { @@ -127,13 +130,20 @@ ComponentIdentity derive_component_identity(const OpcuaClient::DeviceInfo & info // is_valid_entity_id and the whole Component would be silently dropped. const std::string host = host_from_endpoint(endpoint_url); if (!host.empty()) { - const std::string fallback = "opcua-" + slugify(host); + const std::string fallback = std::string(kEndpointIdPrefix) + slugify(host); return {fallback, fallback}; } return {}; } +bool component_identity_has_nameplate(const OpcuaClient::DeviceInfo & info) { + // With no endpoint to fall back on, derive_component_identity returns an id + // only when the device named itself, so asking it keeps this answer and that + // precedence one piece of logic. + return !derive_component_identity(info, /*endpoint_url=*/"").id.empty(); +} + bool opcua_identity_trusted(const OpcuaClientConfig & config) { return OpcuaClient::requires_secure_channel(config) && config.reject_untrusted; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp index f3eaceea7..962758a49 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp @@ -74,8 +74,8 @@ bool ipv4_less(const std::string & a, const std::string & b) { // the GetEndpoints identify so they share the same concurrency bound. // // ``cancelled`` is polled by every worker before it takes the next index, so an -// abort stops the fan-out after at most one more probe per worker instead of -// running the remaining tens of thousands. An empty predicate never cancels. +// abort stops the fan-out after at most one more probe per worker, whatever is +// left of the remaining tens of thousands. An empty predicate never cancels. template void parallel_for(size_t count, int max_workers, const std::function & cancelled, Body && body) { if (count == 0) { @@ -459,7 +459,8 @@ std::vector NetworkDiscovery::run(const std::function & eps, - bool anonymous_none_only) { + bool anonymous_none_only, + const std::string & bound_application_uri) { const DiscoveredEndpoint * best = nullptr; for (const auto & ep : eps) { if (ep.protocol != "opcua" || !ep.identify_error.empty()) { @@ -471,6 +472,16 @@ const DiscoveredEndpoint * NetworkDiscovery::select_auto_endpoint(const std::vec if (anonymous_none_only && !ep.anonymous_none_available) { continue; // secured-only: surfaced as a lead, never auto-connected } + if (!bound_application_uri.empty()) { + // A caller already holding a session's identity is looking for that one + // server at whatever address it now answers on. Address order decides + // nothing here: a foreign server that sorts lower would otherwise win + // every sweep and the bound one would never be reached again. + if (ep.application_uri == bound_application_uri) { + return &ep; + } + continue; + } if (best == nullptr || ipv4_less(ep.ip, best->ip) || (ep.ip == best->ip && ep.port < best->port)) { best = &ep; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp index 219b6b071..fe5c3c476 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp @@ -1503,6 +1503,27 @@ void read_di_nameplate(UA_Client * client, uint16_t di_ns, OpcuaClient::DeviceIn } // namespace +std::string OpcuaClient::read_server_application_uri() { + std::lock_guard lock(impl_->client_mutex); + if (!impl_->connected) { + return {}; + } + UA_Variant value; + UA_Variant_init(&value); + std::string uri; + // ServerArray is a String[] whose element 0 is the server's own + // ApplicationUri (Part 5 SS 12.4). + if (UA_Client_readValueAttribute(impl_->client.handle(), UA_NODEID_NUMERIC(0, UA_NS0ID_SERVER_SERVERARRAY), &value) == + UA_STATUSCODE_GOOD) { + if (UA_Variant_hasArrayType(&value, &UA_TYPES[UA_TYPES_STRING]) && value.arrayLength > 0) { + const auto * entries = static_cast(value.data); + uri.assign(reinterpret_cast(entries[0].data), entries[0].length); + } + } + UA_Variant_clear(&value); + return uri; +} + OpcuaClient::DeviceInfo OpcuaClient::read_device_info() { std::lock_guard lock(impl_->client_mutex); DeviceInfo info; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 75ab2ef12..6b7c0b465 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -37,6 +37,7 @@ #include #include #include +#include namespace ros2_medkit_gateway { @@ -189,6 +190,10 @@ std::vector parse_json_ns_list(const nlohmann::json & arr, const char struct OpcuaPlugin::FaultClients { rclcpp::Client::SharedPtr report; rclcpp::Client::SharedPtr clear; + /// Read side of the store, used by nothing but the link-state clear gate: + /// PLC_COMMS_LOST is keyed by code alone, so before clearing it this asks who + /// reported it (see link_state_clear_permitted). + rclcpp::Client::SharedPtr get_fault; }; OpcuaPlugin::OpcuaPlugin() @@ -307,7 +312,12 @@ void OpcuaPlugin::configure(const nlohmann::json & config) { }; if (config.contains("comms_lost_debounce_ms")) { const long long ms = config["comms_lost_debounce_ms"].get(); - poller_config_.comms_lost_debounce = std::chrono::milliseconds(clamp_debounce_ms(ms)); + const long clamped = clamp_debounce_ms(ms); + if (static_cast(clamped) != ms) { + log_warn("plugins.opcua.comms_lost_debounce_ms=" + std::to_string(ms) + " is outside [0, " + + std::to_string(kMaxCommsLostDebounceMs) + "] ms - using " + std::to_string(clamped) + " ms"); + } + poller_config_.comms_lost_debounce = std::chrono::milliseconds(clamped); } if (auto * env = std::getenv("OPCUA_COMMS_LOST_DEBOUNCE_MS")) { // Keep the existing value on a non-numeric / out-of-range / negative @@ -318,12 +328,33 @@ void OpcuaPlugin::configure(const nlohmann::json & config) { char * end = nullptr; const long ms = std::strtol(env, &end, 10); if (end != env && *end == '\0' && errno != ERANGE && ms >= 0) { - poller_config_.comms_lost_debounce = std::chrono::milliseconds(clamp_debounce_ms(ms)); + const long clamped = clamp_debounce_ms(ms); + if (clamped != ms) { + log_warn(std::string("OPCUA_COMMS_LOST_DEBOUNCE_MS=") + env + " is outside [0, " + + std::to_string(kMaxCommsLostDebounceMs) + "] ms - using " + std::to_string(clamped) + " ms"); + } + poller_config_.comms_lost_debounce = std::chrono::milliseconds(clamped); } else { log_warn(std::string("Ignoring invalid OPCUA_COMMS_LOST_DEBOUNCE_MS='") + env + "' (want a non-negative integer <= 3600000 ms)"); } } + if (config.contains("fault_service_timeout_ms")) { + const auto raw = config["fault_service_timeout_ms"]; + if (raw.is_number_integer()) { + const int64_t ms = raw.get(); + if (ms >= 100 && ms <= 600000) { + fault_service_timeout_ = std::chrono::milliseconds(ms); + } else { + log_warn("plugins.opcua.fault_service_timeout_ms out of range [100, 600000] - keeping " + + std::to_string(fault_service_timeout_.count()) + " ms"); + } + } else { + log_warn("plugins.opcua.fault_service_timeout_ms is not an integer - keeping " + + std::to_string(fault_service_timeout_.count()) + " ms"); + } + } + if (config.contains("comms_lost_severity")) { poller_config_.comms_lost_severity = config["comms_lost_severity"].get(); } @@ -594,6 +625,8 @@ void OpcuaPlugin::set_context(PluginContext & context) { if (node) { fault_clients_->report = node->create_client("/fault_manager/report_fault"); fault_clients_->clear = node->create_client("/fault_manager/clear_fault"); + fault_clients_->get_fault = node->create_client("/fault_manager/get_fault"); + context_ = node->get_node_base_interface()->get_context(); } run_startup_discovery(); @@ -614,10 +647,12 @@ void OpcuaPlugin::set_context(PluginContext & context) { } #endif - const bool connected = client_->connect(client_config_); + // The session is bound to the server it reached, read off the session itself. + // Nothing is bound yet on this path, so this only records the identity; the + // check it performs matters on every later connect. + const bool connected = client_->connect(client_config_) && bind_or_drop_session(); if (connected) { log_info("Connected to OPC-UA server: " + client_config_.endpoint_url); - clear_comms_lost_on_connect(); } else { log_warn("Failed to connect to OPC-UA server: " + client_config_.endpoint_url); } @@ -631,18 +666,29 @@ void OpcuaPlugin::set_context(PluginContext & context) { // consistent. Only in config-less mode: an explicit node map owns the name. if (node_map_path_.empty()) { const OpcuaClient::DeviceInfo info = connected ? client_->read_device_info() : OpcuaClient::DeviceInfo{}; + const bool has_nameplate = component_identity_has_nameplate(info); const ComponentIdentity ci = derive_component_identity(info, client_config_.endpoint_url); if (!ci.id.empty()) { + if (!has_nameplate) { + // A stand-in: the device said nothing, so the id names the endpoint. + // Recording it is what lets a fault reported under it be recognised as + // this bridge's once the device does name itself. + remember_placeholder_id(ci.id); + } node_map_.set_component_identity(ci.id, ci.name); log_info("Component identity derived from device: id='" + ci.id + "', name='" + ci.name + "'"); } - // Which session this identity speaks for. 0 when the connect failed: the id - // above then came from the fallback endpoint and an empty DeviceInfo, so it - // is provisional and the poll thread re-derives it on the first session - // (maybe_rederive_component_identity). Without that a gateway which started - // before its PLC would serve the placeholder for the life of the process, - // even after discovery adopted the real server. - component_identity_generation_ = connected ? client_->connection_generation() : 0; + // Which session this identity speaks for. Stamped only by a read that + // produced a nameplate, on the same rule the poll path follows, so a first + // session whose address space was still coming up is asked again rather + // than serving the stand-in until the next session. + if (connected && has_nameplate) { + component_identity_generation_ = client_->connection_generation(); + } + if (connected) { + component_identity_read_generation_ = client_->connection_generation(); + component_identity_poll_reads_ = 0; + } // Zero-config native A&C: with no node map and no explicit auto_alarms // block, subscribe the Server EventNotifier by default so discovered @@ -659,6 +705,13 @@ void OpcuaPlugin::set_context(PluginContext & context) { } } + // A session is up, so a link-state decision about PLC_COMMS_LOST is due. The + // decision itself waits for the poll thread, which owns the component ids the + // gate compares against. + if (connected) { + owe_comms_lost_decision(); + } + if (connected && node_map_.auto_browse_config().enabled) { // auto_browse needs a live session to walk the address space, so it can // only run here - after connect(), before the node map is consumed to @@ -700,6 +753,24 @@ void OpcuaPlugin::set_context(PluginContext & context) { poller_config_.report_sink_ready = [this]() { return fault_clients_->report && fault_clients_->report->service_is_ready(); }; + // Fired once per session, before the poller (re)subscribes and before the + // server replays a single condition. The config-less rename has to be settled + // by then: apply_condition_state pins a fault's entity at the first sighting + // of its ConditionId, so a rename afterwards would strand the replayed faults + // under the entity it drops. The poller takes its own copy of the alarm + // routing at subscribe time, i.e. straight after this returns, so the copy + // already carries the new name. + poller_config_.on_connected = [this]() { + // The session is checked against the binding before anything is derived + // from it. A session that reached a different server is dropped here, and + // the poller is told so, which keeps the standing outage and the endpoint + // exactly where they were. + if (!bind_or_drop_session()) { + return false; + } + maybe_rederive_component_identity(kIdentityConnectRetries); + return true; + }; // Config-less discovery with no configured endpoint: let the poller's // reconnect arm ask for a fresh scan while it is down. Without this the // startup scan is the only one that ever runs, so a gateway that scanned @@ -752,6 +823,25 @@ void OpcuaPlugin::shutdown() { if (poller_) { poller_->stop(); } + // The poll thread is joined, so nothing else drives the decision. Dropping the + // client releases the plugin's reference to it and takes the outstanding + // request out of its pending map; it does NOT recall a callback the executor + // has already taken off the wait set. Bumping the generation makes such a + // callback a no-op even while the plugin is alive, and once the plugin is + // destroyed the weak_ptr it holds stops locking at all - which is the part a + // mutex here could never provide. + comms_lost_decision_owed_.store(false); + comms_lost_probe_in_flight_ = false; + { + std::lock_guard lock(comms_lost_probe_state_->mutex); + ++comms_lost_probe_state_->generation; + comms_lost_probe_state_->answered = false; + comms_lost_probe_state_->sources.clear(); + } + if (fault_clients_ && fault_clients_->get_fault) { + fault_clients_->get_fault->remove_pending_request(comms_lost_probe_request_id_); + fault_clients_->get_fault.reset(); + } client_->disconnect(); log_info("OPC-UA plugin shutdown complete"); } @@ -1087,14 +1177,27 @@ void OpcuaPlugin::on_alarm_change(const std::string & entity_id, if (signal.active) { log_info("Alarm activated: " + signal.fault_code + " on " + entity_id); + if (clear_origin_for_signal(signal.fault_code) == ClearOrigin::LinkState) { + // A raise of the link-state code. The store can answer a probe from + // before it has processed this report, so the decision that follows earns + // one more (see comms_lost_raise_since_decision_). + comms_lost_raise_since_decision_.store(true); + } send_report_fault(entity_id, signal.fault_code, signal.severity, signal.message); } else { log_info("Alarm cleared: " + signal.fault_code + " on " + entity_id); - // The poller's own comms-lost clear on a successful reconnect is the same - // link-state event as the connect-time one. Every other code here is the - // device reporting its condition inactive, which is a real resolution and a - // one-shot edge, so it keeps the cascade and the buffer treats it as such. - send_clear_fault(signal.fault_code, clear_origin_for_signal(signal.fault_code)); + // The poller's comms-lost clear on a successful reconnect is the same + // link-state event as the connect-time one and takes the same route: the + // poller knows this link came back, not whose PLC_COMMS_LOST is standing in + // a store keyed by code alone, so the decision belongs to the poll thread + // and the gate. Every other code here is the device reporting its condition + // inactive, which is a real resolution and a one-shot edge, so it keeps the + // cascade and the buffer treats it as such. + if (clear_origin_for_signal(signal.fault_code) == ClearOrigin::LinkState) { + owe_comms_lost_decision(); + } else { + send_clear_fault(signal.fault_code, ClearOrigin::DeviceAlarm); + } } } @@ -1339,48 +1442,220 @@ void OpcuaPlugin::send_clear_fault(const std::string & fault_code, ClearOrigin o }}); } -void OpcuaPlugin::clear_comms_lost_on_connect() { +bool OpcuaPlugin::link_state_clear_permitted(bool fault_found, const std::vector & reporting_sources, + const std::unordered_set & my_ids) { + if (!fault_found || reporting_sources.empty()) { + return false; + } + return std::all_of(reporting_sources.begin(), reporting_sources.end(), [&my_ids](const std::string & source) { + return my_ids.count(source) > 0; + }); +} + +std::unordered_set OpcuaPlugin::reporting_ids_of_this_process() const { + std::unordered_set ids(placeholder_ids_.begin(), placeholder_ids_.end()); + // A node map that names no component_id reports under the NodeMap default, so + // a fault carrying it is this bridge's. + ids.insert(NodeMap::kDefaultComponentId); + const std::string current = node_map_.component_id(); + if (!current.empty()) { + ids.insert(current); + } + return ids; +} + +void OpcuaPlugin::owe_comms_lost_decision() { if (!poller_config_.comms_lost_fault_enabled) { return; } - // ClearFault is idempotent from this side: send_clear_fault is - // fire-and-forget, so a "Fault not found" answer for a code that was never - // raised costs nothing here and is the normal case on a healthy start. - // + comms_lost_decision_owed_.store(true); +} + +void OpcuaPlugin::apply_comms_lost_answer(bool found, const std::vector & reporting_sources) { + const std::unordered_set my_ids = reporting_ids_of_this_process(); + if (!link_state_clear_permitted(found, reporting_sources, my_ids)) { + std::string sources; + for (const auto & source : reporting_sources) { + if (!sources.empty()) { + sources += ", "; + } + sources += source; + } + log_info(std::string(kCommsLostFaultCode) + " is held by sources this bridge did not report under [" + sources + + "] - leaving it standing. A fault two bridges raised is cleared by an operator; the fault manager has " + "no per-source de-assert."); + return; + } // LinkState: this says the session came back, not that an operator resolved // anything, so a correlation rule naming PLC_COMMS_LOST as a root cause must - // not cascade-clear the symptoms the outage produced. It is also the one clear - // the next reconnect re-derives, so the pending buffer may drop it before - // anything one-shot. - log_info(std::string("OPC-UA connection established, clearing any standing ") + kCommsLostFaultCode); + // not cascade-clear the symptoms the outage produced. + log_info(std::string("OPC-UA connection established, clearing this bridge's ") + kCommsLostFaultCode); send_clear_fault(kCommsLostFaultCode, ClearOrigin::LinkState); } +void OpcuaPlugin::drive_comms_lost_decision() { + if (!poller_config_.comms_lost_fault_enabled || !fault_clients_->get_fault) { + return; + } + const bool connected = client_ && client_->is_connected(); + const uint64_t live_session = client_ ? client_->connection_generation() : 0; + + // An answer the executor thread parked, and only one belonging to the probe + // this thread is waiting for. Consuming it here keeps the gate's inputs - the + // component ids - on the thread that renames them. + bool found = false; + std::vector sources; + bool have_answer = false; + { + std::lock_guard lock(comms_lost_probe_state_->mutex); + if (comms_lost_probe_state_->answered && comms_lost_generation_current(comms_lost_probe_state_->answered_generation, + comms_lost_probe_state_->generation)) { + found = comms_lost_probe_state_->found; + sources.swap(comms_lost_probe_state_->sources); + comms_lost_probe_state_->answered = false; + have_answer = true; + } + } + if (have_answer) { + comms_lost_probe_in_flight_ = false; + if (!comms_lost_answer_applicable(connected, /*has_answer=*/true, comms_lost_probe_session_, live_session)) { + // The session the probe was sent on is gone. This tick is the first one + // after a reconnect, so the link reads as up while the answer describes + // the store from before an outage that has since been reported. It is + // dropped and the decision is taken again on this session. + comms_lost_decision_owed_.store(true); + RCLCPP_DEBUG(opcua_plugin_logger(), "%s answer belongs to an earlier session; asking again on this one", + kCommsLostFaultCode); + return; + } + apply_comms_lost_answer(found, sources); + // The store may have answered from before a ReportFault this process had + // already sent, so a decision with a raise behind it earns one more. + if (comms_lost_raise_since_decision_.exchange(false)) { + comms_lost_decision_owed_.store(true); + } + return; + } + + // A probe whose answer never came. Dropping the request is what stops the + // client's pending map from growing one entry per connect; bumping the + // generation is what stops a late answer being read as the next probe's. + if (comms_lost_probe_in_flight_) { + if (std::chrono::steady_clock::now() - comms_lost_probe_sent_ >= fault_service_timeout_) { + fault_clients_->get_fault->remove_pending_request(comms_lost_probe_request_id_); + // rclcpp takes the pending entry out before it runs the callback, and + // outside the mutex, so a callback that won that race is still on its way + // here. Moving the generation on is what makes it park at a stale one and + // be dropped. + { + std::lock_guard lock(comms_lost_probe_state_->mutex); + ++comms_lost_probe_state_->generation; + comms_lost_probe_state_->answered = false; + comms_lost_probe_state_->sources.clear(); + } + comms_lost_probe_in_flight_ = false; + comms_lost_decision_owed_.store(true); + log_warn(std::string("fault store did not answer within ") + std::to_string(fault_service_timeout_.count()) + + " ms; the " + kCommsLostFaultCode + " decision stays owed"); + } + return; + } + + if (!comms_lost_probe_due(comms_lost_decision_owed_.load(), connected, comms_lost_probe_in_flight_, + fault_clients_->get_fault->service_is_ready())) { + return; + } + // Taken BEFORE the send: an owe raised by the REST or event-pump thread while + // the request is going out has to survive it, and a store(false) after the + // send would swallow it. + if (!comms_lost_decision_owed_.exchange(false)) { + return; + } + + uint64_t generation = 0; + { + std::lock_guard lock(comms_lost_probe_state_->mutex); + generation = ++comms_lost_probe_state_->generation; + comms_lost_probe_state_->answered = false; + comms_lost_probe_state_->sources.clear(); + } + std::weak_ptr weak_state = comms_lost_probe_state_; + auto request = std::make_shared(); + request->fault_code = kCommsLostFaultCode; + try { + auto future = fault_clients_->get_fault->async_send_request( + request, [weak_state, generation](rclcpp::Client::SharedFuture answer) { + // Executor thread. It parks the answer and nothing else: the decision + // reads component ids, which only the poll thread may do. The state is + // reached through a weak_ptr, so a callback the executor had already + // taken when the plugin went away touches nothing. + const auto state = weak_state.lock(); + if (!state) { + return; + } + try { + const auto response = answer.get(); + std::lock_guard lock(state->mutex); + if (state->generation != generation) { + return; // answer to a probe that has already been given up on + } + state->found = response->success; + state->sources = response->fault.reporting_sources; + state->answered_generation = generation; + state->answered = true; + ++state->answers; + } catch (...) { + // A broken promise (the client was reset under us) leaves the + // decision owed, which the poll thread resolves on its own terms. + } + }); + comms_lost_probe_request_id_ = future.request_id; + comms_lost_probe_sent_ = std::chrono::steady_clock::now(); + comms_lost_probe_session_ = live_session; + comms_lost_probe_in_flight_ = true; + } catch (const std::exception & e) { + // The owe was taken before the send. Nothing went out, so it goes back: + // the poll thread has no catch of its own and a lost owe is a fault left + // standing for good. + comms_lost_decision_owed_.store(true); + RCLCPP_WARN(opcua_plugin_logger(), "fault store request failed to go out (%s); the %s decision stays owed", + e.what(), kCommsLostFaultCode); + } +} + OpcuaPlugin::PendingEnqueueOutcome OpcuaPlugin::enqueue_pending_dispatch(std::vector & buffer, size_t max_size, PendingFaultDispatch entry) { const bool is_clear = entry.kind == PendingFaultDispatch::Kind::Clear; - // At most one pending clear per fault code. A repeat moves to the BACK rather - // than overwriting in place, so an interleaved report-then-clear for the same - // code still flushes in the order the PLC produced it. + // At most one pending clear per fault code, and only while nothing has been + // reported for that code since. Search from the BACK and stop at the first + // entry carrying this code: a Clear there is superseded and moves to the back, + // a Report there means the code was raised again and the incoming clear + // belongs behind it. Coalescing past that Report would reorder the pair - + // Report, Clear, Report, Clear flushed as Report, Report, Clear, leaving the + // second raise standing while the device says inactive. Repeated reconnect + // clears, which have no report between them, still collapse to one. bool replaced = false; if (is_clear) { - const auto same_code = std::find_if(buffer.begin(), buffer.end(), [&entry](const PendingFaultDispatch & pending) { - return pending.kind == PendingFaultDispatch::Kind::Clear && pending.fault_code == entry.fault_code; - }); - if (same_code != buffer.end()) { - buffer.erase(same_code); - replaced = true; + for (auto it = buffer.rbegin(); it != buffer.rend(); ++it) { + if (it->fault_code != entry.fault_code) { + continue; + } + if (it->kind == PendingFaultDispatch::Kind::Clear) { + buffer.erase(std::next(it).base()); + replaced = true; + } + break; } } PendingEnqueueOutcome outcome = replaced ? PendingEnqueueOutcome::ReplacedClear : PendingEnqueueOutcome::Buffered; if (buffer.size() >= max_size) { // Only a link-state clear is re-derivable: the next reconnect sends it - // again. A full buffer gives that up first, and refuses an incoming one - // rather than pushing out a dispatch nothing will re-send. Everything else - - // reports, a device alarm's inactive edge, an operator's scoped clear - is - // one-shot and ages out oldest-first. + // again. A full buffer gives that up first, and refuses an incoming one, so + // every dispatch nothing will re-send stays. Everything else - reports, a + // device alarm's inactive edge, an operator's scoped clear - is one-shot and + // ages out oldest-first. const auto oldest_link_state = std::find_if(buffer.begin(), buffer.end(), [](const PendingFaultDispatch & pending) { return pending.kind == PendingFaultDispatch::Kind::Clear && pending.link_state; }); @@ -1401,8 +1676,10 @@ OpcuaPlugin::PendingEnqueueOutcome OpcuaPlugin::enqueue_pending_dispatch(std::ve void OpcuaPlugin::send_or_buffer(PendingFaultDispatch entry) { // Bound the buffer so a deployment with no fault_manager cannot grow it - // without limit. Runs on both the poll thread and the REST clear_fault thread, - // so the vector mutation is serialised by pending_reports_mutex_. + // without limit. Reached from the poll thread (alarm and event callbacks, and + // the link-state decision in publish_values) and from the REST thread + // (FaultProvider::clear_fault), so the vector mutation is serialised by + // pending_reports_mutex_. PendingEnqueueOutcome outcome = PendingEnqueueOutcome::Buffered; { std::lock_guard lock(pending_reports_mutex_); @@ -1418,6 +1695,13 @@ void OpcuaPlugin::send_or_buffer(PendingFaultDispatch entry) { log_warn("pending fault dispatch buffer full of one-shot dispatches (" + std::to_string(kMaxPendingDispatches) + "), dropping this link-state clear instead"); } + if (outcome == PendingEnqueueOutcome::EvictedLinkStateClear || outcome == PendingEnqueueOutcome::Refused) { + // The buffer ranks a link-state clear last on the grounds that the next + // reconnect re-derives it, which holds for the poller's reconnect clear and + // not for one a successful connect queued. Recording the debt is what + // covers the connect case; the poll thread pays it. + owe_comms_lost_decision(); + } // Drains immediately (in order) if the sink is already matched. flush_pending_reports(); } @@ -1435,9 +1719,6 @@ void OpcuaPlugin::flush_pending_reports() { std::vector batch; { std::lock_guard lock(pending_reports_mutex_); - if (pending_reports_.empty()) { - return; - } batch.swap(pending_reports_); } for (auto & entry : batch) { @@ -1501,17 +1782,54 @@ void OpcuaPlugin::run_auto_browse() { (result.depth_cap_hit ? " [depth cap reached on at least one branch]" : "")); } -std::optional OpcuaPlugin::rederived_component_identity(const std::string & current_id, +std::optional OpcuaPlugin::rederived_component_identity(const ComponentIdentityState & state, const OpcuaClient::DeviceInfo & info, const std::string & endpoint_url) { const ComponentIdentity ci = derive_component_identity(info, endpoint_url); - if (ci.id.empty() || ci.id == current_id) { + if (ci.id.empty() || ci.id == state.current_id) { return std::nullopt; } - return ci; + if (component_identity_has_nameplate(info)) { + // A name the device gave itself takes over from a stand-in. A second + // nameplate is a different device answering on the same endpoint: that is a + // decision about asset identity, so the served name keeps its place and the + // read does not move it. + return state.current_is_placeholder ? std::optional(ci) : std::nullopt; + } + // No nameplate: the derivation is the neutral opcua- stand-in. It + // follows the endpoint from one stand-in to the next, which is the adoption + // case, and leaves a device-given name alone - read_device_info() answers + // empty or partial on the first read of a session whose address space is + // still coming up, and one such read would otherwise move the component URL, + // the _alarms entity and every fault filed under them onto a stand-in. + return state.current_is_placeholder ? std::optional(ci) : std::nullopt; } -void OpcuaPlugin::maybe_rederive_component_identity() { +OpcuaPlugin::IdentityReadOutcome +OpcuaPlugin::identity_reads(int extra_reads, int reads_spent, int max_reads, + const std::function & read_fn, + const std::function & pause_fn) { + IdentityReadOutcome outcome; + outcome.reads_spent = reads_spent; + for (int attempt = 0; attempt <= extra_reads; ++attempt) { + if (outcome.reads_spent >= max_reads) { + return outcome; + } + if (attempt > 0 && pause_fn && !pause_fn(kIdentityConnectRetryPause)) { + outcome.interrupted = true; + return outcome; + } + outcome.info = read_fn(); + ++outcome.reads_spent; + outcome.read_made = true; + if (component_identity_has_nameplate(outcome.info)) { + return outcome; + } + } + return outcome; +} + +void OpcuaPlugin::maybe_rederive_component_identity(int extra_reads) { // An explicit node map owns the component name. Only the config-less path // derives it from the device. if (!node_map_path_.empty() || !client_ || !client_->is_connected()) { @@ -1521,16 +1839,77 @@ void OpcuaPlugin::maybe_rederive_component_identity() { if (generation == component_identity_generation_) { return; // identity already speaks for this session } + if (generation != component_identity_read_generation_) { + component_identity_read_generation_ = generation; + component_identity_poll_reads_ = 0; + } const std::string live_endpoint = client_->endpoint_url(); - const auto rederived = - rederived_component_identity(node_map_.component_id(), client_->read_device_info(), live_endpoint); - component_identity_generation_ = generation; + // The connect burst carries its own budget (one read plus kIdentityConnectRetries + // attempts); the poll path spends kMaxIdentityPollReads across the session. + const bool is_connect_burst = extra_reads > 0; + const int budget = is_connect_burst ? extra_reads + 1 : kMaxIdentityPollReads; + const int spent = is_connect_burst ? 0 : component_identity_poll_reads_; + const auto outcome = identity_reads( + extra_reads, spent, budget, + [this]() { + return client_->read_device_info(); + }, + [this](std::chrono::milliseconds pause) { + // A shutdown must not wait the burst out: the gateway calls shutdown() + // once its executor has returned, and the poll thread is what stop() + // joins. + if (shutdown_requested_.load()) { + return false; + } + std::this_thread::sleep_for(pause); + return !shutdown_requested_.load(); + }); + if (!is_connect_burst) { + component_identity_poll_reads_ = outcome.reads_spent; + } + if (outcome.interrupted) { + // A shutdown ended the burst. Renaming now would take the node-map lock and + // re-pin conditions on the way out, for an identity the process is not + // going to serve. + return; + } + if (!outcome.read_made) { + // The poll budget for this session is spent. Stamping the session stops the + // reads; the stand-in stands until the next one, when the question is asked + // again. + if (!is_connect_burst) { + component_identity_generation_ = generation; + } + return; + } + const bool has_nameplate = component_identity_has_nameplate(outcome.info); + + ComponentIdentityState state; + state.current_id = node_map_.component_id(); + state.current_is_placeholder = + std::find(placeholder_ids_.begin(), placeholder_ids_.end(), state.current_id) != placeholder_ids_.end(); + + const auto rederived = rederived_component_identity(state, outcome.info, live_endpoint); + if (has_nameplate) { + // The device named itself, so this session's identity is settled. A read + // with no nameplate leaves the stamp alone: it is the read that is + // provisional, not the session, and the address space of a PLC that has + // just come up fills in over the following seconds. Stamping it would pin + // the stand-in until the NEXT session. + component_identity_generation_ = generation; + } if (!rederived) { return; } - const std::string previous_id = node_map_.component_id(); + const std::string previous_id = state.current_id; + const std::string previous_alarms_entity = previous_id + "_alarms"; + if (!has_nameplate) { + // The adopted id is another stand-in, so it joins the list a fault reported + // under it is recognised by. + remember_placeholder_id(rederived->id); + } { // Serialize against the REST read paths, which hold references into // node_map_ (entity_defs) while they answer. @@ -1539,7 +1918,7 @@ void OpcuaPlugin::maybe_rederive_component_identity() { // default-derived one has to follow the rename. An operator-chosen entity_id // does not match the derived form and is left alone. auto & auto_alarms = node_map_.mutable_auto_alarms(); - if (auto_alarms.entity_id == previous_id + "_alarms") { + if (auto_alarms.entity_id == previous_alarms_entity) { auto_alarms.entity_id.clear(); } node_map_.set_component_identity(rederived->id, rederived->name); @@ -1547,10 +1926,28 @@ void OpcuaPlugin::maybe_rederive_component_identity() { // reference to the component id moves together. node_map_.finalize_auto_alarms_overlay(); } + if (poller_) { + // Two things in the poller carry the old entity id: the copy of the alarm + // config the event path routes new conditions with, and the conditions it + // already pinned. Both move here. A condition that changes state between + // the pin and this call is reported once under the old entity. + poller_->refresh_alarm_routing(); + poller_->repin_auto_alarms_entity(previous_alarms_entity, node_map_.auto_alarms().entity_id); + } log_info("Component identity re-derived from the adopted device at " + live_endpoint + ": id='" + rederived->id + "', name='" + rederived->name + "' (was '" + previous_id + "')"); } +void OpcuaPlugin::remember_placeholder_id(const std::string & id) { + if (std::find(placeholder_ids_.begin(), placeholder_ids_.end(), id) != placeholder_ids_.end()) { + return; // already one of this process's stand-ins; it keeps its place + } + if (placeholder_ids_.size() >= kMaxPlaceholderIds) { + return; // the first kMaxPlaceholderIds distinct stand-ins are what is kept + } + placeholder_ids_.push_back(id); +} + void OpcuaPlugin::maybe_rebrowse_on_reconnect() { if (!node_map_.auto_browse_config().enabled || !client_ || !client_->is_connected()) { return; @@ -1569,11 +1966,16 @@ void OpcuaPlugin::publish_values(const PollSnapshot & snap) { // Poll-thread hook: drain any fault reports buffered before fault_manager was // discovered, so a late sink still receives them. flush_pending_reports(); - // Poll-thread hook: re-derive the config-less component identity from the - // device once a session is up, so an adopted PLC stops being served under the - // provisional endpoint-derived id. Runs BEFORE the re-walk below, which - // rebuilds entity_defs off the component id. - maybe_rederive_component_identity(); + // Poll-thread hook: one step of the owed link-state decision about + // PLC_COMMS_LOST. Here, because the gate compares against component ids this + // thread is the only writer of. + drive_comms_lost_decision(); + // Poll-thread retry for an identity that is not settled yet. The connect hook + // is where the rename normally happens, ahead of the subscribe; this call + // covers a device-info read that came back with no nameplate because the PLC's + // address space was still filling in. It is a no-op from the moment the device + // names itself, and from the moment kMaxIdentityPollReads is spent. + maybe_rederive_component_identity(/*extra_reads=*/0); // Poll-thread hook: (re)run auto_browse after a fresh session so a PLC that // came up (or restarted) after the initial connect still gets walked. maybe_rebrowse_on_reconnect(); @@ -1702,7 +2104,9 @@ std::optional OpcuaPlugin::rescan_step(int interval_s, // start would make the next one due the moment this one returned - the poll // thread would sweep back to back and only attempt a reconnect once a sweep. // A sweep that threw still consumed that time, so the stamp is owed either - // way, or the next poll iteration would start another one immediately. + // way, or the next poll iteration would start another one immediately. The + // rethrow is what preserves the caller's choice; the poll thread has no try of + // its own, so the caller that runs on it is rescan_guarded. try { const auto result = sweep(); *last_scan_end = now(); @@ -1713,10 +2117,32 @@ std::optional OpcuaPlugin::rescan_step(int interval_s, } } +std::optional +OpcuaPlugin::rescan_guarded(int interval_s, const std::function & now, + std::chrono::steady_clock::time_point * last_scan_end, + const std::function()> & sweep) { + // A handler reached by bad_alloc allocates nothing on our side: the log goes + // through a fixed format string, and the catch-all arm through a literal. + try { + return rescan_step(interval_s, now, last_scan_end, sweep); + } catch (const std::exception & e) { + RCLCPP_WARN(opcua_plugin_logger(), "OPC-UA discovery: rescan sweep failed (%s); the endpoint stands", e.what()); + } catch (...) { + RCLCPP_WARN(opcua_plugin_logger(), + "OPC-UA discovery: rescan sweep failed with a non-standard exception; the endpoint stands"); + } + return std::nullopt; +} + std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryConfig & config, bool endpoint_configured, const PortScanFn & scan, const IdentifyFn & identify, const DiscoveryReporter & reporter, - const std::function & cancelled) { + const std::function & cancelled, + std::string * application_uri, + const std::string & bound_application_uri) { + if (application_uri != nullptr) { + application_uri->clear(); + } if (!config.enabled) { return std::nullopt; } @@ -1726,9 +2152,9 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo return std::nullopt; } - // The pass reports through a buffer rather than straight to the log: whether - // the report is operator-visible or a DEBUG trace depends on the outcome, - // which is only known once the sweep is done. A rescan runs for the life of a + // The pass reports through a buffer, because whether the report is + // operator-visible or a DEBUG trace depends on the outcome, and the outcome is + // only known once the sweep is done. A rescan runs for the life of a // disconnected process, so an unchanged outcome must not repeat its whole // report (a secured-only site would log the same WARN every interval_s). struct ReportLine { @@ -1772,10 +2198,10 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo for (const auto & s : subnets) { subnet_list += (subnet_list.empty() ? "" : ", ") + s; } - // The announcement goes out NOW, not through the buffered report: a sweep of a + // The announcement goes out NOW, ahead of the buffered report: a sweep of a // wide subnet runs for minutes, and an operator watching start-up has to see - // that the gateway is scanning rather than hung. It says what the pass is - // about to do, not what it found, so it stays out of the outcome digest and is + // that the gateway is scanning. It says what the pass is about to do, which is + // known before the outcome is, so it stays out of the outcome digest and is // levelled on its own - the first pass announces at INFO, a rescan at DEBUG so // the recurring sweep does not repeat it every interval. const bool first_pass = reporter.previous_outcome == nullptr || reporter.previous_outcome->empty(); @@ -1817,17 +2243,31 @@ std::optional OpcuaPlugin::discover_endpoint(const OpcuaDiscoveryCo std::to_string(discovery_servers) + " discovery server(s)/LDS, " + std::to_string(secured_only) + " secured-only (need credentials), " + std::to_string(leads) + " non-OPC-UA/unidentified lead(s)."); - const DiscoveredEndpoint * chosen = NetworkDiscovery::select_auto_endpoint(found, config.anonymous_none_only); + const DiscoveredEndpoint * chosen = + NetworkDiscovery::select_auto_endpoint(found, config.anonymous_none_only, bound_application_uri); if (chosen == nullptr) { - warn_line( - "OPC-UA discovery: no auto-connectable None/Anonymous data server found, leaving the endpoint unchanged. " - "Secured-only servers require operator credentials."); + if (!bound_application_uri.empty()) { + warn_line("OPC-UA discovery: the server this bridge is bound to (uri='" + bound_application_uri + + "') is not on the scanned network, leaving the endpoint unchanged."); + } else { + warn_line( + "OPC-UA discovery: no auto-connectable None/Anonymous data server found, leaving the endpoint unchanged. " + "Secured-only servers require operator credentials."); + } emit(); return std::nullopt; } info_line("OPC-UA discovery: selected endpoint " + chosen->endpoint_url + " (uri='" + chosen->application_uri + "')"); + if (chosen->application_uri.empty()) { + warn_line("OPC-UA discovery: " + chosen->endpoint_url + + " reports no ApplicationUri. This bridge cannot bind to it, so after a drop a re-scan accepts whichever " + "server answers."); + } emit(); + if (application_uri != nullptr) { + *application_uri = chosen->application_uri; + } return chosen->endpoint_url; } @@ -1860,9 +2300,9 @@ void OpcuaPlugin::run_startup_discovery() { // The startup scan can legitimately find nothing - a gateway that boots // alongside its PLC routinely scans while the PLC is still coming up. With a // cadence the endpoint stays at its default and the reconnect arm rescans, - // so this is a delay rather than a dead end. With re-scanning switched off - // (an explicit interval_s: 0) it IS the end, and the operator has to be told - // which of the two they configured. + // so the PLC is still adopted once it answers. With re-scanning switched off + // (an explicit interval_s: 0) this scan IS the last one, and the operator + // has to be told which of the two they configured. const int startup_interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); if (startup_interval_s > 0) { log_info("OPC-UA discovery: startup scan selected no endpoint. The reconnect loop rescans every " + @@ -1881,10 +2321,13 @@ void OpcuaPlugin::run_startup_discovery() { } bool OpcuaPlugin::discovery_cancelled() const { - // rclcpp::ok() only reads the default context's atomic shutdown flag, so it is - // safe to call from the set_context thread and the poll thread alike. The rule + // rclcpp::ok() only reads a context's atomic shutdown flag, so it is safe to + // call from the set_context thread and the poll thread alike. Which context is + // the point: the gateway node's own, not the process-wide default, because a + // host that built its node on a private context leaves the default one + // uninitialised and every sweep would cancel before it started. The rule // itself, and why both signals are needed, lives in discovery_cancelled_for. - return discovery_cancelled_for(shutdown_requested_.load(), rclcpp::ok()); + return discovery_cancelled_for_context(shutdown_requested_.load(), context_); } OpcuaPlugin::DiscoveryReporter OpcuaPlugin::discovery_reporter(std::string * previous_outcome) const { @@ -1902,6 +2345,50 @@ OpcuaPlugin::DiscoveryReporter OpcuaPlugin::discovery_reporter(std::string * pre return reporter; } +bool OpcuaPlugin::note_refused_application_uri(const std::string & uri) { + if (std::find(refused_application_uris_.begin(), refused_application_uris_.end(), uri) != + refused_application_uris_.end()) { + return false; + } + refused_application_uris_.push_back(uri); + if (refused_application_uris_.size() > kMaxRefusedApplicationUris) { + refused_application_uris_.erase(refused_application_uris_.begin()); + } + return true; +} + +bool OpcuaPlugin::bind_or_drop_session() { + if (!client_ || !client_->is_connected()) { + return false; + } + const std::string live_uri = client_->read_server_application_uri(); + + if (bound_application_uri_.empty()) { + // Nothing bound yet: this session names the server every later sweep looks + // for. A server that publishes no ApplicationUri leaves the binding empty + // and stays unbindable, which the discovery report already says. + bound_application_uri_ = live_uri; + refused_application_uris_.clear(); + return true; + } + + if (live_uri == bound_application_uri_) { + // Bound server reached. A later outage reports a foreign server again. + refused_application_uris_.clear(); + return true; + } + + binding_mismatch_disconnects_.fetch_add(1); + if (note_refused_application_uri(live_uri)) { + log_warn("OPC-UA: " + client_->endpoint_url() + " answers as uri='" + live_uri + + "', which is not the server this bridge is bound to (uri='" + bound_application_uri_ + + "'). Dropping the session and keeping the standing outage. Replacing a PLC is a recommissioning: " + "restart the plugin against the new one."); + } + client_->disconnect(); + return false; +} + std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { // A sweep is a bounded but multi-second blocking call on the poll thread, and // stop() has to wait for whatever it is in the middle of. Do not start one the @@ -1911,24 +2398,41 @@ std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { } const int interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); - const auto chosen = rescan_step( + std::string candidate_uri; + const auto chosen = rescan_guarded( interval_s, []() { return std::chrono::steady_clock::now(); }, &last_discovery_scan_end_, - [this]() { - return discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, - discovery_reporter(&last_discovery_outcome_), [this]() { - return discovery_cancelled(); - }); + [this, &candidate_uri]() { + return discover_endpoint( + discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, + discovery_reporter(&last_discovery_outcome_), + [this]() { + return discovery_cancelled(); + }, + &candidate_uri, bound_application_uri_); }); // The live client config, not client_config_: this runs on the poll thread // and client_config_ is read by the refresh thread in introspect(). The // client owns the endpoint once connect() has been called with it, and its // accessors are mutex-guarded. const std::string current = client_ ? client_->endpoint_url() : client_config_.endpoint_url; - if (!chosen || *chosen == current) { + if (!chosen) { + // Selection had the binding and found nothing carrying it, so the server + // this bridge polls is not answering anywhere on the scanned network. The + // endpoint stands and the outage with it; the connect the reconnect arm + // keeps attempting is the one that recovers when the server comes back. + if (!bound_application_uri_.empty()) { + rescan_refusals_.fetch_add(1); + } + return std::nullopt; + } + if (*chosen == current) { + // The address did not move. A DIFFERENT server answering at this address is + // caught by the connect that follows (bind_or_drop_session), which is where + // the session's own identity is read. return std::nullopt; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp index 019d0604c..b6a5801d7 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp @@ -188,6 +188,35 @@ OpcuaPoller::~OpcuaPoller() { stop(); } +void OpcuaPoller::refresh_alarm_routing() { + auto routing = std::make_shared(); + routing->event_alarms = node_map_.event_alarms(); + routing->auto_alarms = node_map_.auto_alarms(); + for (const auto & entry : node_map_.entries()) { + routing->entity_by_node_id.emplace(entry.node_id_str, entry.entity_id); + } + std::lock_guard lock(alarm_routing_mutex_); + alarm_routing_ = std::move(routing); +} + +std::shared_ptr OpcuaPoller::alarm_routing() const { + std::lock_guard lock(alarm_routing_mutex_); + return alarm_routing_; +} + +void OpcuaPoller::repin_auto_alarms_entity(const std::string & old_entity_id, const std::string & new_entity_id) { + if (old_entity_id.empty() || new_entity_id.empty() || old_entity_id == new_entity_id) { + return; + } + std::unique_lock lock(conditions_mutex_); + for (auto & [condition_id, runtime] : conditions_) { + (void)condition_id; + if (runtime.entity_id == old_entity_id) { + runtime.entity_id = new_entity_id; + } + } +} + void OpcuaPoller::start(const PollerConfig & config) { if (running_.load()) { return; @@ -196,16 +225,26 @@ void OpcuaPoller::start(const PollerConfig & config) { config_ = config; running_ = true; - // Try subscription mode first - if (config_.prefer_subscriptions) { - setup_subscriptions(); - } + // Same position as the reconnect arm's: whatever the owner must settle before + // the link-state edge and before the server replays a condition happens here. + // Only meaningful with a session already up - start() on a disconnected + // client subscribes nothing, and the reconnect arm fires the hook when one + // appears. A hook that rejects the session leaves the poll loop to its + // reconnect arm. + const bool session_accepted = !client_.is_connected() || !config_.on_connected || config_.on_connected(); + + if (session_accepted) { + // Try subscription mode first + if (config_.prefer_subscriptions) { + setup_subscriptions(); + } - // Issue #386: subscribe to native AlarmConditionType events. Independent - // of data-change subscriptions; runs whenever event_alarms and/or - // auto_alarms are configured. - if (has_alarm_sources()) { - setup_event_subscriptions(); + // Issue #386: subscribe to native AlarmConditionType events. Independent + // of data-change subscriptions; runs whenever event_alarms and/or + // auto_alarms are configured. + if (has_alarm_sources()) { + setup_event_subscriptions(); + } } // Start poll/reconnect thread regardless (handles reconnection and poll fallback) @@ -381,7 +420,14 @@ void OpcuaPoller::setup_event_subscriptions() { event_monitored_item_ids_.clear(); - for (const auto & cfg : effective_alarm_sources(node_map_.event_alarms(), node_map_.auto_alarms())) { + // Subscribe time is when the event path's copy of the alarm configuration is + // taken, so the monitored items and the routing ``on_event`` will use come + // from one state of the node map. Anything the owner renames must therefore + // be renamed before here - which is what PollerConfig::on_connected is for. + refresh_alarm_routing(); + const auto routing = alarm_routing(); + + for (const auto & cfg : effective_alarm_sources(routing->event_alarms, routing->auto_alarms)) { // Per-source select specs so each source can carry its own associated // values (issue #389) in addition to the fixed alarm-state fields. const auto select_specs = build_alarm_event_select_specs(cfg); @@ -533,8 +579,15 @@ void OpcuaPoller::read_fallback_replay() { // before this path can be relied on there (use ConditionRefresh on Siemens). std::set seen; std::set failed_sources; - const auto & auto_cfg = node_map_.auto_alarms(); - for (const auto & cfg : effective_alarm_sources(node_map_.event_alarms(), node_map_.auto_alarms())) { + // The same copy the live event path reads, so a replayed condition and a live + // one are routed identically. Taken by setup_event_subscriptions() just above + // this call, on this thread. + const auto routing = alarm_routing(); + if (!routing) { + return; + } + const AutoAlarmsConfig & auto_cfg = routing->auto_alarms; + for (const auto & cfg : effective_alarm_sources(routing->event_alarms, routing->auto_alarms)) { bool scan_ok = false; auto conditions = client_.read_source_conditions(cfg.source_node_id, &scan_ok); if (!scan_ok) { @@ -625,8 +678,8 @@ void OpcuaPoller::read_fallback_replay() { } eff.fault_code = NodeMap::derive_auto_fault_code(snap.condition_name, /*source_name=*/"", cfg.source_node_id_str, /*event_type_str=*/"", snap.message); - const auto * known_entry = node_map_.find_by_node_id(cfg.source_node_id_str); - eff.entity_id = known_entry != nullptr ? known_entry->entity_id : auto_cfg.entity_id; + const auto known_entry = routing->entity_by_node_id.find(cfg.source_node_id_str); + eff.entity_id = known_entry != routing->entity_by_node_id.end() ? known_entry->second : auto_cfg.entity_id; eff.severity_override = NodeMap::map_auto_severity(snap.severity, auto_cfg.severity_bands); eff.message_override.clear(); if (auto_cfg.auto_clear) { @@ -917,7 +970,14 @@ void OpcuaPoller::on_event(const AlarmEventConfig & cfg, const std::vectorauto_alarms; if (resolved.matched) { // Build the effective config for this specific event (resolved // fault_code + overrides) so apply_condition_state tracks the right @@ -947,7 +1007,7 @@ void OpcuaPoller::on_event(const AlarmEventConfig & cfg, const std::vectorevent_alarms) { if (node_ids_equivalent(explicit_cfg.source_node_id_str, cfg.source_node_id_str)) { continue; // this monitored item's own source (shared-source case) } @@ -971,8 +1031,8 @@ void OpcuaPoller::on_event(const AlarmEventConfig & cfg, const std::vector_alarms" - a separate App, not the PLC root Component; // see AutoAlarmsConfig::entity_id). - const auto * known_entry = node_map_.find_by_node_id(source_node_str); - eff.entity_id = known_entry != nullptr ? known_entry->entity_id : auto_cfg.entity_id; + const auto known_entry = routing->entity_by_node_id.find(source_node_str); + eff.entity_id = known_entry != routing->entity_by_node_id.end() ? known_entry->second : auto_cfg.entity_id; eff.severity_override = NodeMap::map_auto_severity(severity, auto_cfg.severity_bands); eff.message_override.clear(); // description = the raw event Message, verbatim if (auto_cfg.auto_clear) { @@ -1204,14 +1264,24 @@ void OpcuaPoller::poll_loop() { reconnect_wait = config_.reconnect_interval; } - if (client_.connect(reconnect_config)) { + if (client_.connect(reconnect_config) && (!config_.on_connected || config_.on_connected())) { + // The hook runs before the link-state edge and before anything is + // (re)subscribed: the adopted session is the one that can finally name + // the device, the owner's decision about the standing PLC_COMMS_LOST is + // taken against the ids it settles on, and the ConditionRefresh burst + // that follows the event subscribe pins every replayed condition's + // entity at first sight. A hook that rejects the session has dropped it + // already, so this arm falls through to the backoff with the outage + // untouched. See PollerConfig::on_connected. reconnect_wait = config_.reconnect_interval; // reset on success - // Issue #496: connection restored - clear the comms-lost fault. Sent on - // EVERY successful reconnect, not only when this process raised it: the - // fault manager keys faults by fault_code and persists them, so a fault - // raised before a restart is standing in the store with nothing in - // memory to remember it. The clear is fire-and-forget and the store - // answers "not found" harmlessly when there is nothing to clear. + // Issue #496: connection restored - hand the link-state edge to the + // owner on EVERY successful reconnect, whatever this process remembers + // raising. The fault manager keys faults by fault_code and persists + // them, so a fault raised before a restart is standing in the store with + // nothing in memory to remember it. What is standing and who reported it + // is a question for the owner's callback (see PollerConfig and + // OpcuaPlugin's link-state decision), which is where the clear is + // decided. if (config_.comms_lost_fault_enabled) { emit_comms_lost(/*active=*/false); } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp index e49b9a365..f6c9332df 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/fixtures/test_alarm_server/test_alarm_server.cpp @@ -760,6 +760,7 @@ int main(int argc, char ** argv) { bool secure = false; std::string cert_path, key_path, trust_path, username = "medkit", password = "secret"; std::string app_uri = "urn:test:alarms:server"; + bool app_uri_explicit = false; std::string di_serial = "SN-0001-TEST"; UA_UInt32 max_refs_per_node = 0; // 0 = server default (unlimited) for (int i = 1; i < argc; ++i) { @@ -782,6 +783,7 @@ int main(int argc, char ** argv) { trust_path = argv[++i]; } else if (std::strcmp(argv[i], "--app-uri") == 0 && i + 1 < argc) { app_uri = argv[++i]; + app_uri_explicit = true; } else if (std::strcmp(argv[i], "--username") == 0 && i + 1 < argc) { username = argv[++i]; } else if (std::strcmp(argv[i], "--password") == 0 && i + 1 < argc) { @@ -807,6 +809,14 @@ int main(int argc, char ** argv) { #endif } else { UA_ServerConfig_setMinimal(config, port, nullptr); + if (app_uri_explicit) { + // The server identity discovery reads and the plugin binds to. ``--app-uri`` + // gives a second fixture a second identity, so a test that needs two + // servers on one host - each on its own port - can stand for two PLCs. + // Left at open62541's own default when no test names one. + UA_String_clear(&config->applicationDescription.applicationUri); + config->applicationDescription.applicationUri = UA_STRING_ALLOC(app_uri.c_str()); + } } set_build_info(config); @@ -850,8 +860,9 @@ int main(int argc, char ** argv) { std::cout << "READY port=" << port << " namespace=" << ns << " secure=" << (secure ? "true" : "false") << std::endl; std::thread reader(stdin_reader_loop); - // The server is driven by hand rather than by UA_Server_run so that queued - // commands execute between iterations, on this thread. run_iterate is called + // The server is driven by hand so that queued commands execute between + // iterations, on this thread, which UA_Server_run offers no point for. + // run_iterate is called // with waitInternal false and the loop paced by a short sleep: blocking // inside the server would hold a command back until the next scheduled // callback, and a command is how a test makes the alarm it is waiting for diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp index 7ef7c0e4f..99713f5d2 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_network_discovery.cpp @@ -536,6 +536,48 @@ TEST(SelectAutoEndpoint, LdsNeverSelectedEvenWhenAnonymousRelaxed) { EXPECT_EQ(NetworkDiscovery::select_auto_endpoint(eps, false), nullptr); } +TEST(SelectAutoEndpoint, ABoundBridgeLooksForItsOwnServerWhateverItsAddress) { + // A bridge already holding a session names the server it wants. Address order + // decides nothing then: a foreign server that sorts lower would otherwise win + // every sweep and the bound one would never be reached again. + const std::string mine = "urn:siemens:s7-1500:line-a"; + const std::string other = "urn:beckhoff:cx5140:line-b"; + + DiscoveredEndpoint foreign; + foreign.ip = "192.168.1.10"; // sorts FIRST + foreign.port = 4840; + foreign.protocol = "opcua"; + foreign.application_type = 0; + foreign.anonymous_none_available = true; + foreign.application_uri = other; + DiscoveredEndpoint bound = foreign; + bound.ip = "192.168.1.50"; + bound.application_uri = mine; + + std::vector eps = {foreign, bound}; + const auto * chosen = NetworkDiscovery::select_auto_endpoint(eps, true, mine); + ASSERT_NE(chosen, nullptr) << "the bound server is on the network and was not selected"; + EXPECT_EQ(chosen->application_uri, mine) + << "address order decided the selection, so a lower-sorting foreign server blocks the bound one for good"; + EXPECT_EQ(chosen->ip, "192.168.1.50"); + + // The bound server is not answering anywhere: nothing is selected, so the + // endpoint and the standing outage both keep their place. + std::vector foreign_only = {foreign}; + EXPECT_EQ(NetworkDiscovery::select_auto_endpoint(foreign_only, true, mine), nullptr); + + // Nothing bound: the deterministic lowest ip:port. + const auto * first_adoption = NetworkDiscovery::select_auto_endpoint(eps, true, ""); + ASSERT_NE(first_adoption, nullptr); + EXPECT_EQ(first_adoption->ip, "192.168.1.10"); + + // A server that names no identity cannot be the bound one. + DiscoveredEndpoint anonymous_server = foreign; + anonymous_server.application_uri.clear(); + std::vector nameless = {anonymous_server}; + EXPECT_EQ(NetworkDiscovery::select_auto_endpoint(nameless, true, mine), nullptr); +} + TEST(SelectAutoEndpoint, DeterministicLowestAddressWins) { DiscoveredEndpoint a; a.ip = "192.168.1.20"; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 8c6ea478b..02723117e 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -61,6 +62,7 @@ #include #include +#include #include #include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" @@ -696,11 +698,10 @@ class ScopedExecutorSpin { public: using CancelFn = std::function; - // The cancel is injectable so a test can make it fail. A callable rather than - // a virtual override on a derived executor because rclcpp::Executor::cancel() - // is virtual on jazzy and later but NOT on humble, where a subclass's - // cancel() would neither compile with `override` nor be the one called - // through a base reference. + // The cancel is injectable so a test can make it fail. It is a callable + // because rclcpp::Executor::cancel() is virtual on jazzy and later but NOT on + // humble, where a subclass's cancel() would neither compile with `override` + // nor be the one called through a base reference. explicit ScopedExecutorSpin(rclcpp::executors::MultiThreadedExecutor & executor, CancelFn cancel = nullptr) : executor_(executor) , cancel_(cancel ? std::move(cancel) : CancelFn([this]() { @@ -796,6 +797,205 @@ class RealNodePluginContext : public FakePluginContext { rclcpp::Node * node_; }; +// Poll until an OPC-UA session can be opened at ``endpoint``. A fixture prints +// READY before its listen socket is accepting, so nothing may rely on one +// before this returns. +bool wait_for_connectable(const std::string & endpoint) { + for (int attempt = 0; attempt < 50; ++attempt) { + OpcuaClient probe; + OpcuaClientConfig config; + config.endpoint_url = endpoint; + config.connect_timeout = std::chrono::milliseconds(1000); + if (probe.connect(config)) { + probe.disconnect(); + return true; + } + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + return false; +} + +// The ApplicationUri the server at ``endpoint`` publishes, read the same way +// the plugin reads it off a live session. Empty when the session cannot be +// opened or the server publishes none. +std::string live_application_uri(const std::string & endpoint) { + OpcuaClient probe; + OpcuaClientConfig config; + config.endpoint_url = endpoint; + config.connect_timeout = std::chrono::milliseconds(5000); + if (!probe.connect(config)) { + return {}; + } + const std::string uri = probe.read_server_application_uri(); + probe.disconnect(); + return uri; +} + +// The component id a config-less plugin derives from the server at +// ``endpoint``, read over a throwaway session so no test pins the fixture's +// nameplate spelling. +std::string device_derived_component_id(const std::string & endpoint) { + OpcuaClient probe; + OpcuaClientConfig config; + config.endpoint_url = endpoint; + config.connect_timeout = std::chrono::milliseconds(5000); + if (!probe.connect(config)) { + return {}; + } + const std::string id = derive_component_identity(probe.read_device_info(), endpoint).id; + probe.disconnect(); + return id; +} + +// A stand-in fault manager carrying the three services the plugin talks to, +// keyed the way the real one is: one row per fault code holding the set of +// sources that reported it, a read that answers from those rows, and a clear +// that removes the whole row - ClearFault has no source field. +// +// The services are created one at a time so a test can decide in which order +// the plugin discovers them; ``open_reads(false)`` parks GetFault requests +// unanswered until ``release_reads()``. +class FaultStoreStub { + public: + explicit FaultStoreStub(rclcpp::Node::SharedPtr node) : node_(std::move(node)) { + } + + void open_reports() { + report_srv_ = node_->create_service( + "/fault_manager/report_fault", [this](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(mutex_); + reported_.push_back(req->fault_code); + auto & sources = rows_[req->fault_code]; + if (std::find(sources.begin(), sources.end(), req->source_id) == sources.end()) { + sources.push_back(req->source_id); + } + } + res->accepted = true; + }); + } + + void open_clears() { + clear_srv_ = node_->create_service( + "/fault_manager/clear_fault", [this](const std::shared_ptr req, + std::shared_ptr res) { + { + std::lock_guard lock(mutex_); + cleared_.push_back(*req); + rows_.erase(req->fault_code); // ClearFault carries no source: the row goes + } + res->success = true; + }); + } + + void open_reads(bool answer_immediately = true) { + answer_reads_.store(answer_immediately); + read_srv_ = node_->create_service( + "/fault_manager/get_fault", [this](const std::shared_ptr header, + const std::shared_ptr req) { + if (answer_reads_.load()) { + answer_read(*header, req->fault_code); + return; + } + std::lock_guard lock(mutex_); + parked_reads_.emplace_back(*header, req->fault_code); + }); + } + + /// Answer the OLDEST parked read and keep parking the ones that follow, so a + /// test can let a probe the plugin has already given up on answer late. + bool release_one_read() { + std::pair parked; + { + std::lock_guard lock(mutex_); + if (parked_reads_.empty()) { + return false; + } + parked = parked_reads_.front(); + parked_reads_.erase(parked_reads_.begin()); + } + answer_read(parked.first, parked.second); + return true; + } + + /// Answer every parked read and keep answering the ones that follow. + void release_reads() { + std::vector> parked; + { + std::lock_guard lock(mutex_); + parked.swap(parked_reads_); + } + answer_reads_.store(true); + for (auto & entry : parked) { + answer_read(entry.first, entry.second); + } + } + + size_t parked_read_count() const { + std::lock_guard lock(mutex_); + return parked_reads_.size(); + } + + void seed(const std::string & fault_code, const std::vector & sources) { + std::lock_guard lock(mutex_); + rows_[fault_code] = sources; + } + + std::vector sources_of(const std::string & fault_code) const { + std::lock_guard lock(mutex_); + const auto it = rows_.find(fault_code); + return it == rows_.end() ? std::vector{} : it->second; + } + + std::vector cleared_codes() const { + std::lock_guard lock(mutex_); + std::vector codes; + codes.reserve(cleared_.size()); + for (const auto & req : cleared_) { + codes.push_back(req.fault_code); + } + return codes; + } + + std::vector cleared() const { + std::lock_guard lock(mutex_); + return cleared_; + } + + std::vector reported() const { + std::lock_guard lock(mutex_); + return reported_; + } + + private: + void answer_read(const rmw_request_id_t & header, const std::string & fault_code) { + ros2_medkit_msgs::srv::GetFault::Response response; + { + std::lock_guard lock(mutex_); + const auto it = rows_.find(fault_code); + response.success = it != rows_.end() && !it->second.empty(); + if (response.success) { + response.fault.fault_code = fault_code; + response.fault.reporting_sources = it->second; + } + } + rmw_request_id_t id = header; + read_srv_->send_response(id, response); + } + + rclcpp::Node::SharedPtr node_; + mutable std::mutex mutex_; + std::map> rows_; + std::vector cleared_; + std::vector reported_; + std::vector> parked_reads_; + std::atomic answer_reads_{true}; + rclcpp::Service::SharedPtr report_srv_; + rclcpp::Service::SharedPtr clear_srv_; + rclcpp::Service::SharedPtr read_srv_; +}; + } // namespace // A cancel() that throws must not cost the join. If it does, the guard's thread @@ -811,8 +1011,8 @@ TEST(ScopedExecutorSpinTest, AThrowingCancelStillJoinsTheThread) { { // Fails the way rclcpp documents cancel() can - the guard condition cannot // be triggered - after actually stopping the spin, so what is under test is - // the join and not a hang. Injected rather than overridden: cancel() is not - // virtual on every distro this builds on. + // the join and not a hang. Injected because cancel() is not virtual on every + // distro this builds on. ScopedExecutorSpin spin(executor, [&executor]() { executor.cancel(); throw std::runtime_error("cancel failed"); @@ -826,34 +1026,23 @@ TEST(ScopedExecutorSpinTest, AThrowingCancelStillJoinsTheThread) { SUCCEED() << "the guard joined its thread despite cancel() throwing"; } -// The connect-time clear, read off the wire. clear_comms_lost_on_connect() is -// only reachable through a connect that SUCCEEDS, so it needs the live fixture, -// and the flag it sets is only observable with a real fault-manager service on -// the other end. A correlation rule may name PLC_COMMS_LOST as the root cause of -// every symptom an outage produced, and the link coming back is not an operator -// resolving those, so this clear must not cascade. +// The connect-time clear, read off the wire. The decision is only reachable +// through a connect that SUCCEEDS, so it needs the live fixture, and the flag it +// sets is only observable with a real fault-manager service on the other end. A +// correlation rule may name PLC_COMMS_LOST as the root cause of every symptom an +// outage produced, and the link coming back is not an operator resolving those, +// so this clear must not cascade. TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade) { ScopedRclcpp rclcpp_scope; auto node = std::make_shared("opcua_identity_connect_clear"); auto fault_manager = std::make_shared("opcua_identity_connect_clear_faultmgr"); - std::mutex received_mutex; - std::vector cleared_requests; - auto report_srv = fault_manager->create_service( - "/fault_manager/report_fault", [](const std::shared_ptr, - std::shared_ptr res) { - res->accepted = true; - }); - auto clear_srv = fault_manager->create_service( - "/fault_manager/clear_fault", - [&cleared_requests, &received_mutex](const std::shared_ptr req, - std::shared_ptr res) { - { - std::lock_guard lock(received_mutex); - cleared_requests.push_back(*req); - } - res->success = true; - }); + FaultStoreStub store(fault_manager); + // A PLC_COMMS_LOST this bridge raised before the process restarted. + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(); rclcpp::executors::MultiThreadedExecutor executor; executor.add_node(node); @@ -871,18 +1060,11 @@ TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade RealNodePluginContext ctx(node.get()); ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; // The connect inside set_context() succeeds against the fixture, which is the - // only way to reach the connect-time clear. + // only way to reach the connect-time decision. plugin.set_context(ctx); - // The clear may be buffered until the stub service is DDS-matched. The poll - // thread drains the buffer on its next cycle. const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); - bool delivered = false; - while (!delivered && std::chrono::steady_clock::now() < deadline) { - { - std::lock_guard lock(received_mutex); - delivered = !cleared_requests.empty(); - } + while (store.cleared().empty() && std::chrono::steady_clock::now() < deadline) { std::this_thread::sleep_for(std::chrono::milliseconds(50)); } @@ -890,10 +1072,10 @@ TEST_F(OpcuaIdentityE2ETest, ConnectTimeCommsLostClearSkipsTheCorrelationCascade plugin.shutdown(); std::remove(yaml_path.c_str()); - std::lock_guard lock(received_mutex); - ASSERT_FALSE(cleared_requests.empty()) << "a successful connect sent no ClearFault at all"; - EXPECT_EQ(cleared_requests.front().fault_code, std::string(kCommsLostFaultCode)); - EXPECT_TRUE(cleared_requests.front().skip_correlation_auto_clear) + const auto cleared = store.cleared(); + ASSERT_FALSE(cleared.empty()) << "a successful connect sent no ClearFault at all"; + EXPECT_EQ(cleared.front().fault_code, std::string(kCommsLostFaultCode)); + EXPECT_TRUE(cleared.front().skip_correlation_auto_clear) << "the connect-time clear cascade-cleared the symptoms of the outage it ended"; } @@ -907,29 +1089,17 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) auto node = std::make_shared("opcua_identity_device_clear"); auto fault_manager = std::make_shared("opcua_identity_device_clear_faultmgr"); - std::mutex received_mutex; - std::vector reported; - std::vector cleared_requests; - auto report_srv = fault_manager->create_service( - "/fault_manager/report_fault", - [&reported, &received_mutex](const std::shared_ptr req, - std::shared_ptr res) { - { - std::lock_guard lock(received_mutex); - reported.push_back(req->fault_code); - } - res->accepted = true; - }); - auto clear_srv = fault_manager->create_service( - "/fault_manager/clear_fault", - [&cleared_requests, &received_mutex](const std::shared_ptr req, - std::shared_ptr res) { - { - std::lock_guard lock(received_mutex); - cleared_requests.push_back(*req); - } - res->success = true; - }); + // Config-less: the component names itself from the device, and that is the id + // the gate asks the store about. Read it the same way the plugin will, so no + // test pins the fixture's nameplate spelling. + const std::string component_id = device_derived_component_id(endpoint_); + ASSERT_FALSE(component_id.empty()); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {component_id}); + store.open_reports(); + store.open_clears(); + store.open_reads(); rclcpp::executors::MultiThreadedExecutor executor; executor.add_node(node); @@ -948,15 +1118,13 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) RealNodePluginContext ctx(node.get()); plugin.set_context(ctx); - const auto reported_count = [&received_mutex, &reported]() { - std::lock_guard lock(received_mutex); - return reported.size(); + const auto reported_count = [&store]() { + return store.reported().size(); }; // The connect-time PLC_COMMS_LOST clear also lands here (this connect // succeeded), so a clear is looked up by the code it names. - const auto clear_for = [&received_mutex, &cleared_requests](const std::string & code) -> std::optional { - std::lock_guard lock(received_mutex); - for (const auto & req : cleared_requests) { + const auto clear_for = [&store](const std::string & code) -> std::optional { + for (const auto & req : store.cleared()) { if (req.fault_code == code) { return req.skip_correlation_auto_clear; } @@ -974,11 +1142,7 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) } ASSERT_GT(reported_count(), 0u) << "the fixture's AlarmCondition never reached the fault manager"; - std::string alarm_code; - { - std::lock_guard lock(received_mutex); - alarm_code = reported.front(); - } + const std::string alarm_code = store.reported().front(); ASSERT_TRUE(server_.send("clear Overpressure")); const auto clear_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); while (!clear_for(alarm_code).has_value() && std::chrono::steady_clock::now() < clear_deadline) { @@ -1002,4 +1166,708 @@ TEST_F(OpcuaIdentityE2ETest, DeviceReportedAlarmClearKeepsTheCorrelationCascade) EXPECT_TRUE(*link_state_clear_skips); } +// One gateway may bridge two field buses, both raising PLC_COMMS_LOST, and +// ClearFault carries no source: it clears the whole row. A row this bridge +// shares with another is not this link's to clear, whether the foreign id is the +// only source or one of several. The sibling test above is the positive control +// - identical setup, the row naming only this bridge, and the clear goes out. +TEST_F(OpcuaIdentityE2ETest, ASharedCommsLostRowIsLeftStanding) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_foreign_clear"); + auto fault_manager = std::make_shared("opcua_identity_foreign_clear_faultmgr"); + + FaultStoreStub store(fault_manager); + // Two bridges hold the row: the other one's link says nothing about ours. + store.seed(kCommsLostFaultCode, {"beckhoff_cx5140", "test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + // The probe is asked, answered and acted on within a couple of poll cycles; + // the sibling test's clear lands well inside this window on the same harness, + // so an empty result here is a decision and not a missed deadline. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); + bool probe_answered = false; + while (!probe_answered && std::chrono::steady_clock::now() < deadline) { + probe_answered = plugin.comms_lost_probe_count_for_test() > 0; + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + // Give any (wrong) clear the time the sibling test's right one needs. + std::this_thread::sleep_for(std::chrono::seconds(2)); + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + EXPECT_TRUE(probe_answered) << "the store was never asked who holds " << kCommsLostFaultCode; + for (const auto & req : store.cleared()) { + EXPECT_NE(req.fault_code, std::string(kCommsLostFaultCode)) + << "this link coming back cleared a row another bridge also holds"; + } + EXPECT_EQ(store.sources_of(kCommsLostFaultCode).size(), 2u) << "the shared row was cleared"; +} + +// A link-state clear the bounded buffer refuses is owed, not abandoned. A +// gateway restarting with a persisted PLC_COMMS_LOST while the fault manager is +// still down has no reconnect coming - its connect SUCCEEDED - so nothing +// re-derives that clear and the fault stands CONFIRMED against a healthy link. +// The decision is taken again once the buffer has drained, and its clear lands +// behind everything it must not overtake. +TEST_F(OpcuaIdentityE2ETest, AnOwedLinkStateClearIsSentAfterTheBufferDrains) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_owed_clear"); + auto fault_manager = std::make_shared("opcua_identity_owed_clear_faultmgr"); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + // Reads and clears are reachable; REPORTS are not, which is what holds the + // pending buffer full so the link-state clear is refused by it. + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + // More one-shot dispatches than the buffer can hold. Each is an operator's + // scoped clear: nothing re-derives one, so they outrank the link-state clear, + // which the buffer gives up first. + const size_t queued = OpcuaPlugin::kMaxPendingDispatches + 44; + for (size_t i = 0; i < queued; ++i) { + static_cast(plugin.clear_fault("tank", "PLC_OPERATOR_" + std::to_string(i))); + } + + // The refusal needs a decision to have happened against the full buffer. + const auto refusal_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); + while (plugin.comms_lost_probe_count_for_test() == 0 && std::chrono::steady_clock::now() < refusal_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + ASSERT_GT(plugin.comms_lost_probe_count_for_test(), 0u) << "the store was never asked while the buffer was full"; + EXPECT_TRUE(store.cleared_codes().empty()) << "the buffer dispatched while the report sink was unreachable"; + + store.open_reports(); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + const auto comms_lost_position = [&store]() -> std::optional { + const auto codes = store.cleared_codes(); + for (size_t i = 0; i < codes.size(); ++i) { + if (codes[i] == kCommsLostFaultCode) { + return i; + } + } + return std::nullopt; + }; + while (!comms_lost_position().has_value() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + const auto codes = store.cleared_codes(); + const auto position = comms_lost_position(); + ASSERT_TRUE(position.has_value()) << "the owed " << kCommsLostFaultCode + << " clear was never re-issued, so the fault stands against a live link"; + EXPECT_EQ(*position, codes.size() - 1) << "the owed clear overtook the buffered dispatches"; + EXPECT_EQ(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 1) + << "the owed clear was re-issued more than once"; +} + +// A decision is taken only while the session is up. A clear decided against a +// link that is down would clear a fault that is genuinely standing, which is the +// state the poller has just reported. The probe predicate carries that term and +// CommsLostProbeDue is where it is falsified; this drives the same promise +// through the whole plugin. +TEST_F(OpcuaIdentityE2ETest, NoLinkStateClearIsDecidedWhileTheLinkIsDown) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_link_down"); + auto fault_manager = std::make_shared("opcua_identity_link_down_faultmgr"); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 200; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + // Connect succeeds, so a decision is owed; no fault services exist yet, so it + // cannot be taken. + plugin.set_context(ctx); + + server_.stop(); // the PLC goes away with the decision still owed + store.open_reports(); + store.open_clears(); + store.open_reads(); + + // The link being down is what the poller reports, and that report is the + // control: it proves the harness is live while no clear travels. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + const auto reported_comms_lost = [&store]() { + const auto reported = store.reported(); + return std::find(reported.begin(), reported.end(), std::string(kCommsLostFaultCode)) != reported.end(); + }; + while (!reported_comms_lost() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + std::this_thread::sleep_for(std::chrono::seconds(2)); + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + EXPECT_TRUE(reported_comms_lost()) << "the dead link was never reported, so this proves nothing"; + const auto codes = store.cleared_codes(); + EXPECT_EQ(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 0) + << "a clear was decided against a link that is down"; +} + +// A store that never answers must not hold the decision for the life of the +// process. The probe is dropped once it outlives fault_service_timeout_ms, the +// decision is owed again, and the clear goes out on the answer that does come. +TEST_F(OpcuaIdentityE2ETest, AnUnansweredProbeIsDroppedAndTheDecisionIsTakenAgain) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_probe_timeout"); + auto fault_manager = std::make_shared("opcua_identity_probe_timeout_faultmgr"); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(/*answer_immediately=*/false); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["fault_service_timeout_ms"] = 1000; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + const auto probe_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.parked_read_count() < 2 && std::chrono::steady_clock::now() < probe_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const size_t probes_before_release = store.parked_read_count(); + + store.release_reads(); + const auto clear_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.cleared_codes().empty() && std::chrono::steady_clock::now() < clear_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + EXPECT_GE(probes_before_release, 2u) << "a probe the store never answered held the decision for good"; + const auto codes = store.cleared_codes(); + EXPECT_GE(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 1) + << "the decision was never taken once the store answered"; +} + +// A probe the poll thread has given up on may still be answered by the store. +// That answer describes the question asked before the timeout, and the decision +// waiting now belongs to the probe that replaced it, so the late one is ignored +// and only the current probe's answer decides. +// +// Two mechanisms carry that, and neither is reachable alone from here: +// remove_pending_request takes the entry out of the client, and the generation +// the timeout branch moves on catches a callback that won the race against that +// erase (rclcpp erases before it invokes the callback, outside its mutex). The +// race is not reproducible on demand, so this pins the behaviour and the unit +// test on the consume-side check is what discriminates the generation. +TEST_F(OpcuaIdentityE2ETest, AnAnswerToATimedOutProbeIsNotReadAsTheNextOnes) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_stale_answer"); + auto fault_manager = std::make_shared("opcua_identity_stale_answer_faultmgr"); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(/*answer_immediately=*/false); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["fault_service_timeout_ms"] = 1000; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + // Probe A parks, times out, and probe B parks behind it. + const auto probe_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.parked_read_count() < 2 && std::chrono::steady_clock::now() < probe_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + ASSERT_GE(store.parked_read_count(), 2u) << "the probe never timed out, so there is no stale answer to ignore"; + + // A answers late. + ASSERT_TRUE(store.release_one_read()); + std::this_thread::sleep_for(std::chrono::seconds(2)); + EXPECT_TRUE(store.cleared_codes().empty()) << "an answer to a probe already given up on decided the clear"; + + // B answers, and that is the answer the decision is waiting for. + store.release_reads(); + const auto clear_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.cleared_codes().empty() && std::chrono::steady_clock::now() < clear_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + const auto codes = store.cleared_codes(); + EXPECT_EQ(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 1) + << "the current probe's answer did not decide exactly one clear"; +} + +// An answer parked before the link dropped is not applied after the reconnect. +// +// The decision is driven from publish_values, which the poll loop reaches only +// while connected, so an answer parked just before a drop is first looked at on +// the tick AFTER the reconnect - when the link reads as up and the store's +// answer describes a store from before an outage the poller has since reported. +// Here a second bridge raises the shared code during that outage, so acting on +// the stale answer would clear a row another bridge now holds. +TEST_F(OpcuaIdentityE2ETest, AnAnswerParkedBeforeAnOutageIsNotAppliedAfterTheReconnect) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_stale_session"); + auto fault_manager = std::make_shared("opcua_identity_stale_session_faultmgr"); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(/*answer_immediately=*/false); // the first probe parks + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 200; + // Long enough that the probe does not time out while the link is down: the + // answer has to survive to the tick after the reconnect, which is the case + // under test. + config["fault_service_timeout_ms"] = 120000; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + const auto wait_for = [](const std::function & done, std::chrono::seconds budget) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (!done() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + return done(); + }; + + // The connect-time probe is parked, unanswered. + ASSERT_TRUE(wait_for( + [&store]() { + return store.parked_read_count() >= 1; + }, + std::chrono::seconds(30))) + << "the connect never asked the store, so this proves nothing"; + + // The link drops with the probe still outstanding. The poll loop sits in its + // reconnect arm from here, so nothing consumes an answer until it is back. + server_.stop(); + ASSERT_TRUE(wait_for( + [&store]() { + const auto reported = store.reported(); + return std::find(reported.begin(), reported.end(), std::string(kCommsLostFaultCode)) != reported.end(); + }, + std::chrono::seconds(30))) + << "the outage was never reported, so this proves nothing"; + + // The store answers the outstanding probe now, describing the row as it was + // when the probe was sent: ours alone. + store.release_reads(); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + + // A second bridge reports the same code while the outage lasts, so the row + // the parked answer describes is not the row standing now. + store.seed(kCommsLostFaultCode, {"test_runtime", "beckhoff_cx5140"}); + + // The PLC comes back. The first tick after the reconnect is where the stale + // answer would be consumed. + ASSERT_TRUE(server_.start(fixture_binary(), port_)) << "the fixture did not come back"; + ASSERT_TRUE(wait_until_connectable()); + ASSERT_TRUE(wait_for( + [&plugin]() { + return plugin.comms_lost_probe_count_for_test() >= 2; + }, + std::chrono::seconds(30))) + << "the decision was never taken again on the new session"; + std::this_thread::sleep_for(std::chrono::seconds(2)); + + spin.stop(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + const auto codes = store.cleared_codes(); + EXPECT_EQ(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 0) + << "an answer from the session before the outage cleared a row two bridges now hold"; + EXPECT_EQ(store.sources_of(kCommsLostFaultCode).size(), 2u) << "the shared row was cleared"; +} + +// The config-less restart heal, end to end. A gateway that starts while its PLC +// is down names the component after the endpoint and raises PLC_COMMS_LOST under +// that stand-in. When the PLC returns the device names itself, so the id the +// store holds is no id the component still carries - and the fault this very +// process raised has to heal all the same. +TEST_F(OpcuaIdentityE2ETest, AFaultRaisedUnderTheStandInHealsAfterTheDeviceNamesItself) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_standin_heal"); + auto fault_manager = std::make_shared("opcua_identity_standin_heal_faultmgr"); + + const std::string nameplate_id = device_derived_component_id(endpoint_); + ASSERT_FALSE(nameplate_id.empty()); + + FaultStoreStub store(fault_manager); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + server_.stop(); // the PLC is down when the gateway starts + + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; // config-less: no node map, so the device names the component + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 200; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + + // The outage is reported under the endpoint-derived stand-in. + const auto raise_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.sources_of(kCommsLostFaultCode).empty() && std::chrono::steady_clock::now() < raise_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const auto raised_sources = store.sources_of(kCommsLostFaultCode); + ASSERT_EQ(raised_sources.size(), 1u) << "the outage was never reported, so this proves nothing"; + EXPECT_EQ(raised_sources.front(), derive_component_identity(OpcuaClient::DeviceInfo{}, endpoint_).id) + << "expected the endpoint-derived stand-in, got '" << raised_sources.front() << "'"; + EXPECT_NE(raised_sources.front(), nameplate_id); + + ASSERT_TRUE(server_.start(fixture_binary(), port_)) << "the fixture did not come back"; + ASSERT_TRUE(wait_until_connectable()); + + const auto heal_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(45); + while (store.sources_of(kCommsLostFaultCode).size() != 0 && std::chrono::steady_clock::now() < heal_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + spin.stop(); + plugin.shutdown(); + + const auto codes = store.cleared_codes(); + EXPECT_GE(std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)), 1) + << "a fault this process raised under its stand-in was never healed after the device named itself"; + EXPECT_TRUE(store.sources_of(kCommsLostFaultCode).empty()); +} + +// The binding, driven end to end against live fixtures. +// +// The sweep is substituted (which is what the injected discovery I/O is for) so +// the test decides which address is open and which identity is served there; +// every session, identity read and connect is real. Discovery identifies OPC-UA +// only on 4840 (network_discovery.cpp), so the fixtures listen there and it is +// the ADDRESS that varies - a fixture binds every interface, so one process is +// reachable at 127.0.0.1 and 127.0.0.2 alike, which is what makes "the same +// server at a new address" and "a different server at the bound address" both +// reachable. +// +// Three claims: a live, reachable foreign server is not adopted by a sweep; a +// live foreign server at the bound address is dropped at connect; the bound +// server at a new address is re-adopted. +TEST_F(OpcuaIdentityE2ETest, TheBridgeStaysBoundToItsOwnServerAcrossAddressAndSwap) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_binding"); + auto fault_manager = std::make_shared("opcua_identity_binding_faultmgr"); + + server_.stop(); // the base class's fixture does not take part + constexpr int kOpcuaPort = 4840; + const std::string bound_address = "127.0.0.1"; + const std::string other_address = "127.0.0.2"; + const std::string foreign_uri = "urn:test:a-different-plc"; + const auto url_for = [](const std::string & ip) { + return "opc.tcp://" + ip + ":4840"; + }; + + // What the sweep reports: one address, and the identity served there. + std::mutex sweep_mutex; + std::string open_address = bound_address; + std::string served_uri; + const auto scan = [&sweep_mutex, &open_address](const std::string & ip, uint16_t port, int) { + std::lock_guard lock(sweep_mutex); + return port == kOpcuaPort && ip == open_address; + }; + const auto identify = [&sweep_mutex, &served_uri](const std::string & url, int) { + IdentifyResult result; + result.ok = true; + result.advertised_url = url; + { + std::lock_guard lock(sweep_mutex); + result.application_uri = served_uri; + } + result.application_name = "Test PLC"; + result.application_type = 0; // Server + result.anonymous_none_available = true; + return result; + }; + const auto sweep_reports = [&sweep_mutex, &open_address, &served_uri](const std::string & ip, + const std::string & uri) { + std::lock_guard lock(sweep_mutex); + open_address = ip; + served_uri = uri; + }; + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) + << "this test needs TCP 4840 on loopback, the only port discovery identifies OPC-UA on"; + ASSERT_TRUE(wait_for_connectable(url_for(bound_address))); + // The identity the fixture actually serves is what the binding becomes, so + // the sweep reports the same one and selection has something to look for. + const std::string bound_uri = live_application_uri(url_for(bound_address)); + ASSERT_FALSE(bound_uri.empty()) << "the fixture publishes no ApplicationUri, so nothing can bind to it"; + ASSERT_NE(bound_uri, foreign_uri); + sweep_reports(bound_address, bound_uri); + + FaultStoreStub store(fault_manager); + // A standing outage, so every accepted session produces a ClearFault: that + // clear is how the test sees which server the plugin is polling. + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + plugin.set_discovery_io_for_test(scan, identify); + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 200; + config["discovery"] = nlohmann::json{{"enabled", true}, + {"subnets", nlohmann::json::array({"127.0.0.0/30"})}, + {"ports", nlohmann::json::array({kOpcuaPort})}, + {"interval_s", 2}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + + const auto cleared_count = [&store]() { + const auto codes = store.cleared_codes(); + return std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)); + }; + const auto wait_for = [](const std::function & done, std::chrono::seconds budget) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (!done() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + return done(); + }; + const auto outage_reported = [&store]() { + const auto reported = store.reported(); + return std::find(reported.begin(), reported.end(), std::string(kCommsLostFaultCode)) != reported.end(); + }; + + ASSERT_TRUE(wait_for( + [&]() { + return cleared_count() > 0; + }, + std::chrono::seconds(30))) + << "the startup scan never adopted the fixture, so this proves nothing"; + const auto clears_after_binding = cleared_count(); + + // ---- a LIVE foreign server, reachable at the bound address and at the one + // the sweep offers, is neither adopted nor polled --------------------- + bound_server.stop(); + AlarmServer foreign_server; + ASSERT_TRUE(foreign_server.start(fixture_binary(), kOpcuaPort, {"--app-uri", foreign_uri})); + ASSERT_TRUE(wait_for_connectable(url_for(bound_address))) << "the foreign fixture never became connectable"; + ASSERT_EQ(live_application_uri(url_for(bound_address)), foreign_uri); + sweep_reports(other_address, foreign_uri); + + ASSERT_TRUE(wait_for(outage_reported, std::chrono::seconds(30))) + << "the outage was never reported, so this proves nothing"; + // The sweep saw the foreign server and found no hit carrying the binding. + ASSERT_TRUE(wait_for( + [&]() { + return plugin.rescan_refused_count_for_test() > 0; + }, + std::chrono::seconds(30))) + << "the sweep never reported the bound server missing, so nothing here is about the binding"; + // ... and the connect the reconnect arm keeps attempting at the bound address + // reaches that same foreign server, which the session's own identity catches. + ASSERT_TRUE(wait_for( + [&]() { + return plugin.binding_mismatch_count_for_test() > 0; + }, + std::chrono::seconds(30))) + << "a different server at the bound address was polled as if it were the bound one"; + std::this_thread::sleep_for(std::chrono::seconds(3)); + + EXPECT_EQ(cleared_count(), clears_after_binding) << "a server this bridge is not bound to cleared the outage"; + EXPECT_FALSE(store.sources_of(kCommsLostFaultCode).empty()) + << "the outage was healed by a server this bridge is not bound to"; + + // ---- the bound server, at the address the foreign one was offered on, is + // re-adopted ---------------------------------------------------------- + foreign_server.stop(); + AlarmServer moved_server; + ASSERT_TRUE(moved_server.start(fixture_binary(), kOpcuaPort)); + ASSERT_TRUE(wait_for_connectable(url_for(other_address))) << "the moved fixture never became connectable"; + sweep_reports(other_address, bound_uri); + + const bool recovered = wait_for( + [&]() { + return cleared_count() > clears_after_binding; + }, + std::chrono::seconds(60)); + + spin.stop(); + plugin.shutdown(); + moved_server.stop(); + std::remove(yaml_path.c_str()); + + EXPECT_TRUE(recovered) << "the bound server at " << url_for(other_address) + << " was not re-adopted, so the outage never ended"; + EXPECT_TRUE(store.sources_of(kCommsLostFaultCode).empty()) << "the outage was cleared but the row still stands"; +} + +// The order the whole rename fix rests on: PollerConfig::on_connected runs +// before the link-state edge and before anything is subscribed, so whatever it +// renames is what the event path is handed. apply_condition_state pins a fault's +// entity at the first sighting of its ConditionId, and the ConditionRefresh +// burst that follows the subscribe is that first sighting for every condition +// the device had standing - so a rename after it files those faults under an +// entity the rename then drops. +TEST_F(OpcuaIdentityE2ETest, TheConnectedHookRunsBeforeTheEventRoutingIsCopied) { + OpcuaClient client; + OpcuaClientConfig config; + config.endpoint_url = endpoint_; + config.connect_timeout = std::chrono::milliseconds(5000); + ASSERT_TRUE(client.connect(config)); + + NodeMap node_map; // config-less: named after the endpoint until a device answers + node_map.set_component_identity("opcua-127_0_0_1", "opcua-127_0_0_1"); + node_map.mutable_auto_alarms().enabled = true; + ASSERT_TRUE(node_map.finalize_auto_alarms_overlay()); + ASSERT_EQ(node_map.auto_alarms().entity_id, "opcua-127_0_0_1_alarms"); + + OpcuaPoller poller(client, node_map); + std::atomic hook_calls{0}; + PollerConfig poller_config; + poller_config.poll_interval = std::chrono::milliseconds(100); + poller_config.on_connected = [&hook_calls, &node_map]() { + hook_calls.fetch_add(1); + // Exactly what the plugin's hook does once the adopted device names itself. + node_map.mutable_auto_alarms().entity_id.clear(); + node_map.set_component_identity("siemens_ag_cpu_1505sp_f", "Siemens AG CPU 1505SP F"); + node_map.finalize_auto_alarms_overlay(); + return true; + }; + poller.start(poller_config); + + const auto routing = poller.alarm_routing(); + poller.stop(); + client.disconnect(); + + EXPECT_EQ(hook_calls.load(), 1) << "the connected hook never fired on a session that was already up"; + ASSERT_TRUE(routing) << "no event subscription was made, so nothing was copied"; + EXPECT_EQ(routing->auto_alarms.entity_id, "siemens_ag_cpu_1505sp_f_alarms") + << "the routing was copied before the rename, so every condition replayed on this session " + "would be pinned under an entity the rename drops"; +} + } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp index b50935eef..c6f2fb4e9 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -989,6 +990,12 @@ TEST(RescanStep, AThrowingSweepStillStampsTheCadence) { // on the normal path, the next poll iteration would find the cadence due and // start another sweep immediately, so a server that makes the identify throw // would turn the poll thread into a continuous scanner. + // + // The rethrow here is not the poll thread surviving on its own - it has no + // try, so an exception reaching it is std::terminate. What catches is + // rescan_guarded, which is what the plugin's hook actually calls; this + // function only guarantees that the stamp is taken before the exception + // travels there. const auto t0 = std::chrono::steady_clock::time_point{}; auto clock_now = t0 + std::chrono::seconds(30); const auto now = [&clock_now]() { @@ -1017,6 +1024,49 @@ TEST(RescanStep, AThrowingSweepStillStampsTheCadence) { EXPECT_EQ(sweeps, 2); } +TEST(RescanGuarded, AThrowingSweepEndsAsNoEndpoint) { + // What the poll thread runs. The poll thread has no try of its own, so an + // exception reaching it is std::terminate for the whole gateway - a + // std::system_error from a thread the parallel scan cannot create, or a + // bad_alloc on a wide target list. A sweep that failed is a sweep that found + // nothing. + const auto t0 = std::chrono::steady_clock::time_point{}; + auto clock_now = t0 + std::chrono::seconds(30); + const auto now = [&clock_now]() { + return clock_now; + }; + auto last_end = t0; + const auto std_throw = [&clock_now]() -> std::optional { + clock_now += std::chrono::seconds(120); + throw std::system_error(std::make_error_code(std::errc::resource_unavailable_try_again), + "Resource temporarily unavailable"); + }; + std::optional chosen; + ASSERT_NO_THROW(chosen = OpcuaPlugin::rescan_guarded(30, now, &last_end, std_throw)); + EXPECT_FALSE(chosen.has_value()) << "a failed sweep must not name an endpoint"; + EXPECT_EQ(last_end, clock_now) << "the cadence stamp is owed even when the sweep threw"; + + // Anything at all, std::exception or otherwise: open62541 and the identify + // path are C libraries wrapped by C++, and a non-standard throw is still a + // terminate without the catch-all. + clock_now += std::chrono::seconds(30); + const auto alien_throw = []() -> std::optional { + throw 42; + }; + ASSERT_NO_THROW(chosen = OpcuaPlugin::rescan_guarded(30, now, &last_end, alien_throw)); + EXPECT_FALSE(chosen.has_value()); + + // Positive control on the same harness: a sweep that works still hands its + // endpoint through, so the guard is not swallowing the normal path. + clock_now += std::chrono::seconds(30); + const auto good_sweep = []() -> std::optional { + return std::string("opc.tcp://192.168.1.10:4840"); + }; + chosen = OpcuaPlugin::rescan_guarded(30, now, &last_end, good_sweep); + ASSERT_TRUE(chosen.has_value()); + EXPECT_EQ(*chosen, "opc.tcp://192.168.1.10:4840"); +} + TEST(RescanStep, DoesNothingWithoutACadence) { const auto t0 = std::chrono::steady_clock::time_point{}; auto last_end = t0; @@ -1064,7 +1114,7 @@ TEST(NextReconnectWait, DoublesUpToTheCeiling) { EXPECT_EQ(OpcuaPoller::next_reconnect_wait(40000ms, 60000ms), 60000ms); EXPECT_EQ(OpcuaPoller::next_reconnect_wait(60000ms, 60000ms), 60000ms); // Capped at a 30 s rescan cadence: the wait never exceeds it, so the rescan is - // consulted every cadence instead of every max(cadence, backoff). + // consulted once per cadence, which is what the cap is for. EXPECT_EQ(OpcuaPoller::next_reconnect_wait(20000ms, 30000ms), 30000ms); EXPECT_EQ(OpcuaPoller::next_reconnect_wait(30000ms, 30000ms), 30000ms); } @@ -1231,7 +1281,20 @@ TEST(DiscoverEndpoint, WithNoRepeatMemoryEveryPassIsReported) { // Config-less component identity across an adoption // --------------------------------------------------------------------------- -TEST(RederivedComponentIdentity, AdoptionReplacesTheProvisionalEndpointDerivedId) { +namespace { + +// The identity the plugin is serving, as the rule sees it: an id the plugin +// assigned from a read with no nameplate is a stand-in, anything else is not. +OpcuaPlugin::ComponentIdentityState serving(const std::string & current_id, bool is_placeholder = false) { + OpcuaPlugin::ComponentIdentityState state; + state.current_id = current_id; + state.current_is_placeholder = is_placeholder; + return state; +} + +} // namespace + +TEST(RederivedComponentIdentity, ANameplateTakesOverFromAStandInAndFromTheDefault) { // The config-less race, end to end over the derivation path: the gateway // starts before the PLC, its start-up scan finds nothing, and the identity is // derived from the fallback endpoint plus an empty DeviceInfo. @@ -1248,32 +1311,91 @@ TEST(RederivedComponentIdentity, AdoptionReplacesTheProvisionalEndpointDerivedId fake_identify({{"opc.tcp://192.168.1.10:4840", plc_identity()}}), silent_reporter()); ASSERT_TRUE(adopted.has_value()); - // The session is up, so the device can finally name itself: the component - // must stop being served under the placeholder. OpcuaClient::DeviceInfo info; info.di_manufacturer = "Siemens AG"; info.di_model = "CPU 1505SP F"; - const auto rederived = OpcuaPlugin::rederived_component_identity(provisional.id, info, *adopted); + const auto rederived = + OpcuaPlugin::rederived_component_identity(serving(provisional.id, /*is_placeholder=*/true), info, *adopted); ASSERT_TRUE(rederived.has_value()) << "an adopted device with a nameplate must replace opcua-localhost"; EXPECT_EQ(rederived->id, "siemens_ag_cpu_1505sp_f"); EXPECT_EQ(rederived->name, "Siemens AG CPU 1505SP F"); + + // The NodeMap default is not a stand-in this plugin assigned, and config-less + // mode never serves it - the endpoint always derives an id - so the rule + // leaves it where it is. It stays in the clear gate's id set, which is a + // question about who reported a fault, not about renaming. + EXPECT_FALSE( + OpcuaPlugin::rederived_component_identity(serving(NodeMap::kDefaultComponentId), info, *adopted).has_value()); } -TEST(RederivedComponentIdentity, KeepsTheIdentityWhenNothingChanged) { - // Same device on a later reconnect: no rename, so no entity churn and no INFO - // line claiming an identity change that did not happen. +TEST(RederivedComponentIdentity, ANameplateDoesNotUnseatANameplate) { OpcuaClient::DeviceInfo info; info.di_manufacturer = "Siemens AG"; info.di_model = "CPU 1505SP F"; - EXPECT_FALSE(OpcuaPlugin::rederived_component_identity("siemens_ag_cpu_1505sp_f", info, "opc.tcp://192.168.1.10:4840") + // Same device on a later reconnect: no rename, so no entity churn and no INFO + // line claiming an identity change that did not happen. + EXPECT_FALSE( + OpcuaPlugin::rederived_component_identity(serving("siemens_ag_cpu_1505sp_f"), info, "opc.tcp://192.168.1.10:4840") + .has_value()); + + // A different device answering on the same endpoint is a question about asset + // identity, not about a read: the served name stays put. + OpcuaClient::DeviceInfo other; + other.di_manufacturer = "Beckhoff"; + other.di_model = "CX5140"; + EXPECT_FALSE(OpcuaPlugin::rederived_component_identity(serving("siemens_ag_cpu_1505sp_f"), other, + "opc.tcp://192.168.1.10:4840") .has_value()); +} - // A nameplate-less server on an adopted endpoint still moves off the - // fallback host it was provisionally named after. - const auto host_derived = OpcuaPlugin::rederived_component_identity("opcua-localhost", OpcuaClient::DeviceInfo{}, - "opc.tcp://192.168.1.10:4840"); +TEST(RederivedComponentIdentity, AReadWithoutANameplateMovesOneStandInToAnother) { + // Adoption with a nameplate-less server: the stand-in follows the endpoint, + // because the host it names is not the host being polled any more. + const auto host_derived = OpcuaPlugin::rederived_component_identity( + serving("opcua-plc1", /*is_placeholder=*/true), OpcuaClient::DeviceInfo{}, "opc.tcp://192.168.1.10:4840"); ASSERT_TRUE(host_derived.has_value()); EXPECT_EQ(host_derived->id, "opcua-192_168_1_10"); + + // The same host on a different port derives the same stand-in, so nothing + // moves: the id is what the rule compares, not the endpoint string. + EXPECT_FALSE(OpcuaPlugin::rederived_component_identity(serving("opcua-plc1", /*is_placeholder=*/true), + OpcuaClient::DeviceInfo{}, "opc.tcp://plc1:4841") + .has_value()) + << "an endpoint re-spelling that derives the same id renamed the component"; +} + +TEST(RederivedComponentIdentity, AnEmptyReadNeverReplacesANameplateWithTheEndpointFallback) { + // read_device_info() answers empty or partial on the first read of a session + // whose address space is still coming up, and the re-derivation runs on every + // fresh session. Accepting that answer renames a device that had named itself + // back to the endpoint stand-in, which moves the component URL and the + // _alarms entity and orphans every fault filed under the old id. + const std::string nameplate_id = "siemens_ag_cpu_1505sp_f"; + EXPECT_FALSE(OpcuaPlugin::rederived_component_identity(serving(nameplate_id), OpcuaClient::DeviceInfo{}, + "opc.tcp://192.168.1.10:4840") + .has_value()) + << "an empty read renamed a nameplate-derived component to the endpoint fallback"; + + // A partial read - the DI nameplate answered, BuildInfo did not, or the other + // way round - is a nameplate, so it takes over from a stand-in. + OpcuaClient::DeviceInfo partial; + partial.product_name = "CPU 1505SP F"; + const auto from_partial = OpcuaPlugin::rederived_component_identity( + serving("opcua-localhost", /*is_placeholder=*/true), partial, "opc.tcp://192.168.1.10:4840"); + ASSERT_TRUE(from_partial.has_value()); + EXPECT_EQ(from_partial->id, "cpu_1505sp_f"); +} + +TEST(ComponentIdentityShape, ANameplateReadIsWhatSettlesASession) { + // The rename rule turns on membership in the plugin's list of stand-in ids, + // not on the shape of a string. What a read produced is the other input. + EXPECT_FALSE(component_identity_has_nameplate(OpcuaClient::DeviceInfo{})); + OpcuaClient::DeviceInfo di; + di.di_model = "SPX-1000"; + EXPECT_TRUE(component_identity_has_nameplate(di)); + OpcuaClient::DeviceInfo build_info; + build_info.manufacturer_name = "SelfPatch Test Manufacturer"; + EXPECT_TRUE(component_identity_has_nameplate(build_info)); } // --------------------------------------------------------------------------- @@ -1294,6 +1416,33 @@ TEST(DiscoveryCancelledFor, EitherStopSignalEndsASweep) { EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for(true, false)); } +TEST(DiscoveryCancelledForContext, ReadsTheNodesOwnContextNotTheProcessDefault) { + // A host that constructs its GatewayNode with NodeOptions().context(...) never + // initialises the default context, and rclcpp::ok() with no argument is false + // for the life of such a process, which cancels every sweep before it starts. + // The private context here is that case, and it is alive while the default one + // in this process may be either way, so the answer can only come from the + // context handed in. + auto context = std::make_shared(); + context->init(0, nullptr); + ASSERT_TRUE(rclcpp::ok(context)); + + EXPECT_FALSE(OpcuaPlugin::discovery_cancelled_for_context(/*shutdown_requested=*/false, context)) + << "a sweep on a live private context cancelled itself"; + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for_context(/*shutdown_requested=*/true, context)) + << "shutdown() must still end a rescan sweep"; + + context->shutdown("simulated SIGTERM"); + ASSERT_FALSE(rclcpp::ok(context)); + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for_context(/*shutdown_requested=*/false, context)) + << "a signal on the node's own context left the sweep running"; + + // Before set_context() the plugin has no node and no context, which is the + // ordinary gateway's default-context case. + EXPECT_EQ(OpcuaPlugin::discovery_cancelled_for_context(/*shutdown_requested=*/false, nullptr), !rclcpp::ok()); + EXPECT_TRUE(OpcuaPlugin::discovery_cancelled_for_context(/*shutdown_requested=*/true, nullptr)); +} + TEST(DiscoveryCancelledFor, RclcppOkIsTheSignalTheStartUpSweepWatches) { // The second input is not hypothetical: rclcpp's shutdown is what a SIGTERM // turns into, and it is observable exactly this way. A private context keeps @@ -1346,6 +1495,179 @@ TEST(ClearOrigin, OnlyADeviceReportedClearKeepsTheCorrelationCascade) { EXPECT_FALSE(OpcuaPlugin::clear_is_link_state(Origin::DeviceAlarm)); } +TEST(LinkStateClearPermitted, EverySourceMustBeOneThisProcessReportedUnder) { + const std::string mine = "siemens_ag_cpu_1505sp_f"; + const std::string placeholder = "opcua-192_168_1_10"; + const std::unordered_set my_ids{mine, placeholder, NodeMap::kDefaultComponentId}; + + // The restart case, which is why the decision cannot rest on this process's + // own memory: the fault persisted across a gateway restart and still names + // the component that raised it. + EXPECT_TRUE(OpcuaPlugin::link_state_clear_permitted(/*fault_found=*/true, {mine}, my_ids)); + // A fault raised while the device had not named itself carries the stand-in + // the plugin was serving then, and the same process is the one healing it. + EXPECT_TRUE(OpcuaPlugin::link_state_clear_permitted(true, {placeholder}, my_ids)); + EXPECT_TRUE(OpcuaPlugin::link_state_clear_permitted(true, {placeholder, mine}, my_ids)); + EXPECT_TRUE(OpcuaPlugin::link_state_clear_permitted(true, {NodeMap::kDefaultComponentId}, my_ids)); + + // ClearFault carries no source and clears the whole row, so a fault another + // bridge also holds is not this link's to clear - whether it is the only + // source or one of several. + EXPECT_FALSE(OpcuaPlugin::link_state_clear_permitted(true, {"beckhoff_cx5140"}, my_ids)) + << "another bridge's outage was cleared by this link coming back"; + EXPECT_FALSE(OpcuaPlugin::link_state_clear_permitted(true, {"beckhoff_cx5140", mine}, my_ids)) + << "a row two bridges hold was cleared whole"; + // A fault with no sources says nothing about who holds it. + EXPECT_FALSE(OpcuaPlugin::link_state_clear_permitted(true, {}, my_ids)); + // Nothing standing: the normal case on a healthy start, and nothing to send. + EXPECT_FALSE(OpcuaPlugin::link_state_clear_permitted(/*fault_found=*/false, {mine}, my_ids)); + // No ids to compare against is not a match. + EXPECT_FALSE(OpcuaPlugin::link_state_clear_permitted(true, {mine}, {})); +} + +TEST(CommsLostAnswerApplicable, AnAnswerFromAnEarlierSessionIsNotActedOn) { + // The answer describes the store at the moment the probe was served. + EXPECT_TRUE(OpcuaPlugin::comms_lost_answer_applicable(/*connected=*/true, /*has_answer=*/true, + /*probe_session=*/7, /*live_session=*/7)); + EXPECT_FALSE(OpcuaPlugin::comms_lost_answer_applicable(/*connected=*/false, /*has_answer=*/true, 7, 7)) + << "an answer was acted on while the link was down"; + EXPECT_FALSE(OpcuaPlugin::comms_lost_answer_applicable(/*connected=*/true, /*has_answer=*/false, 7, 7)); + + // The case "the link is up" cannot see on its own: the decision is driven + // from publish_values, which the poll loop reaches only while connected, so + // an answer parked just before a drop is first looked at on the tick AFTER + // the reconnect - link up, answer stale, and the outage it describes already + // reported by the poller. + EXPECT_FALSE(OpcuaPlugin::comms_lost_answer_applicable(/*connected=*/true, /*has_answer=*/true, + /*probe_session=*/7, /*live_session=*/8)) + << "an answer from the session before the outage decided the clear after the reconnect"; + EXPECT_FALSE(OpcuaPlugin::comms_lost_answer_applicable(true, true, 0, 1)); +} + +// --------------------------------------------------------------------------- +// The device-info read budget a session spends on its identity +// --------------------------------------------------------------------------- + +namespace { + +OpcuaClient::DeviceInfo nameplate_info() { + OpcuaClient::DeviceInfo info; + info.di_manufacturer = "Siemens AG"; + info.di_model = "CPU 1505SP F"; + return info; +} + +} // namespace + +TEST(IdentityReads, TheBudgetStopsTheReadsAndTheSpendIsCarried) { + int reads = 0; + const auto empty_read = [&reads]() { + ++reads; + return OpcuaClient::DeviceInfo{}; + }; + const auto no_pause = [](std::chrono::milliseconds) { + return true; + }; + + // A nameplate-less server: every attempt is spent and none of them settles. + const auto first = OpcuaPlugin::identity_reads(/*extra_reads=*/0, /*reads_spent=*/0, + OpcuaPlugin::kMaxIdentityPollReads, empty_read, no_pause); + EXPECT_TRUE(first.read_made); + EXPECT_EQ(first.reads_spent, 1); + EXPECT_EQ(reads, 1); + + int spent = first.reads_spent; + for (int i = 1; i < OpcuaPlugin::kMaxIdentityPollReads; ++i) { + spent = OpcuaPlugin::identity_reads(0, spent, OpcuaPlugin::kMaxIdentityPollReads, empty_read, no_pause).reads_spent; + } + EXPECT_EQ(spent, OpcuaPlugin::kMaxIdentityPollReads); + EXPECT_EQ(reads, OpcuaPlugin::kMaxIdentityPollReads); + + // Past the bound the server is not asked again. + const auto past = OpcuaPlugin::identity_reads(0, spent, OpcuaPlugin::kMaxIdentityPollReads, empty_read, no_pause); + EXPECT_FALSE(past.read_made) << "the budget was spent and the device was asked anyway"; + EXPECT_EQ(reads, OpcuaPlugin::kMaxIdentityPollReads) << "a read was made past the bound"; + EXPECT_EQ(past.reads_spent, OpcuaPlugin::kMaxIdentityPollReads); +} + +TEST(IdentityReads, TheConnectBurstPausesBetweenAttemptsAndStopsOnANameplate) { + std::vector pauses; + const auto record_pause = [&pauses](std::chrono::milliseconds pause) { + pauses.push_back(pause); + return true; + }; + + // A server whose address space is still filling in: the burst keeps asking, + // kIdentityConnectRetryPause apart. + int reads = 0; + const auto empty_read = [&reads]() { + ++reads; + return OpcuaClient::DeviceInfo{}; + }; + const auto burst = OpcuaPlugin::identity_reads(OpcuaPlugin::kIdentityConnectRetries, /*reads_spent=*/0, + OpcuaPlugin::kIdentityConnectRetries + 1, empty_read, record_pause); + EXPECT_EQ(reads, OpcuaPlugin::kIdentityConnectRetries + 1); + ASSERT_EQ(pauses.size(), static_cast(OpcuaPlugin::kIdentityConnectRetries)); + for (const auto & pause : pauses) { + EXPECT_EQ(pause, OpcuaPlugin::kIdentityConnectRetryPause); + } + EXPECT_FALSE(burst.interrupted); + + // A server that names itself on the second read costs one pause, not three. + pauses.clear(); + int attempt = 0; + const auto late_nameplate = [&attempt]() { + return ++attempt >= 2 ? nameplate_info() : OpcuaClient::DeviceInfo{}; + }; + const auto settled = OpcuaPlugin::identity_reads( + OpcuaPlugin::kIdentityConnectRetries, 0, OpcuaPlugin::kIdentityConnectRetries + 1, late_nameplate, record_pause); + EXPECT_EQ(attempt, 2); + EXPECT_EQ(pauses.size(), 1u); + EXPECT_TRUE(component_identity_has_nameplate(settled.info)); +} + +TEST(IdentityReads, ARefusedPauseEndsTheBurst) { + // What a shutdown does to the burst: the poll thread is what stop() joins, so + // it must not sit out the remaining pauses. + int reads = 0; + const auto empty_read = [&reads]() { + ++reads; + return OpcuaClient::DeviceInfo{}; + }; + const auto refuse = [](std::chrono::milliseconds) { + return false; + }; + const auto outcome = OpcuaPlugin::identity_reads(OpcuaPlugin::kIdentityConnectRetries, 0, + OpcuaPlugin::kIdentityConnectRetries + 1, empty_read, refuse); + EXPECT_EQ(reads, 1) << "the burst carried on after the pause was refused"; + EXPECT_TRUE(outcome.interrupted); + EXPECT_TRUE(outcome.read_made); +} + +TEST(CommsLostAnswerApplicable, AProbeGivenUpOnHasItsAnswerDropped) { + // The generation the timeout branch moves on, read from the consume side: an + // answer parked against a probe number the poll thread is no longer waiting + // for is not the answer to the probe that replaced it. rclcpp takes a pending + // entry out before it invokes the callback and outside its own mutex, so a + // callback that won that race is still on its way in when the timeout fires. + EXPECT_TRUE(OpcuaPlugin::comms_lost_generation_current(/*answered_generation=*/4, /*current_generation=*/4)); + EXPECT_FALSE(OpcuaPlugin::comms_lost_generation_current(/*answered_generation=*/4, /*current_generation=*/5)) + << "an answer to a probe already given up on was read as the next probe's"; +} + +TEST(CommsLostProbeDue, AllFourConditionsHold) { + // A probe costs a round trip and its answer decides whether a fault is + // cleared, so each input carries its own veto. The connected term is the one + // that keeps a clear off a link that is down. + EXPECT_TRUE(OpcuaPlugin::comms_lost_probe_due(/*owed=*/true, /*connected=*/true, /*probe_in_flight=*/false, + /*store_ready=*/true)); + EXPECT_FALSE(OpcuaPlugin::comms_lost_probe_due(false, true, false, true)) << "nothing is owed"; + EXPECT_FALSE(OpcuaPlugin::comms_lost_probe_due(true, false, false, true)) + << "a clear decided while the link is down would clear a fault that is genuinely standing"; + EXPECT_FALSE(OpcuaPlugin::comms_lost_probe_due(true, true, true, true)) << "a probe is already outstanding"; + EXPECT_FALSE(OpcuaPlugin::comms_lost_probe_due(true, true, false, false)) << "the store cannot be asked"; +} + TEST(MakeClearFaultRequest, CarriesTheSkipFlagAndCodeVerbatim) { const auto skipping = OpcuaPlugin::make_clear_fault_request(kCommsLostFaultCode, true); EXPECT_EQ(skipping.fault_code, kCommsLostFaultCode); @@ -1487,23 +1809,63 @@ TEST(EnqueuePendingDispatch, ADeviceAlarmClearIsNotEvictedAheadOfAnOlderReport) EXPECT_LT(raise - buffer.begin(), device_clear - buffer.begin()); } -TEST(EnqueuePendingDispatch, ARequeuedClearMovesToTheBackSoOrderStillHolds) { - // Report-then-clear for one code must still flush in that order after the - // clear is re-enqueued, or the flush would leave the fault standing. +TEST(EnqueuePendingDispatch, AClearBehindAReportIsAppendedSoThePlcOrderHolds) { + // Report-then-clear for one code flushes in that order after the clear is + // re-enqueued, or the flush leaves the fault standing. std::vector buffer; OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, link_state_clear_entry("PLC_FLAP")); OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_FLAP")); - EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, - link_state_clear_entry("PLC_FLAP")), - OpcuaPlugin::PendingEnqueueOutcome::ReplacedClear); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, link_state_clear_entry("PLC_FLAP")); - ASSERT_EQ(buffer.size(), 2u); - EXPECT_EQ(buffer[0].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Report); - EXPECT_EQ(buffer[1].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear) + // The stale leading clear predates the report, so it is not the newest entry + // for this code and is not what the incoming clear supersedes. It stays, and + // the pair the PLC produced keeps its order behind it. + ASSERT_EQ(buffer.size(), 3u); + EXPECT_EQ(buffer[0].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear); + EXPECT_EQ(buffer[1].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Report); + EXPECT_EQ(buffer[2].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear) << "the newest clear must flush after the report it supersedes"; // Clears for DIFFERENT codes are independent. OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, device_clear_entry("PLC_OTHER")); - EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 2u); + EXPECT_EQ(count_kind(buffer, OpcuaPlugin::PendingFaultDispatch::Kind::Clear), 3u); +} + +TEST(EnqueuePendingDispatch, RepeatedReconnectClearsCollapseOnlyWhileNothingWasReportedBetween) { + // Back-to-back reconnect clears with no report between them are the same + // statement made twice: one is enough, and each one after the first says so. + std::vector buffer; + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + link_state_clear_entry(kCommsLostFaultCode)), + OpcuaPlugin::PendingEnqueueOutcome::Buffered); + for (int i = 0; i < 4; ++i) { + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + link_state_clear_entry(kCommsLostFaultCode)), + OpcuaPlugin::PendingEnqueueOutcome::ReplacedClear); + } + EXPECT_EQ(buffer.size(), 1u) << "repeated reconnect clears must still collapse to one"; +} + +TEST(EnqueuePendingDispatch, AClearNeverCoalescesPastAReportForTheSameCode) { + // The PLC raised, cleared, raised and cleared the same code while the fault + // manager was unreachable. Coalescing the second clear onto the first one - + // anywhere in the buffer - flushes Report, Report, Clear, and the second + // raise then stands CONFIRMED for good against a device reporting it + // inactive. Each pair has to survive in the order the device produced it. + std::vector buffer; + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_TANK_HIGH")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + device_clear_entry("PLC_TANK_HIGH")); + OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, report_entry("PLC_TANK_HIGH")); + EXPECT_EQ(OpcuaPlugin::enqueue_pending_dispatch(buffer, OpcuaPlugin::kMaxPendingDispatches, + device_clear_entry("PLC_TANK_HIGH")), + OpcuaPlugin::PendingEnqueueOutcome::Buffered) + << "a clear behind a newer report is a new edge, not a supersede"; + + ASSERT_EQ(buffer.size(), 4u) << "the second raise/clear pair was coalesced onto the first"; + EXPECT_EQ(buffer[0].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Report); + EXPECT_EQ(buffer[1].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear); + EXPECT_EQ(buffer[2].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Report); + EXPECT_EQ(buffer[3].kind, OpcuaPlugin::PendingFaultDispatch::Kind::Clear); } TEST(AdoptRediscoveredEndpoint, AdoptsOnlyADifferentNonEmptyUrl) { diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_poller.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_poller.cpp index 9e9cf5575..e3c6c54a5 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_poller.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_poller.cpp @@ -21,6 +21,8 @@ #include +#include +#include #include namespace ros2_medkit_gateway { @@ -150,6 +152,101 @@ TEST(NodeIdsEquivalentTest, UnparseableSpellingsFallBackToRawEquality) { EXPECT_TRUE(OpcuaPoller::node_ids_equivalent("not-a-node-id", "not-a-node-id")); } +TEST(AlarmRoutingTest, TheEventPathsCopyDoesNotFollowARenameItNeverAskedFor) { + // on_event runs on the event pump thread; the config-less rename runs on the + // poll thread and clears then reassigns auto_alarms.entity_id under a lock the + // event path neither holds nor can take. Reading the node map from on_event + // is therefore a data race on a std::string, and a ConditionRefresh burst on + // the first adopted session lands on exactly that window. The poller keeps its + // own copy, and only the thread that renames replaces it. + OpcuaClient client; // never connected: the routing copy is pure bookkeeping + NodeMap node_map; + node_map.set_component_identity("opcua-127_0_0_1", "opcua-127_0_0_1"); + node_map.mutable_auto_alarms().enabled = true; + ASSERT_TRUE(node_map.finalize_auto_alarms_overlay()); + const std::string placeholder_entity = node_map.auto_alarms().entity_id; + ASSERT_EQ(placeholder_entity, "opcua-127_0_0_1_alarms"); + + OpcuaPoller poller(client, node_map); + poller.refresh_alarm_routing(); + const auto subscribed_with = poller.alarm_routing(); + ASSERT_TRUE(subscribed_with); + EXPECT_EQ(subscribed_with->auto_alarms.entity_id, placeholder_entity); + + // The rename the poll thread performs once the adopted device names itself. + node_map.mutable_auto_alarms().entity_id.clear(); + node_map.set_component_identity("siemens_ag_cpu_1505sp_f", "Siemens AG CPU 1505SP F"); + ASSERT_TRUE(node_map.finalize_auto_alarms_overlay()); + ASSERT_EQ(node_map.auto_alarms().entity_id, "siemens_ag_cpu_1505sp_f_alarms"); + + // The event path still reads what it was handed. Both the snapshot it already + // holds and a fresh read of the accessor: the copy is what the poller owns, + // not a view onto the map. + EXPECT_EQ(subscribed_with->auto_alarms.entity_id, placeholder_entity); + EXPECT_EQ(poller.alarm_routing()->auto_alarms.entity_id, placeholder_entity) + << "the event path's copy tracked a rename it never asked for - it is reading the node map"; + + // ... until the thread that renamed replaces it, which is what + // setup_event_subscriptions does at subscribe time. + poller.refresh_alarm_routing(); + EXPECT_EQ(poller.alarm_routing()->auto_alarms.entity_id, "siemens_ag_cpu_1505sp_f_alarms"); + EXPECT_EQ(subscribed_with->auto_alarms.entity_id, placeholder_entity) + << "a refresh rewrote the snapshot a callback was already holding"; +} + +TEST(AlarmRoutingTest, ARepinMovesConditionsAlreadyPinnedToTheNewEntity) { + // apply_condition_state pins a fault's entity at the first sighting of its + // ConditionId, so a config-less rename has to reach the conditions the poller + // already holds as well as the routing new ones are derived with. Without the + // re-pin, every later report and clear for those ConditionIds is filed under + // an entity the rename dropped. + OpcuaClient client; + NodeMap node_map; + OpcuaPoller poller(client, node_map); + + std::mutex deliveries_mutex; + std::vector deliveries; + poller.set_event_alarm_callback([&deliveries_mutex, &deliveries](const AlarmEventDelivery & delivery) { + std::lock_guard lock(deliveries_mutex); + deliveries.push_back(delivery); + }); + + AlarmEventConfig cfg; + cfg.source_node_id_str = "i=2253"; + cfg.entity_id = "opcua-127_0_0_1_alarms"; + cfg.fault_code = "PLC_OVERPRESSURE"; + const opcua::NodeId condition(3, static_cast(1845)); + + AlarmEventInput raise; + raise.enabled_state = true; + raise.active_state = true; + raise.active_state_present = true; + poller.apply_condition_state_for_test(cfg, condition, raise, /*severity=*/750, "Overpressure", + /*event_id=*/nullptr, /*require_confirm_for_clear=*/false); + { + std::lock_guard lock(deliveries_mutex); + ASSERT_EQ(deliveries.size(), 1u) << "the raise was not delivered, so the pin cannot be observed"; + EXPECT_EQ(deliveries.front().entity_id, "opcua-127_0_0_1_alarms"); + deliveries.clear(); + } + + poller.repin_auto_alarms_entity("opcua-127_0_0_1_alarms", "siemens_ag_cpu_1505sp_f_alarms"); + + // The same condition going inactive. Its entity is the renamed one, so the + // clear reaches the entity the raise will have been moved to. + AlarmEventInput heal = raise; + heal.active_state = false; + heal.acked_state = true; + poller.apply_condition_state_for_test(cfg, condition, heal, /*severity=*/750, "Overpressure", + /*event_id=*/nullptr, /*require_confirm_for_clear=*/false); + + std::lock_guard lock(deliveries_mutex); + ASSERT_EQ(deliveries.size(), 1u); + EXPECT_EQ(deliveries.front().entity_id, "siemens_ag_cpu_1505sp_f_alarms") + << "a condition pinned before the rename kept the entity the rename dropped"; + EXPECT_EQ(deliveries.front().fault_code, "PLC_OVERPRESSURE"); +} + TEST(IsConditionEventTest, NullConditionIdIsRejected) { // Part 9 §5.5.2.13: a non-condition event (e.g. a Siemens Server-object // system message such as "CPU not in RUN") resolves the ConditionId SAO From 77303bb4e04dfe8a40a4b1f19aa72c1d4b45de67 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 15 Sep 2026 14:51:58 +0200 Subject: [PATCH 17/25] opcua: make the discovery race scenario assert the rename and drop the restart checks The config-less pass asserts that the stand-in component left /components in the same sample that shows the renamed one. The containers run with no restart policy, so a gateway that dies is reported by the endpoint and component assertions; the RestartCount checks are gone. The deadline follows RESCAN_INTERVAL_S like its siblings. --- .../docker/scripts/run_discovery_race_test.sh | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh index 5b807472b..b4c3ccebf 100755 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh @@ -215,12 +215,9 @@ done || fail "endpoint still '${endpoint}' (connected='${connected}') after $((2 * RESCAN_INTERVAL_S + 40))s" echo " OK re-scan adopted ${adopted} without a gateway restart" -# The gateway must have adopted the server in the process that started before -# it, not in a fresh one: a restarted container would pass the check above -# while proving nothing. -restarts="$(docker inspect -f '{{.RestartCount}}' "${GATEWAY_NAME}")" -[[ "${restarts}" == "0" ]] || fail "gateway restarted ${restarts} time(s) during the run" -echo " OK gateway never restarted" +# The gateway containers run with no --restart policy, so a gateway that died +# stays dead and the endpoint and component assertions above are what report it: +# the REST API stops answering and they fail. Nothing separate is checked. # --------------------------------------------------------------------------- # Config-less variant: the same race with NO node map. @@ -310,12 +307,25 @@ connected="$(status_field_for "${renamed}" connected)" || fail "renamed component reports endpoint '${endpoint}', expected the adopted server" [[ "${connected}" == "True" ]] \ || fail "renamed component reports connected='${connected}', expected a live session" -[[ "${renamed}" != "${FALLBACK_COMPONENT_ID}" ]] \ - || fail "component id never moved off ${FALLBACK_COMPONENT_ID}" echo " OK component renamed to ${renamed}, connected at ${endpoint}" -restarts="$(docker inspect -f '{{.RestartCount}}' "${GATEWAY_NAME}")" -[[ "${restarts}" == "0" ]] || fail "gateway restarted ${restarts} time(s) during the config-less run" -echo " OK gateway never restarted" +# The provisional component must LEAVE /components while the renamed one is +# there, read from ONE sample: two components for one PLC is an entity tree the +# operator has to disambiguate, and everything already filed under the old id +# would point at a component nothing polls. Given its own deadline because the +# rename and the discovery refresh that republishes entities are a cycle apart. +DEADLINE=$((SECONDS + 2 * RESCAN_INTERVAL_S + 40)) +settled="" +while [[ ${SECONDS} -lt ${DEADLINE} ]]; do + ids="$(component_ids)" + if [[ " ${ids} " == *" ${renamed} "* && " ${ids} " != *" ${FALLBACK_COMPONENT_ID} "* ]]; then + settled="${ids}" + break + fi + sleep 2 +done +[[ -n "${settled}" ]] \ + || fail "expected '${renamed}' served and '${FALLBACK_COMPONENT_ID}' gone in one sample, got '${ids}'" +echo " OK ${renamed} is served and ${FALLBACK_COMPONENT_ID} is gone" echo "Discovery race scenario passed." From 846885391f84f644eef717f5e5d0ddf01909194a Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 09:56:57 +0200 Subject: [PATCH 18/25] opcua: keep the bound server across a restart, steer the sweeps by it, and own every reconnect Binding file. When the binding is established, the plugin writes the bound ApplicationUri into discovery.binding_file. The default path is /var/lib/ros2_medkit/opcua/binding. OPCUA_DISCOVERY_BINDING_FILE sets it, and an empty value turns it off. configure() reads the file, so a restart keeps the identity. The start-up sweep and every rescan then select only the bound server, at any address. A different server at the bound address is dropped at connect, and the outage stays standing. The refusal names the file as the way to recommission the box. A configured endpoint_url neither reads nor writes the file. A server that publishes no ApplicationUri is never persisted. The file is written to a temporary file, fsynced and renamed over the resolved target. It holds one line. On read, a UTF-8 BOM and the blanks around the URI are stripped, and anything after the first line is ignored. A control character in the first line is refused with a warning and reads as no binding. A URI with a line break is refused at the write. Reconnects. The reconnect arm owns every reconnect. After each client iterate the client reads its session state. A session that the library opened again on its own is dropped and handed to the arm. The arm connects through its own path, runs the binding check and creates the alarm subscription again. Every path that marks the client disconnected drops the subscription bookkeeping of the dead session. The new monitored item therefore delivers events again after a PLC reboot. Tests. Unit tests cover the file format and the write contract. End-to-end tests against the fixture cover: - a restarted plugin that refuses a foreign server at the bound address - a configured endpoint_url that ignores the file - the start-up sweep that picks the bound server over a lower address - a swap and an outage under a config-less session - a device alarm delivered after the bound server reboots The ctest timeout of the identity E2E is twice the idle pace plus one stuck test's wait budget, net of its passing time. The registration shows the arithmetic. --- .../ros2_medkit_opcua/CMakeLists.txt | 11 +- .../ros2_medkit_opcua/network_discovery.hpp | 14 + .../ros2_medkit_opcua/opcua_client.hpp | 19 + .../ros2_medkit_opcua/opcua_plugin.hpp | 58 +- .../src/network_discovery.cpp | 6 +- .../ros2_medkit_opcua/src/opcua_client.cpp | 98 ++- .../ros2_medkit_opcua/src/opcua_plugin.cpp | 218 ++++++- .../ros2_medkit_opcua/src/opcua_poller.cpp | 15 +- .../test/test_opcua_identity.cpp | 608 +++++++++++++++++- .../test/test_opcua_plugin.cpp | 196 +++++- 10 files changed, 1177 insertions(+), 66 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/CMakeLists.txt b/src/ros2_medkit_plugins/ros2_medkit_opcua/CMakeLists.txt index 698e41733..c0f5b2518 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/CMakeLists.txt +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/CMakeLists.txt @@ -588,6 +588,15 @@ if(BUILD_TESTING) # INV2: end-to-end identity test. Boots test_alarm_server as a subprocess and # proves connect -> device-info read -> AssetIdentity via both OpcuaClient and # OpcuaPlugin::introspect(). GTEST_SKIP when the fixture binary is missing. + # + # TIMEOUT arithmetic. A passing run of this binary is 94 s on an idle box + # (the 27 E2Es take 89 s, 18 s of them fixed settle sleeps the tests carry). + # A failing test runs its whole wait budget before it reports; the largest + # is TheBridgeStaysBoundToItsOwnServerAcrossAddressAndSwap at 180 s (six + # 30 s waits) against 11 s when it passes. Twice the idle pace for a loaded + # or sanitizer runner (188 s) plus that one budget net of its passing time + # (169 s) is 357 s, so 360 lets one stuck test reach its own FAIL + # diagnostics before CTest kills the run. medkit_add_gtest(test_opcua_identity test/test_opcua_identity.cpp src/opcua_plugin.cpp @@ -598,7 +607,7 @@ if(BUILD_TESTING) src/network_discovery.cpp src/network_discovery_io.cpp src/address_space_browser.cpp - TIMEOUT 120 + TIMEOUT 360 ) add_dependencies(test_opcua_identity test_alarm_server) target_include_directories(test_opcua_identity PRIVATE diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp index 51076717a..7b63841eb 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/network_discovery.hpp @@ -122,6 +122,20 @@ struct OpcuaDiscoveryConfig { /// the plugin connects with today). Secured-only servers are surfaced as /// leads requiring operator credentials, never auto-connected. bool anonymous_none_only{true}; + + /// Where the ApplicationUri of the server the plugin is bound to is kept, so + /// the binding survives a restart and a restarted process does not adopt + /// whichever server answers. One line, the URI. Empty disables persistence, + /// which makes every process's first adoption unconstrained again. The + /// default sits under /var/lib/ros2_medkit, which the shipped image creates + /// for the fault manager database and declares no VOLUME for, so surviving a + /// re-created container (``docker restart`` keeps the writable layer, a new + /// ``docker run`` starts without it) needs the operator to mount that + /// directory. A path whose + /// directory cannot be created or written is reported once and the process + /// runs without persistence. Read and written only while endpoint_url is + /// unset. + std::string binding_file = "/var/lib/ros2_medkit/opcua/binding"; }; /// Probe whether a TCP port is open. Injected so the orchestrator is unit diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp index 9031f39a4..42ab9ab31 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_client.hpp @@ -481,6 +481,25 @@ class OpcuaClient { private: struct Impl; std::unique_ptr impl_; + + /// Drops the bookkeeping of a session that is gone, with client_mutex held: + /// the generation moves first so an in-flight trampoline from the old + /// subscription drops its work, then the event contexts and subscription + /// handles are released. Nothing is deleted server-side, because the session + /// that held them is gone. Every path that marks the client disconnected + /// ends here, and connect() runs it before a new session is published, so + /// the monitored-item ids the next session hands out, which start from 1 + /// again, register fresh contexts under their own ids. + void drop_session_bookkeeping_locked(); + + /// Marks the client disconnected when ``e`` is a transport-level loss + /// (connection closed, secure channel closed, not connected) and drops the + /// session's bookkeeping; a per-node error such as BadNodeIdUnknown leaves + /// the session up. Called with client_mutex held from every operation that + /// can observe the drop, so OpcuaPoller's reconnect logic, keyed off + /// is_connected(), fires whichever operation saw it first. + /// @return true when this call flipped the client to disconnected. + bool mark_disconnected_on_transport_error(const opcua::BadStatus & e); }; } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index 619c5d5e5..d294bdfb0 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -157,6 +157,16 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return rescan_refusals_.load(); } + /// The endpoint the client is pointed at, and whether a session is up. Read + /// by tests that have to see which address a sweep settled on, which no + /// stubbed route response carries. + std::string endpoint_url_for_test() const { + return client_ ? client_->endpoint_url() : client_config_.endpoint_url; + } + bool connected_for_test() const { + return client_ && client_->is_connected(); + } + /// How many sessions have been dropped for reaching a server other than the /// one this bridge is bound to. Read by tests that have to tell that drop /// apart from a connect that simply failed. @@ -164,6 +174,14 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return binding_mismatch_disconnects_.load(); } + /// The warning the last binding refusal was logged with. Read by tests that + /// have to see which recommissioning gesture it names, which no counter + /// carries. + std::string last_binding_refusal_for_test() const { + std::lock_guard lock(binding_refusal_mutex_); + return last_binding_refusal_; + } + /// The address-space walk configuration after configure() has merged the /// node map and the ROS parameters. Read by tests that need to see which /// source supplied a setting, which no REST response exposes. @@ -381,7 +399,7 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, /// ``auto_alarms`` condition, or a threshold rule going false). A one-shot /// edge nothing will re-send, and a real resolution, so the cascade stands. DeviceAlarm, - /// The OPC-UA session came back, so ``PLC_COMMS_LOST`` no longer holds. + /// The OPC-UA session came back, which ends the ``PLC_COMMS_LOST`` outage. /// Re-derived on the next reconnect if it is lost, and not an operator /// resolving a root cause, so it must not cascade. LinkState, @@ -423,6 +441,27 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, static bool link_state_clear_permitted(bool fault_found, const std::vector & reporting_sources, const std::unordered_set & my_ids); + // The ApplicationUri kept in ``path``, or empty when there is none to read. + // + // Empty for an empty ``path`` (persistence disabled), for a file that is + // absent or unreadable, and for one whose first line is blank. Trailing + // whitespace and the newline are stripped, so a file an operator edited by + // hand reads the same as one this plugin wrote. Pure I/O + static so the + // round trip is testable without a plugin. + static std::string read_persisted_binding(const std::string & path); + + // Write ``application_uri`` to ``path`` as the only line, atomically: a + // temporary beside it, then a rename, so a reader never sees half a URI and a + // crash mid-write leaves the previous binding intact. + // + // The parent directory is created when it is missing. Returns an empty string + // on success, otherwise the reason, which the caller reports: a binding that + // cannot be persisted costs the next process its constraint, and nothing + // else - the session it was established on is unaffected. An empty ``path`` + // or an empty ``application_uri`` writes nothing and succeeds: there is no + // binding to keep. + static std::string write_persisted_binding(const std::string & path, const std::string & application_uri); + // Whether a parked answer belongs to the probe the poll thread is waiting for. // // Each probe carries a number, stamped when the request goes out. The timeout @@ -760,11 +799,14 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // (OpcuaClient::read_server_application_uri). It is what discovery looks for // on every later sweep and what a fresh session is checked against. // - // Empty until a session has been held: a process that has never connected is - // bound to nothing, so its first adoption is unconstrained. It is also empty - // for a server that publishes no ApplicationUri, which therefore cannot be - // bound to. A restart clears it, because it is never persisted. Written on - // the set_context thread, then the poll thread. + // Empty until a session has been held or a binding was read from the file: a + // process that has neither is bound to nothing, so its first adoption is + // unconstrained. It is also empty for a server that publishes no + // ApplicationUri, which therefore cannot be bound to. On the discovery path + // it is read from discovery.binding_file in configure() and written there on + // the first bind, so it survives a restart; with endpoint_url configured the + // file is neither read nor written. Written on the set_context thread, then + // the poll thread. std::string bound_application_uri_; // ApplicationUris already reported as not this bridge's, so one outage does // not log the same foreign server every interval_s. Bounded, and cleared on @@ -775,6 +817,10 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // for reaching a different server (see the *_for_test accessors). std::atomic rescan_refusals_{0}; std::atomic binding_mismatch_disconnects_{0}; + // The text of the last refusal warning, for the *_for_test accessor. Written + // on the set_context thread and the poll thread, read from test threads. + mutable std::mutex binding_refusal_mutex_; + std::string last_binding_refusal_; // Outcome digest of the previous discovery pass, so an unchanged rescan // reports at DEBUG and the whole report goes out once per change, not once diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp index 962758a49..ee1434790 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/network_discovery.cpp @@ -222,7 +222,8 @@ OpcuaDiscoveryConfig parse_discovery_config(const nlohmann::json & j, "scan_concurrency", "identify_timeout_ms", "interval_s", - "anonymous_none_only"}; + "anonymous_none_only", + "binding_file"}; for (const auto & item : j.items()) { if (std::find(kKnown.begin(), kKnown.end(), item.key()) == kKnown.end()) { warn_fn("discovery: unknown key '" + item.key() + "' ignored"); @@ -232,6 +233,9 @@ OpcuaDiscoveryConfig parse_discovery_config(const nlohmann::json & j, if (j.contains("enabled") && j["enabled"].is_boolean()) { cfg.enabled = j["enabled"].get(); } + if (j.contains("binding_file") && j["binding_file"].is_string()) { + cfg.binding_file = j["binding_file"].get(); + } if (j.contains("mode") && j["mode"].is_string()) { const std::string mode = j["mode"].get(); if (mode == "active" || mode == "passive" || mode == "both") { diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp index fe5c3c476..263b537af 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_client.cpp @@ -94,24 +94,6 @@ struct EventCallbackContext { namespace { -/// Set the connected flag to false when the BadStatus code indicates a -/// terminal connection loss (as opposed to e.g. BadNodeIdUnknown which is a -/// per-node issue). Called from read_value, read_values and write_value so -/// that OpcuaPoller's reconnect logic (which keys off is_connected()) fires -/// regardless of which operation detected the drop first. Also bumps the -/// subscription generation so any in-flight event callbacks from the dying -/// subscription are filtered out by the trampoline. -void maybe_mark_disconnected(std::atomic & connected_flag, std::atomic & generation, - const opcua::BadStatus & e) { - const auto code = e.code(); - if (code == UA_STATUSCODE_BADCONNECTIONCLOSED || code == UA_STATUSCODE_BADSECURECHANNELCLOSED || - code == UA_STATUSCODE_BADNOTCONNECTED) { - if (connected_flag.exchange(false)) { - generation.fetch_add(1, std::memory_order_release); - } - } -} - OpcuaValue variant_to_value(const opcua::Variant & var) { if (var.isEmpty()) { return std::string(""); @@ -535,6 +517,11 @@ bool OpcuaClient::connect(const OpcuaClientConfig & config) { impl_->client.config().setTimeout(static_cast(config.connect_timeout.count())); impl_->client.connect(config.endpoint_url); + // Whatever the previous session left behind is dropped before this one is + // published: the ids this server hands out start from 1 again, and an + // event context still registered under one of them would keep the new + // context out of the map while open62541 holds a pointer to it. + drop_session_bookkeeping_locked(); impl_->connected = true; impl_->connect_generation.fetch_add(1, std::memory_order_release); @@ -563,9 +550,10 @@ void OpcuaClient::disconnect() { // dying subscription drop their work in the trampoline (they read // generation atomically) before we touch the storage they reference. // The ``if (impl_->connected)`` guard ensures we bump exactly once even - // when ``maybe_mark_disconnected`` already fired earlier on a transport - // error path - that helper uses ``exchange(false)`` and would have - // already bumped, leaving impl_->connected = false here. + // when ``mark_disconnected_on_transport_error`` already fired earlier on a + // transport error path - that helper uses ``exchange(false)`` and has + // already bumped and dropped the bookkeeping, leaving + // impl_->connected = false here. impl_->generation.fetch_add(1, std::memory_order_release); try { // Issue #386: clear event monitored items BEFORE deleting subscriptions. @@ -703,7 +691,7 @@ std::vector OpcuaClient::browse_detailed(const opcua:: result.push_back(std::move(child)); } } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); } return result; @@ -724,7 +712,7 @@ std::string OpcuaClient::read_variable_type_name(const opcua::NodeId & variable_ } return builtin_data_type_name(data_type.getIdentifierAs()); } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); return {}; } } @@ -756,7 +744,7 @@ OpcuaClient::AccessLevelInfo OpcuaClient::read_access_level(const opcua::NodeId info.writable = effective.anyOf(opcua::AccessLevel::CurrentWrite); info.ok = true; } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); } return info; @@ -780,7 +768,7 @@ ReadResult OpcuaClient::read_value(const opcua::NodeId & node_id) { result.good = true; } catch (const opcua::BadStatus & e) { result.good = false; - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); } return result; @@ -809,7 +797,7 @@ std::vector OpcuaClient::read_values(const std::vectorconnected, impl_->generation, e); + mark_disconnected_on_transport_error(e); } results.push_back(std::move(r)); } @@ -981,7 +969,7 @@ OpcuaClient::write_value(const opcua::NodeId & node_id, const OpcuaValue & value } return {}; } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); auto code = e.code(); if (code == UA_STATUSCODE_BADTYPEMISMATCH) { return tl::make_unexpected(WriteErrorInfo{WriteError::TypeMismatch, e.what()}); @@ -1273,7 +1261,7 @@ std::vector OpcuaClient::read_source_condit *scan_ok = true; } } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); // scan_ok stays false: a browse failure must not be read as "no conditions". } @@ -1696,8 +1684,58 @@ void OpcuaClient::run_iterate(uint16_t timeout_ms) { try { impl_->client.runIterate(timeout_ms); } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); + return; + } + // runIterate drives open62541's own connectIterate whenever the session is + // below ACTIVATED, so a channel that dropped is re-opened and a session + // activated against whatever answers the address now - underneath this + // client, with no connect() of ours and nothing to check what it reached. + // Reading the state here is what turns that into a disconnect the owner's + // reconnect path handles: the session it rebuilds is one it opened and + // checked, and the subscriptions it lost are re-created there. + UA_SecureChannelState channel_state = UA_SECURECHANNELSTATE_CLOSED; + UA_SessionState session_state = UA_SESSIONSTATE_CLOSED; + UA_StatusCode connect_status = UA_STATUSCODE_GOOD; + UA_Client_getState(impl_->client.handle(), &channel_state, &session_state, &connect_status); + if (session_state >= UA_SESSIONSTATE_ACTIVATED) { + return; } + if (!impl_->connected.exchange(false)) { + return; + } + drop_session_bookkeeping_locked(); + RCLCPP_WARN(opcua_client_logger(), + "OPC-UA session dropped underneath the client (session state %d, channel %d, status %s); " + "disconnecting so the reconnect path owns the next session", + static_cast(session_state), static_cast(channel_state), UA_StatusCode_name(connect_status)); + try { + impl_->client.disconnect(); + } catch (...) { + } +} + +void OpcuaClient::drop_session_bookkeeping_locked() { + impl_->generation.fetch_add(1, std::memory_order_release); + { + std::lock_guard ev_lock(impl_->event_callbacks_mutex); + impl_->event_callbacks.clear(); + } + std::lock_guard sub_lock(impl_->sub_mutex); + impl_->subscriptions.clear(); +} + +bool OpcuaClient::mark_disconnected_on_transport_error(const opcua::BadStatus & e) { + const auto code = e.code(); + if (code != UA_STATUSCODE_BADCONNECTIONCLOSED && code != UA_STATUSCODE_BADSECURECHANNELCLOSED && + code != UA_STATUSCODE_BADNOTCONNECTED) { + return false; + } + if (!impl_->connected.exchange(false)) { + return false; + } + drop_session_bookkeeping_locked(); + return true; } uint32_t OpcuaClient::add_event_monitored_item(uint32_t subscription_id, const opcua::NodeId & source_node, @@ -1908,7 +1946,7 @@ OpcuaClient::call_method(const opcua::NodeId & object_id, const opcua::NodeId & } return outputs; } catch (const opcua::BadStatus & e) { - maybe_mark_disconnected(impl_->connected, impl_->generation, e); + mark_disconnected_on_transport_error(e); return tl::make_unexpected(status_to_method_error(e.code(), e.what())); } } diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 6b7c0b465..e4b5d9a6b 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -33,16 +33,38 @@ #include #include #include +#include +#include +#include + +#include +#include #include #include #include #include +#include #include namespace ros2_medkit_gateway { namespace { +// Flush one path's data to the device. Returns the reason on failure. +std::string fsync_path(const std::string & path, bool is_directory) { + const int fd = ::open(path.c_str(), is_directory ? (O_RDONLY | O_DIRECTORY) : O_WRONLY); + if (fd < 0) { + return "cannot open " + path + " to flush it: " + std::strerror(errno); + } + const int rc = ::fsync(fd); + const int saved = errno; + ::close(fd); + if (rc != 0) { + return "flushing " + path + " failed: " + std::strerror(saved); + } + return {}; +} + // Named logger so per-operation traces respect ROS log level filtering // (bburda review on PR #387). Quiet at INFO; ``--log-level // opcua.plugin:=debug`` re-enables for diagnostics. @@ -469,6 +491,23 @@ void OpcuaPlugin::configure(const nlohmann::json & config) { } } + if (auto * env = std::getenv("OPCUA_DISCOVERY_BINDING_FILE")) { + discovery_config_.binding_file = env; + } + // Before the startup sweep, which the binding steers: a process that was + // bound when it stopped looks for that server and no other. + // Discovery's binding, and only discovery's: an operator who pinned + // endpoint_url has already said which server this is, and re-pointing it at a + // replacement is the whole gesture. A file left behind by an earlier + // config-less run must not refuse the server the operator named. + if (!endpoint_configured_) { + bound_application_uri_ = read_persisted_binding(discovery_config_.binding_file); + if (!bound_application_uri_.empty()) { + log_info("OPC-UA: bound to uri='" + bound_application_uri_ + "' from " + discovery_config_.binding_file + + ". Remove that file and restart to bind to a different server."); + } + } + // Default the discovery I/O to the real probes; test builds override these // before set_context() to exercise the auto-endpoint path offline. if (!discovery_scan_fn_) { @@ -647,9 +686,9 @@ void OpcuaPlugin::set_context(PluginContext & context) { } #endif - // The session is bound to the server it reached, read off the session itself. - // Nothing is bound yet on this path, so this only records the identity; the - // check it performs matters on every later connect. + // The session is checked against the binding, which on the discovery path may + // already have been read from discovery.binding_file, and establishes it when + // there is none. A session that reached a different server is dropped here. const bool connected = client_->connect(client_config_) && bind_or_drop_session(); if (connected) { log_info("Connected to OPC-UA server: " + client_config_.endpoint_url); @@ -2287,10 +2326,18 @@ void OpcuaPlugin::run_startup_discovery() { // executor spins, so nothing can call shutdown() until this returns. What ends // it is the SIGINT / SIGTERM that rclcpp's own handler turns into // !rclcpp::ok() - see discovery_cancelled(). - const auto chosen = discover_endpoint(discovery_config_, endpoint_configured_, discovery_scan_fn_, - discovery_identify_fn_, discovery_reporter(&last_discovery_outcome_), [this]() { - return discovery_cancelled(); - }); + std::string selected_uri; + const auto chosen = discover_endpoint( + discovery_config_, endpoint_configured_, discovery_scan_fn_, discovery_identify_fn_, + discovery_reporter(&last_discovery_outcome_), + [this]() { + return discovery_cancelled(); + }, + // The binding steers this sweep the way it steers a rescan: a process + // that was bound when it stopped looks for that server at whatever + // address it now answers on, and selects nothing when it is absent. + &selected_uri, bound_application_uri_); + static_cast(selected_uri); // Stamp when the sweep FINISHED: the rescan cadence is measured from the end // of the previous sweep, so a long sweep is not immediately followed by // another one. @@ -2345,6 +2392,125 @@ OpcuaPlugin::DiscoveryReporter OpcuaPlugin::discovery_reporter(std::string * pre return reporter; } +std::string OpcuaPlugin::read_persisted_binding(const std::string & path) { + if (path.empty()) { + return {}; + } + std::ifstream in(path); + if (!in) { + return {}; + } + std::string line; + if (!std::getline(in, line)) { + return {}; + } + // A UTF-8 BOM is what an editor on Windows leaves in front of the URI. It is + // invisible in a log, so a binding carrying one shows the refusal as two + // identical URIs. + static const std::string kUtf8Bom = "\xEF\xBB\xBF"; + if (line.compare(0, kUtf8Bom.size(), kUtf8Bom) == 0) { + line.erase(0, kUtf8Bom.size()); + } + const auto end = line.find_last_not_of(" \t\r\n"); + if (end == std::string::npos) { + return {}; + } + line.erase(end + 1); + const auto begin = line.find_first_not_of(" \t"); + if (begin == std::string::npos) { + return {}; + } + line = line.substr(begin); + // An ApplicationUri is printable. A NUL or another control character means + // the file holds something else - a binary blob, a truncated write - and a + // binding read out of it would refuse every server for a reason nobody can + // see in a log. + const auto control = std::find_if(line.begin(), line.end(), [](char c) { + return static_cast(c) < 0x20 || static_cast(c) == 0x7F; + }); + if (control != line.end()) { + RCLCPP_WARN(opcua_plugin_logger(), + "%s holds a control character where an ApplicationUri belongs; reading it as no binding", path.c_str()); + return {}; + } + return line; +} + +std::string OpcuaPlugin::write_persisted_binding(const std::string & path, const std::string & application_uri) { + if (path.empty() || application_uri.empty()) { + return {}; + } + if (application_uri.find_first_of("\r\n") != std::string::npos) { + return "the ApplicationUri contains a line break and would read back truncated"; + } + std::error_code ec; + // Follow a symlink to what it points at, so the rename below replaces the + // target an operator pointed the path at. The link is read first, because a + // link whose target does not exist yet is the state before the first bind + // and weakly_canonical leaves such a link unresolved; the chain is bounded so + // a loop ends in the fallback. weakly_canonical then resolves the components + // that exist and leaves the rest. + constexpr int kMaxSymlinkHops = 32; + std::filesystem::path target(path); + for (int hops = 0; hops < kMaxSymlinkHops && std::filesystem::is_symlink(target, ec) && !ec; ++hops) { + const std::filesystem::path link_target = std::filesystem::read_symlink(target, ec); + if (ec) { + break; + } + target = link_target.is_absolute() ? link_target : target.parent_path() / link_target; + } + ec.clear(); + const std::filesystem::path resolved = std::filesystem::weakly_canonical(target, ec); + if (!ec && !resolved.empty()) { + target = resolved; + } + ec.clear(); + const std::filesystem::path parent = target.parent_path(); + if (!parent.empty()) { + std::filesystem::create_directories(parent, ec); + if (ec && !std::filesystem::is_directory(parent)) { + return "cannot create " + parent.string() + ": " + ec.message(); + } + ec.clear(); + } + // Same directory, so the rename below stays within one filesystem. + const std::filesystem::path tmp = target.string() + ".tmp"; + { + std::ofstream out(tmp, std::ios::trunc); + if (!out) { + return "cannot open " + tmp.string() + " for writing"; + } + out << application_uri << "\n"; + out.flush(); + if (!out) { + return "write to " + tmp.string() + " failed"; + } + } + // The fsync makes the temporary's bytes durable before the rename publishes + // its name, so a power loss after the rename finds the URI on disk. Without + // it the name can outlive the bytes, leaving an empty file that reads as + // "never bound" and lets the next process adopt whichever server answers. + const std::string sync_failure = fsync_path(tmp.string(), /*is_directory=*/false); + if (!sync_failure.empty()) { + std::error_code ignored; + std::filesystem::remove(tmp, ignored); + return sync_failure; + } + std::filesystem::rename(tmp, target, ec); + if (ec) { + std::error_code ignored; + std::filesystem::remove(tmp, ignored); + return "rename to " + target.string() + " failed: " + ec.message(); + } + // The directory entry the rename created, so the file is findable after a + // power loss. Best effort: a filesystem that refuses this has still written + // the file. + if (!parent.empty()) { + static_cast(fsync_path(parent.string(), /*is_directory=*/true)); + } + return {}; +} + bool OpcuaPlugin::note_refused_application_uri(const std::string & uri) { if (std::find(refused_application_uris_.begin(), refused_application_uris_.end(), uri) != refused_application_uris_.end()) { @@ -2365,10 +2531,25 @@ bool OpcuaPlugin::bind_or_drop_session() { if (bound_application_uri_.empty()) { // Nothing bound yet: this session names the server every later sweep looks - // for. A server that publishes no ApplicationUri leaves the binding empty - // and stays unbindable, which the discovery report already says. + // for, and the next process reads it back from the binding file. A server + // that publishes no ApplicationUri leaves the binding empty and stays + // unbindable, which the discovery report already says, and nothing is + // persisted for it. bound_application_uri_ = live_uri; refused_application_uris_.clear(); + if (!live_uri.empty() && !endpoint_configured_) { + const std::string failure = write_persisted_binding(discovery_config_.binding_file, live_uri); + if (failure.empty()) { + if (!discovery_config_.binding_file.empty()) { + log_info("OPC-UA: bound to uri='" + live_uri + "', kept in " + discovery_config_.binding_file); + } + } else { + log_warn("OPC-UA: bound to uri='" + live_uri + "' but could not keep it in " + discovery_config_.binding_file + + " (" + failure + + "). This session is unaffected; the next process starts unbound and adopts whichever server " + "answers."); + } + } return true; } @@ -2380,10 +2561,21 @@ bool OpcuaPlugin::bind_or_drop_session() { binding_mismatch_disconnects_.fetch_add(1); if (note_refused_application_uri(live_uri)) { - log_warn("OPC-UA: " + client_->endpoint_url() + " answers as uri='" + live_uri + - "', which is not the server this bridge is bound to (uri='" + bound_application_uri_ + - "'). Dropping the session and keeping the standing outage. Replacing a PLC is a recommissioning: " - "restart the plugin against the new one."); + // The gesture that rebinds: on the discovery path the file holds the + // binding, so the file is what an operator removes; with endpoint_url + // pinned, or with persistence off, nothing outlives the process and a + // restart against the new server is the whole gesture. + const std::string gesture = !endpoint_configured_ && !discovery_config_.binding_file.empty() + ? "remove " + discovery_config_.binding_file + " and restart" + : "restart the plugin against the new one"; + const std::string refusal = "OPC-UA: " + client_->endpoint_url() + " answers as uri='" + live_uri + + "', which is not the server this bridge is bound to (uri='" + bound_application_uri_ + + "'). Dropping the session and keeping the standing outage. Replacing a PLC is a " + "recommissioning: " + + gesture + "."; + log_warn(refusal); + std::lock_guard lock(binding_refusal_mutex_); + last_binding_refusal_ = refusal; } client_->disconnect(); return false; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp index b6a5801d7..ff8880500 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_poller.cpp @@ -1290,14 +1290,13 @@ void OpcuaPoller::poll_loop() { if (config_.prefer_subscriptions) { setup_subscriptions(); } - // Issue #386: re-subscribe to AlarmCondition events after reconnect - // and re-fire ConditionRefresh so we recover any conditions that - // changed state while we were offline. The OpcuaClient's - // generation counter has already advanced (incremented in - // disconnect()/maybe_mark_disconnected), so any stale event - // callbacks captured from the previous subscription are filtered - // out by the trampoline before re-subscription registers fresh - // contexts. + // Re-subscribe to AlarmCondition events on the new session and re-fire + // ConditionRefresh so conditions that changed state while the link was + // down are recovered. The client dropped the previous session's event + // contexts and subscription handles when it marked itself disconnected, + // and again inside connect(), so the monitored-item ids this session + // hands out, which start from 1 again, register fresh contexts under + // their own ids. event_subscription_id_ = 0; event_monitored_item_ids_.clear(); if (has_alarm_sources()) { diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 02723117e..154902f94 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -42,10 +42,12 @@ #include #include +#include #include #include #include #include +#include #include #include #include @@ -611,11 +613,9 @@ TEST_F(OpcuaIdentityE2ETest, DiNameplateReadFollowsBrowseContinuationPoints) { // A gateway that restarts after a comms outage never raised PLC_COMMS_LOST in // THIS process, yet the fault manager keys faults by fault_code alone and // persists them, so the fault raised before the restart is still standing. -// The reconnect arm used to clear only when its own in-memory -// ``comms_lost_raised_`` flag was set, which no restart can satisfy, so the -// fault stayed CONFIRMED for good. The clear now goes out on every successful -// connect. Driven against the live fixture because the arm can only be reached -// by a connect that actually succeeds. +// The clear goes out on every successful connect, whatever this process's own +// ``comms_lost_raised_`` flag says. Driven against the live fixture because +// the arm can only be reached by a connect that actually succeeds. TEST_F(OpcuaIdentityE2ETest, SuccessfulConnectClearsCommsLostNeverRaisedHere) { OpcuaClient client; OpcuaClientConfig config; @@ -797,6 +797,37 @@ class RealNodePluginContext : public FakePluginContext { rclcpp::Node * node_; }; +// A directory of this process's own, so two worktrees running the suite at once +// do not delete each other's fixtures. +class ScopedTempDir { + public: + ScopedTempDir() { + std::string pattern = (std::filesystem::temp_directory_path() / "medkit_opcua_e2e_XXXXXX").string(); + std::vector buffer(pattern.begin(), pattern.end()); + buffer.push_back('\0'); + const char * made = mkdtemp(buffer.data()); + if (made == nullptr) { + throw std::runtime_error("mkdtemp(" + pattern + ") failed: " + std::strerror(errno)); + } + dir_ = std::filesystem::path(made); + } + ~ScopedTempDir() { + std::error_code ec; + std::filesystem::remove_all(dir_, ec); + } + ScopedTempDir(const ScopedTempDir &) = delete; + ScopedTempDir & operator=(const ScopedTempDir &) = delete; + ScopedTempDir(ScopedTempDir &&) = delete; + ScopedTempDir & operator=(ScopedTempDir &&) = delete; + + std::string file(const std::string & name) const { + return (dir_ / name).string(); + } + + private: + std::filesystem::path dir_; +}; + // Poll until an OPC-UA session can be opened at ``endpoint``. A fixture prints // READY before its listen socket is accepting, so nothing may rely on one // before this returns. @@ -1825,6 +1856,573 @@ TEST_F(OpcuaIdentityE2ETest, TheBridgeStaysBoundToItsOwnServerAcrossAddressAndSw EXPECT_TRUE(store.sources_of(kCommsLostFaultCode).empty()) << "the outage was cleared but the row still stands"; } +// The binding outlives the process that made it. The second instance here +// starts from the file the first one wrote, refuses the foreign fixture +// answering at the bound address while the standing outage is kept, and takes +// the original fixture when it returns. Its refusal names the file, which is +// what an operator removes to bind to a different server. +TEST_F(OpcuaIdentityE2ETest, ARestartedPluginKeepsTheBindingAndRefusesAForeignServer) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_persisted_binding"); + auto fault_manager = std::make_shared("opcua_identity_persisted_binding_faultmgr"); + + server_.stop(); // this test drives its own fixtures + constexpr int kOpcuaPort = 4840; + const std::string bound_address = "127.0.0.1"; + const std::string foreign_uri = "urn:test:a-different-plc"; + ScopedTempDir binding_dir; + const std::string binding_file = binding_dir.file("binding"); + + std::mutex sweep_mutex; + std::string served_uri; + const auto scan = [](const std::string & ip, uint16_t port, int) { + return port == kOpcuaPort && ip == "127.0.0.1"; + }; + const auto identify = [&sweep_mutex, &served_uri](const std::string & url, int) { + IdentifyResult result; + result.ok = true; + result.advertised_url = url; + { + std::lock_guard lock(sweep_mutex); + result.application_uri = served_uri; + } + result.application_name = "Test PLC"; + result.application_type = 0; // Server + result.anonymous_none_available = true; + return result; + }; + const auto sweep_serves = [&sweep_mutex, &served_uri](const std::string & uri) { + std::lock_guard lock(sweep_mutex); + served_uri = uri; + }; + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) + << "this test needs TCP 4840 on loopback, the only port discovery identifies OPC-UA on"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://" + bound_address + ":4840")); + const std::string bound_uri = live_application_uri("opc.tcp://" + bound_address + ":4840"); + ASSERT_FALSE(bound_uri.empty()); + ASSERT_NE(bound_uri, foreign_uri); + sweep_serves(bound_uri); + + FaultStoreStub store(fault_manager); + store.seed(kCommsLostFaultCode, {"test_runtime"}); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + const std::string yaml_path = write_minimal_node_map(); + const auto plugin_config = [&]() { + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 200; + config["discovery"] = nlohmann::json{{"enabled", true}, + {"subnets", nlohmann::json::array({"127.0.0.1/32"})}, + {"ports", nlohmann::json::array({kOpcuaPort})}, + {"interval_s", 2}, + {"binding_file", binding_file}}; + return config; + }; + const auto wait_for = [](const std::function & done, std::chrono::seconds budget) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (!done() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + return done(); + }; + const auto cleared_count = [&store]() { + const auto codes = store.cleared_codes(); + return std::count(codes.begin(), codes.end(), std::string(kCommsLostFaultCode)); + }; + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + + // ---- the process that binds --------------------------------------------- + { + OpcuaPlugin first; + first.set_discovery_io_for_test(scan, identify); + first.configure(plugin_config()); + first.set_context(ctx); + ASSERT_TRUE(wait_for( + [&]() { + return cleared_count() > 0; + }, + std::chrono::seconds(30))) + << "the first instance never adopted the fixture, so nothing was bound"; + first.shutdown(); + } + ASSERT_TRUE(std::filesystem::exists(binding_file)) << "the binding was not kept, so a restart is unconstrained"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(binding_file), bound_uri); + const auto clears_after_first = cleared_count(); + + // ---- the PLC is swapped while nothing is running ------------------------- + // The first instance's clear emptied the row; the outage that follows its + // shutdown is what stands now, and clearing it is what adopting a server + // would look like. + store.seed(kCommsLostFaultCode, {"test_runtime"}); + bound_server.stop(); + AlarmServer squatter; + ASSERT_TRUE(squatter.start(fixture_binary(), kOpcuaPort, {"--app-uri", foreign_uri})); + ASSERT_TRUE(wait_for_connectable("opc.tcp://" + bound_address + ":4840")); + ASSERT_EQ(live_application_uri("opc.tcp://" + bound_address + ":4840"), foreign_uri); + // The sweep reports the bound identity at the bound address, so nothing + // before the connect can tell the two apart. + sweep_serves(bound_uri); + + // ---- the process that starts from the file ------------------------------- + OpcuaPlugin second; + second.set_discovery_io_for_test(scan, identify); + second.configure(plugin_config()); + second.set_context(ctx); + + ASSERT_TRUE(wait_for( + [&]() { + return second.binding_mismatch_count_for_test() > 0; + }, + std::chrono::seconds(30))) + << "the restarted instance polled a different PLC as if it were the one it had been bound to"; + const std::string refusal = second.last_binding_refusal_for_test(); + EXPECT_NE(refusal.find("remove " + binding_file + " and restart"), std::string::npos) + << "the refusal does not name the file an operator removes to rebind: " << refusal; + std::this_thread::sleep_for(std::chrono::seconds(2)); + EXPECT_EQ(cleared_count(), clears_after_first) << "the swapped-in server cleared the outage"; + EXPECT_FALSE(store.sources_of(kCommsLostFaultCode).empty()); + + // ---- the bound PLC comes back -------------------------------------------- + squatter.stop(); + AlarmServer restored; + ASSERT_TRUE(restored.start(fixture_binary(), kOpcuaPort)); + ASSERT_TRUE(wait_for_connectable("opc.tcp://" + bound_address + ":4840")); + const bool recovered = wait_for( + [&]() { + return cleared_count() > clears_after_first; + }, + std::chrono::seconds(60)); + + spin.stop(); + second.shutdown(); + restored.stop(); + std::remove(yaml_path.c_str()); + + EXPECT_TRUE(recovered) << "the bound server was not taken back, so the outage never ended"; +} + +// open62541's run_iterate drives its own connect whenever the session is below +// ACTIVATED, so a channel that dropped is re-opened against whatever answers the +// address now. These three shapes are the ones where nothing else notices: no +// scalar read to fail, or one slow enough that the re-connect wins first. What +// makes them observable is the client reading its session state after each +// iterate and handing the reconnect back to the arm. +TEST_F(OpcuaIdentityE2ETest, ASwapUnderAConfigLessSessionIsCaught) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_swap_configless"); + + constexpr int kOpcuaPort = 4840; + const std::string foreign_uri = "urn:test:swapped-plc"; + server_.stop(); + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) << "this test needs TCP 4840 on loopback"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + OpcuaPlugin plugin; + nlohmann::json config; + // Config-less: no node map, so nothing is polled by value and only the event + // pump iterates. auto_alarms is what the shipped DiagBox shape runs. + config["endpoint_url"] = "opc.tcp://127.0.0.1:4840"; + config["poll_interval_ms"] = 100; + config["auto_alarms"] = nlohmann::json{{"enabled", true}, {"auto_clear", true}}; + config["discovery"] = nlohmann::json{{"binding_file", ""}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + ASSERT_TRUE(plugin.connected_for_test()) << "the plugin never connected, so this proves nothing"; + + // The PLC is swapped for a different one at the same address. + bound_server.stop(); + AlarmServer swapped; + ASSERT_TRUE(swapped.start(fixture_binary(), kOpcuaPort, {"--app-uri", foreign_uri})); + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (plugin.binding_mismatch_count_for_test() == 0 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const auto mismatches = plugin.binding_mismatch_count_for_test(); + const std::string refusal = plugin.last_binding_refusal_for_test(); + plugin.shutdown(); + swapped.stop(); + + EXPECT_GT(mismatches, 0u) + << "the session was re-opened underneath the client against a different PLC and nothing checked it"; + // A pinned endpoint_url holds no file, so the gesture the refusal names is a + // restart against the new server. + EXPECT_NE(refusal.find("restart the plugin against the new one"), std::string::npos) << refusal; + EXPECT_EQ(refusal.find("remove "), std::string::npos) << "a pinned endpoint_url names no file to remove: " << refusal; +} + +TEST_F(OpcuaIdentityE2ETest, AnOutageUnderAConfigLessSessionIsObserved) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_outage_configless"); + auto fault_manager = std::make_shared("opcua_identity_outage_configless_faultmgr"); + + constexpr int kOpcuaPort = 4840; + server_.stop(); + + FaultStoreStub store(fault_manager); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) << "this test needs TCP 4840 on loopback"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = "opc.tcp://127.0.0.1:4840"; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 500; + config["auto_alarms"] = nlohmann::json{{"enabled", true}, {"auto_clear", true}}; + config["discovery"] = nlohmann::json{{"binding_file", ""}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + ASSERT_TRUE(plugin.connected_for_test()) << "the plugin never connected, so this proves nothing"; + + // The PLC goes away with nothing polling a value: the outage has to be seen + // through the session state alone. + bound_server.stop(); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + const auto outage_reported = [&store]() { + const auto reported = store.reported(); + return std::find(reported.begin(), reported.end(), std::string(kCommsLostFaultCode)) != reported.end(); + }; + while (!outage_reported() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const bool reported = outage_reported(); + const bool still_connected = plugin.connected_for_test(); + + spin.stop(); + plugin.shutdown(); + + EXPECT_FALSE(still_connected) << "the client reports a live session against a PLC that is gone"; + EXPECT_TRUE(reported) << "the outage was never observed, so PLC_COMMS_LOST has no input in this shape"; +} + +// The shipped shape: config-less with native alarms at the default 1000 ms poll +// cadence. Nothing reads a value, so the session state read after each iterate +// is the only thing that can see the swap, and the event pump iterates ten +// times per poll, so a session re-opened underneath the client would win long +// before anything else noticed. After the bound server returns, the alarm +// subscription the arm re-creates on it carries a device alarm to the store. +TEST_F(OpcuaIdentityE2ETest, ASwapUnderTheShippedPollCadenceIsCaught) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_swap_shipped"); + auto fault_manager = std::make_shared("opcua_identity_swap_shipped_faultmgr"); + + constexpr int kOpcuaPort = 4840; + const std::string foreign_uri = "urn:test:swapped-plc"; + server_.stop(); + + FaultStoreStub store(fault_manager); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) << "this test needs TCP 4840 on loopback"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = "opc.tcp://127.0.0.1:4840"; + config["poll_interval_ms"] = 1000; + config["auto_alarms"] = nlohmann::json{{"enabled", true}, {"auto_clear", true}}; + config["discovery"] = nlohmann::json{{"binding_file", ""}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + ASSERT_TRUE(plugin.connected_for_test()) << "the plugin never connected, so this proves nothing"; + + bound_server.stop(); + AlarmServer swapped; + ASSERT_TRUE(swapped.start(fixture_binary(), kOpcuaPort, {"--app-uri", foreign_uri})); + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(45); + while (plugin.binding_mismatch_count_for_test() == 0 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const auto mismatches = plugin.binding_mismatch_count_for_test(); + + // The bound server returns: the arm takes it and re-creates the alarm + // subscription on it. + swapped.stop(); + AlarmServer restored; + ASSERT_TRUE(restored.start(fixture_binary(), kOpcuaPort)); + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + const auto recover_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(45); + while (!plugin.connected_for_test() && std::chrono::steady_clock::now() < recover_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + const bool recovered = plugin.connected_for_test(); + + // The ConditionRefresh burst that follows the re-subscribe settles before the + // baseline is taken, so the report counted below is the one the fire caused. + // The fire is retried because an event fired before the subscribe is not + // delivered: the retry is the subscription handshake. + bool delivered = false; + if (recovered) { + std::this_thread::sleep_for(std::chrono::seconds(2)); + const size_t before = store.reported().size(); + const auto fire_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (store.reported().size() <= before && std::chrono::steady_clock::now() < fire_deadline) { + ASSERT_TRUE(restored.send("fire Overpressure 750")); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + } + delivered = store.reported().size() > before; + } + + spin.stop(); + plugin.shutdown(); + restored.stop(); + + EXPECT_GT(mismatches, 0u) << "a swapped PLC was polled as the bound one under the shipped cadence"; + EXPECT_TRUE(recovered) << "the bound server came back and the arm never took it"; + EXPECT_TRUE(delivered) << "a device alarm fired on the returned server never reached the fault store"; +} + +// A PLC reboot is an outage the same server ends, so the identity check has +// nothing to refuse and the arm reconnects to the server it is bound to. The +// alarm subscription the arm re-creates there carries device alarms: the +// monitored-item id the rebooted server hands out is the one the dead session +// held, and a context still registered under it would keep the new one out of +// the map while open62541 holds a pointer to it, silencing every alarm until a +// restart. +TEST_F(OpcuaIdentityE2ETest, ADeviceAlarmIsDeliveredAfterTheBoundServerReboots) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_reboot_alarm"); + auto fault_manager = std::make_shared("opcua_identity_reboot_alarm_faultmgr"); + + constexpr int kOpcuaPort = 4840; + server_.stop(); + + FaultStoreStub store(fault_manager); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + AlarmServer bound_server; + ASSERT_TRUE(bound_server.start(fixture_binary(), kOpcuaPort)) << "this test needs TCP 4840 on loopback"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + + OpcuaPlugin plugin; + nlohmann::json config; + // Config-less with native alarms: nothing reads a value, so the outage is seen + // through the session state and the reconnect is the arm's. + config["endpoint_url"] = "opc.tcp://127.0.0.1:4840"; + config["poll_interval_ms"] = 100; + config["comms_lost_debounce_ms"] = 500; + config["auto_alarms"] = nlohmann::json{{"enabled", true}, {"auto_clear", true}}; + config["discovery"] = nlohmann::json{{"binding_file", ""}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + plugin.set_context(ctx); + ASSERT_TRUE(plugin.connected_for_test()) << "the plugin never connected, so this proves nothing"; + + const auto reported_count = [&store]() { + return store.reported().size(); + }; + // The fire is retried because an event fired before the subscribe is not + // delivered: the retry is the subscription handshake. + const auto fire_until_reported = [&](AlarmServer & fixture, size_t above) { + const auto fire_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (reported_count() <= above && std::chrono::steady_clock::now() < fire_deadline) { + if (!fixture.send("fire Overpressure 750")) { + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + } + return reported_count() > above; + }; + // The first session delivers, so a silent path after the reboot is the + // reconnect's doing. + ASSERT_TRUE(fire_until_reported(bound_server, 0)) << "no alarm reached the fault store on the first session"; + + // The PLC reboots: the same server goes away and comes back. + bound_server.stop(); + const auto down_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (plugin.connected_for_test() && std::chrono::steady_clock::now() < down_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + ASSERT_FALSE(plugin.connected_for_test()) << "the outage was never observed"; + + AlarmServer rebooted; + ASSERT_TRUE(rebooted.start(fixture_binary(), kOpcuaPort)); + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + const auto up_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(45); + while (!plugin.connected_for_test() && std::chrono::steady_clock::now() < up_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + ASSERT_TRUE(plugin.connected_for_test()) << "the arm never took the rebooted server"; + + // The ConditionRefresh burst that follows the re-subscribe settles before the + // baseline is taken, so the report counted is the one the fire caused. + std::this_thread::sleep_for(std::chrono::seconds(2)); + const bool delivered = fire_until_reported(rebooted, reported_count()); + + spin.stop(); + plugin.shutdown(); + rebooted.stop(); + + EXPECT_TRUE(delivered) + << "a device alarm fired after the reboot never reached the fault store: the re-created subscription is dead"; +} + +// A pinned endpoint_url is the operator saying which server this is, so the +// binding file is neither read nor written on that path: a file an earlier +// config-less run left behind must not refuse the server the operator named. +TEST_F(OpcuaIdentityE2ETest, AConfiguredEndpointIgnoresTheBindingFile) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_pinned_endpoint"); + + ScopedTempDir binding_dir; + const std::string binding_file = binding_dir.file("binding"); + ASSERT_EQ(OpcuaPlugin::write_persisted_binding(binding_file, "urn:test:a-different-plc"), ""); + const auto written_at = std::filesystem::last_write_time(binding_file); + + const std::string yaml_path = write_minimal_node_map(); + OpcuaPlugin plugin; + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["discovery"] = nlohmann::json{{"enabled", true}, {"binding_file", binding_file}}; + plugin.configure(config); + + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + std::this_thread::sleep_for(std::chrono::seconds(1)); + + const auto mismatches = plugin.binding_mismatch_count_for_test(); + const bool connected = plugin.connected_for_test(); + plugin.shutdown(); + std::remove(yaml_path.c_str()); + + EXPECT_EQ(mismatches, 0u) + << "a file from an earlier config-less run refused the server the operator pinned endpoint_url at"; + EXPECT_TRUE(connected) << "the session the operator asked for was dropped"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(binding_file), "urn:test:a-different-plc") + << "the pinned-endpoint path wrote the binding file"; + EXPECT_EQ(std::filesystem::last_write_time(binding_file), written_at) + << "the pinned-endpoint path rewrote the binding file"; +} + +// The startup sweep looks for the bound server, at whatever address it answers +// on. Selection is what carries the binding; without it the sweep takes the +// lowest address and the process spends its life being refused there. +TEST_F(OpcuaIdentityE2ETest, TheStartupSweepSelectsTheBoundServerNotTheLowestAddress) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_startup_binding"); + + server_.stop(); + constexpr int kOpcuaPort = 4840; + const std::string foreign_uri = "urn:test:a-different-plc"; + auto binding_dir = ScopedTempDir(); + const std::string binding_file = binding_dir.file("binding"); + + AlarmServer fixture; + ASSERT_TRUE(fixture.start(fixture_binary(), kOpcuaPort)) + << "this test needs TCP 4840 on loopback, the only port discovery identifies OPC-UA on"; + ASSERT_TRUE(wait_for_connectable("opc.tcp://127.0.0.1:4840")); + const std::string bound_uri = live_application_uri("opc.tcp://127.0.0.1:4840"); + ASSERT_FALSE(bound_uri.empty()); + ASSERT_NE(bound_uri, foreign_uri); + ASSERT_EQ(OpcuaPlugin::write_persisted_binding(binding_file, bound_uri), ""); + + // Both addresses are open; the lower one serves a foreign identity and the + // higher one the bound identity. + const auto scan = [](const std::string & ip, uint16_t port, int) { + return port == kOpcuaPort && (ip == "127.0.0.1" || ip == "127.0.0.2"); + }; + const auto identify = [&foreign_uri, &bound_uri](const std::string & url, int) { + IdentifyResult result; + result.ok = true; + result.advertised_url = url; + result.application_uri = url.find("127.0.0.2") != std::string::npos ? bound_uri : foreign_uri; + result.application_name = "Test PLC"; + result.application_type = 0; // Server + result.anonymous_none_available = true; + return result; + }; + + const std::string yaml_path = write_minimal_node_map(); + const auto run_with_cadence = [&](int interval_s) { + OpcuaPlugin plugin; + plugin.set_discovery_io_for_test(scan, identify); + nlohmann::json config; + config["node_map_path"] = yaml_path; + config["poll_interval_ms"] = 100; + config["discovery"] = nlohmann::json{{"enabled", true}, + {"subnets", nlohmann::json::array({"127.0.0.0/30"})}, + {"ports", nlohmann::json::array({kOpcuaPort})}, + {"interval_s", interval_s}, + {"binding_file", binding_file}}; + plugin.configure(config); + RealNodePluginContext ctx(node.get()); + ctx.entities["tank"] = {SovdEntityType::APP, "tank", "/test_plc", "/test_plc/test_runtime/tank"}; + plugin.set_context(ctx); + const std::string endpoint = plugin.endpoint_url_for_test(); + plugin.shutdown(); + return endpoint; + }; + + // The address the sweep settled on is the whole assertion: one fixture + // answers on both loopback addresses with the bound identity, so the session + // passes the identity check whichever address it was opened at. + EXPECT_EQ(run_with_cadence(2), "opc.tcp://127.0.0.2:4840") + << "the startup sweep took the lowest address, so the bound server is reached only by a later rescan"; + + // interval_s: 0 keeps discovery on with the startup scan one-shot, so a + // startup sweep that ignores the binding pins the process to the foreign + // address for life. + EXPECT_EQ(run_with_cadence(0), "opc.tcp://127.0.0.2:4840") + << "with no rescan the startup sweep is the only one, and it selected a server this bridge is not bound to"; + + fixture.stop(); + std::remove(yaml_path.c_str()); +} + // The order the whole rename fix rests on: PollerConfig::on_connected runs // before the link-state edge and before anything is subscribed, so whatever it // renames is what the event path is handed. apply_condition_state pins a fault's diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp index c6f2fb4e9..f910825bb 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_plugin.cpp @@ -24,8 +24,12 @@ #include #include +#include #include #include +#include +#include +#include #include #include #include @@ -907,6 +911,194 @@ TEST(DiscoverEndpoint, DisabledDiscoveryScansNothing) { EXPECT_FALSE(scanned) << "a disabled discovery must not touch the network"; } +// --------------------------------------------------------------------------- +// The binding file: what a restarted process reads its constraint from +// --------------------------------------------------------------------------- + +namespace { + +// A directory of this process's own, so two worktrees running the suite at once +// do not delete each other's fixtures. +class ScopedBindingDir { + public: + ScopedBindingDir() { + std::string pattern = (std::filesystem::temp_directory_path() / "medkit_opcua_binding_XXXXXX").string(); + std::vector buffer(pattern.begin(), pattern.end()); + buffer.push_back('\0'); + const char * made = mkdtemp(buffer.data()); + if (made == nullptr) { + throw std::runtime_error("mkdtemp(" + pattern + ") failed: " + std::strerror(errno)); + } + dir_ = std::filesystem::path(made); + } + ~ScopedBindingDir() { + std::error_code ec; + std::filesystem::remove_all(dir_, ec); + } + ScopedBindingDir(const ScopedBindingDir &) = delete; + ScopedBindingDir & operator=(const ScopedBindingDir &) = delete; + ScopedBindingDir(ScopedBindingDir &&) = delete; + ScopedBindingDir & operator=(ScopedBindingDir &&) = delete; + + std::string file(const std::string & name) const { + return (dir_ / name).string(); + } + const std::filesystem::path & dir() const { + return dir_; + } + + private: + std::filesystem::path dir_; +}; + +void write_file(const std::string & path, const std::string & contents) { + std::ofstream out(path, std::ios::binary | std::ios::trunc); + ASSERT_TRUE(out.is_open()) << "cannot open " << path; + out << contents; + ASSERT_TRUE(out.good()) << "write to " << path << " failed"; +} + +} // namespace + +TEST(PersistedBinding, AUriWrittenIsTheUriReadBack) { + ScopedBindingDir tmp; + const std::string path = tmp.file("nested/binding"); + const std::string uri = "urn:siemens:s7-1500:line-a"; + + // The parent directory does not exist yet: the write makes it, which is what + // the state directory of a container that mounts one needs. + EXPECT_EQ(OpcuaPlugin::write_persisted_binding(path, uri), ""); + EXPECT_TRUE(std::filesystem::exists(path)); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(path), uri); + + // The temporary the atomic write goes through does not survive it, so a + // later reader cannot find a half-written binding lying beside the real one. + EXPECT_FALSE(std::filesystem::exists(path + ".tmp")); + + // A second binding replaces the first, and the file still holds one line. + const std::string other = "urn:beckhoff:cx5140:line-b"; + EXPECT_EQ(OpcuaPlugin::write_persisted_binding(path, other), ""); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(path), other); + std::ifstream in(path); + std::string first; + std::getline(in, first); + EXPECT_EQ(first, other); + EXPECT_FALSE(std::getline(in, first)) << "the binding file carries more than the URI"; +} + +TEST(PersistedBinding, NothingToKeepAndNowhereToKeepIt) { + ScopedBindingDir tmp; + + // Persistence disabled: nothing is written and nothing is read. + EXPECT_EQ(OpcuaPlugin::write_persisted_binding("", "urn:siemens:s7-1500:line-a"), ""); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(""), ""); + + // A server with no ApplicationUri has no binding to keep. + const std::string path = tmp.file("binding"); + EXPECT_EQ(OpcuaPlugin::write_persisted_binding(path, ""), ""); + EXPECT_FALSE(std::filesystem::exists(path)); + + // Never written: a process starting here has never been bound. + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(tmp.file("absent")), ""); + + const std::string uri = "urn:siemens:s7-1500:line-a"; + + // Written by hand with nothing on the first line. + const std::string blank = tmp.file("blank"); + write_file(blank, " \n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(blank), "") + << "a blank line was read as a binding, which constrains the plugin to a server named by nothing"; + + // Written by hand with the URI and a trailing newline and spaces. + const std::string padded = tmp.file("padded"); + write_file(padded, " " + uri + " \n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(padded), uri); + + // An editor that writes CRLF. + const std::string crlf = tmp.file("crlf"); + write_file(crlf, uri + "\r\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(crlf), uri) << "the carriage return stayed in the URI"; + + // An editor that writes a UTF-8 BOM. It is invisible in a log, so a binding + // carrying one shows the refusal as two identical URIs. + const std::string bom = tmp.file("bom"); + write_file(bom, std::string("\xEF\xBB\xBF") + uri + "\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(bom), uri) << "the byte-order mark stayed in the URI"; + + // Anything after the first line is not part of the binding. + const std::string extra = tmp.file("extra"); + write_file(extra, uri + "\nurn:test:ignored\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(extra), uri); + + // A file holding something that is not an ApplicationUri. + const std::string nul = tmp.file("nul"); + write_file(nul, std::string("urn:test:\0plc", 13) + "\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(nul), "") << "a NUL byte was read as part of a binding"; + const std::string control = tmp.file("control"); + write_file(control, "urn:test:\x01plc\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(control), "") << "a control character was read as part of a binding"; + const std::string del = tmp.file("del"); + write_file(del, "urn:test:\x7Fplc\n"); + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(del), "") << "DEL was read as part of a binding"; + + // A URI that would read back truncated is refused at the write, whichever + // line break it carries. + const std::string split = tmp.file("split"); + EXPECT_NE(OpcuaPlugin::write_persisted_binding(split, "urn:test:a\nurn:test:b"), ""); + EXPECT_FALSE(std::filesystem::exists(split)); + const std::string split_cr = tmp.file("split_cr"); + EXPECT_NE(OpcuaPlugin::write_persisted_binding(split_cr, "urn:test:a\rurn:test:b"), ""); + EXPECT_FALSE(std::filesystem::exists(split_cr)); +} + +TEST(PersistedBinding, AParentThatCannotHoldTheFileIsReportedAndNotThrown) { + ScopedBindingDir tmp; + // A regular FILE where the parent directory has to be: create_directories + // fails with ENOTDIR for every user, root included, so this exercises the + // failure branch wherever the suite runs. + const std::string parent = tmp.file("afile"); + write_file(parent, "not a directory\n"); + const std::string path = parent + "/binding"; + + std::string failure; + ASSERT_NO_THROW(failure = OpcuaPlugin::write_persisted_binding(path, "urn:siemens:s7-1500:line-a")); + EXPECT_NE(failure, "") << "a parent that cannot hold the file was reported as a successful write"; + EXPECT_NE(failure.find(parent), std::string::npos) + << "the failure does not name the path an operator has to fix: " << failure; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(path), ""); +} + +TEST(PersistedBinding, ASymlinkedPathKeepsItsTarget) { + ScopedBindingDir tmp; + const std::string uri = "urn:siemens:s7-1500:line-a"; + std::error_code ec; + + // A file symlink: an operator points the configured path at a file they keep + // elsewhere. Replacing the link with a plain file leaves that file stale. + const std::string real_file = tmp.file("real_binding"); + write_file(real_file, "urn:test:stale\n"); + const std::string link = tmp.file("link_binding"); + std::filesystem::create_symlink(real_file, link, ec); + ASSERT_FALSE(ec) << ec.message(); + + EXPECT_EQ(OpcuaPlugin::write_persisted_binding(link, uri), ""); + EXPECT_TRUE(std::filesystem::is_symlink(link)) << "the symlink was replaced by a plain file"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(real_file), uri) << "the symlink's target was not updated"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(link), uri); + + // A file symlink whose target does not exist yet, which is the state before + // the first bind: the target is created and the link is kept. + const std::string absent_target = tmp.file("absent_binding"); + const std::string dangling = tmp.file("dangling_binding"); + std::filesystem::create_symlink(absent_target, dangling, ec); + ASSERT_FALSE(ec) << ec.message(); + + EXPECT_EQ(OpcuaPlugin::write_persisted_binding(dangling, uri), ""); + EXPECT_TRUE(std::filesystem::is_symlink(dangling)) << "the dangling symlink was replaced by a plain file"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(absent_target), uri) << "the symlink's target was not created"; + EXPECT_EQ(OpcuaPlugin::read_persisted_binding(dangling), uri); +} + TEST(EffectiveRescanInterval, DefaultsWhenDiscoveryIsOnWithNoCadenceAndIsOffOtherwise) { OpcuaDiscoveryConfig cfg = rescan_cfg(); // Config-less: discovery on, no interval stated -> the built-in cadence, not @@ -1646,8 +1838,8 @@ TEST(IdentityReads, ARefusedPauseEndsTheBurst) { TEST(CommsLostAnswerApplicable, AProbeGivenUpOnHasItsAnswerDropped) { // The generation the timeout branch moves on, read from the consume side: an - // answer parked against a probe number the poll thread is no longer waiting - // for is not the answer to the probe that replaced it. rclcpp takes a pending + // answer parked against a probe number the poll thread has moved past is not + // the answer to the probe that replaced it. rclcpp takes a pending // entry out before it invokes the callback and outside its own mutex, so a // callback that won that race is still on its way in when the timeout fires. EXPECT_TRUE(OpcuaPlugin::comms_lost_generation_current(/*answered_generation=*/4, /*current_generation=*/4)); From e00c495526beff279b97c493836319d9bab0e21e Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 09:56:58 +0200 Subject: [PATCH 19/25] test(gateway): make the openapi contract test sweep every entity and check each built item test_a_scoped_item_says_what_its_templated_sibling_says compares every entity in both listings, counts built items per entity type, collection and scope, requires a built item for every listed resource, and checks that the item at each key names that resource. test_every_advertised_collection_is_served follows every app's hrefs and requires a cache-built href per entity type. The class waits for calibration's operation through REQUIRED_OPERATIONS and polls temp_sensor's data with the discovery budget before comparing, so the counters do not depend on how fast the runner propagates a service. --- .../features/test_openapi_contract.test.py | 234 +++++++++++++----- 1 file changed, 170 insertions(+), 64 deletions(-) diff --git a/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py b/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py index aa9755188..9f201f93e 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py @@ -33,7 +33,7 @@ import launch_testing.actions import requests -from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, DISCOVERY_TIMEOUT from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase from ros2_medkit_test_utils.launch_helpers import ( create_test_launch, @@ -148,6 +148,16 @@ class TestOpenApiContract(GatewayTestCase): MIN_EXPECTED_APPS = 2 REQUIRED_APPS = {'calibration', 'temp_sensor'} + # The app entities appearing is not enough for this file. A node is listed + # in the ROS graph before its service endpoints have propagated, so a + # discovery sweep can build the App with an empty service list, and the + # cache-derived operation items in `/docs` are built from exactly that + # list. Until one service is in the cache every operations sub-document + # publishes only projections, and the comparison over `operations` in + # `test_a_scoped_item_says_what_its_templated_sibling_says` has nothing to + # compare. Waiting for the capability the assertion reads is what makes + # the file independent of how fast the runner propagates a service. + REQUIRED_OPERATIONS = {'/apps/calibration': 'calibrate'} _spec = None @@ -1241,6 +1251,7 @@ def test_every_advertised_collection_is_served(self): """ offenders = [] covered = {} + covered_built = {} for entity_type in ('areas', 'components', 'apps', 'functions'): items = self.get_json(f'/{entity_type}').get('items', []) if not items: @@ -1249,24 +1260,45 @@ def test_every_advertised_collection_is_served(self): # the per-type lists themselves are pinned by the # `EntityCapabilities` unit tests. continue - entity_id = items[0]['id'] - detail = self.get_json(f'/{entity_type}/{entity_id}') - subtree = self.get_json(f'/{entity_type}/{entity_id}/docs') - advertised = {c['href'] for c in detail.get('capabilities', [])} - advertised |= {f'/api/v1{p}' - for p, item in subtree['paths'].items() - if 'get' in item} followed = 0 - for href in sorted(advertised): - if '{' in href: - # A templated path names no concrete resource to fetch. - continue - resp = requests.get( - f'{self.BASE_URL}{href[len("/api/v1"):]}', timeout=10) - followed += 1 - if resp.status_code == 404: - offenders.append(f'{entity_type}: {href}') + built = 0 + for entity_id in [item['id'] for item in items]: + detail = self.get_json(f'/{entity_type}/{entity_id}') + subtree = self.get_json(f'/{entity_type}/{entity_id}/docs') + advertised = {c['href'] for c in detail.get('capabilities', [])} + advertised |= {f'/api/v1{p}' + for p, item in subtree['paths'].items() + if 'get' in item} + # `x-sovd-name` marks a path the cache built out of a concrete + # resource id, which is the only kind whose href can 404 from an + # id that does not resolve. They live in the collection + # sub-documents, so the capability list and the entity subtree + # alone never reach one. + built_hrefs = set() + for collection in ('data', 'operations'): + doc = requests.get( + f'{self.BASE_URL}/{entity_type}/{entity_id}/' + f'{collection}/docs', timeout=10) + if doc.status_code != 200: + continue + built_hrefs |= {f'/api/v1{p}' + for p, item in doc.json().get( + 'paths', {}).items() + if 'get' in item and 'x-sovd-name' in item} + advertised |= built_hrefs + for href in sorted(advertised): + if '{' in href: + # A templated path names no concrete resource to fetch. + continue + resp = requests.get( + f'{self.BASE_URL}{href[len("/api/v1"):]}', timeout=10) + followed += 1 + if href in built_hrefs: + built += 1 + if resp.status_code == 404: + offenders.append(f'{entity_type}: {href}') covered[entity_type] = followed + covered_built[entity_type] = built self.assertEqual(offenders, [], f'advertised but 404: {offenders}') # Guard against a vacuous pass: an entity type that advertised nothing, # or a listing that came back empty, must not read as green. @@ -1274,6 +1306,16 @@ def test_every_advertised_collection_is_served(self): self.assertGreater( covered.get(entity_type, 0), 8, f'{entity_type}: only {covered.get(entity_type, 0)} hrefs followed') + # The capability list and the docs subtree are the same shape for every + # entity, so following them proves nothing about an id that has to + # resolve. A cache-built href is the only kind that can 404 from an id + # the cache does not hold, and every entity type that had a listing + # aggregates at least one data point or operation in this fixture. + for entity_type, built in sorted(covered_built.items()): + self.assertGreater( + built, 0, + f'{entity_type}: no cache-built href was followed, so no ' + f'advertised resource id was resolved') def test_the_root_list_and_the_document_agree(self): """`GET /api/v1` and `GET /api/v1/docs` describe the same gateway. @@ -1347,37 +1389,58 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): inherited from the projected route they describe, because they *are* that route. - Both scopes are compared, not merely visited. ``project()`` substitutes - the ids it was given into the path **keys**, so at collection scope the - sibling is still templated (``/data/{data_id}``) while at - specific-resource scope it has already become concrete - (``/data/temperature``). An earlier version used the collection - document only as a source of siblings, which meant every - collection-scope item could be published completely un-inherited and - this test stayed green. + Both scopes are compared. ``project()`` substitutes the ids it was + given into the path **keys**, so at collection scope the sibling is + still templated (``/data/{data_id}``) while at specific-resource scope + it has already become concrete (``/data/temperature``). The collection + document is both the source of siblings and a publisher of built + items, so its own items are compared here too. Read from the document alone: the sibling states the contract and the built item must match it, so no second source is needed and none is trusted. + + Every entity in both listings is swept, and the count is kept per + entity type, per collection and per scope: an app whose node exposes no + service publishes no built operation item at all, and the two scopes + are separate branches of the same producer, so each of those counts is + what makes its own branch falsifiable. + + The fixture is pinned on the capabilities this reads. + `REQUIRED_OPERATIONS` holds the class wait until calibration's + `calibrate` is in the cache; the poll below does the same for + temp_sensor's data, which the base class has no equivalent for. A node + is listed in the ROS graph before its services and topics have + propagated, so without both the counters are whatever the graph had + reached at the instant the request went out. """ + self.poll_endpoint_until( + '/apps/temp_sensor/data', + lambda d: d if d.get('items') else None, + timeout=DISCOVERY_TIMEOUT, + ) compared = 0 - built_items = {'data': 0, 'operations': 0} + built_items = {(entity_type, collection, scope): 0 + for entity_type in ('apps', 'components') + for collection in ('data', 'operations') + for scope in ('collection', 'resource')} offenders = [] for entity_type in ('apps', 'components'): items = self.get_json(f'/{entity_type}').get('items', []) if not items: continue - entity_id = items[0]['id'] - for collection in ('data', 'operations'): + entity_ids = [item['id'] for item in items] + pairs = [(entity_id, collection) + for entity_id in entity_ids + for collection in ('data', 'operations')] + for entity_id, collection in pairs: base = f'/{entity_type}/{entity_id}/{collection}' collection_doc = self.get_json(f'{base}/docs') collection_paths = collection_doc.get('paths', {}) # The item parameter is read from the served document, the way - # `CapabilityGenerator` reads it from the registry. Spelling - # `data_id`/`operation_id` here was a second copy of the fact - # that fix removed from production: renaming the registry - # parameter left production working and this guard green having - # compared nothing. + # `CapabilityGenerator` reads it from the registry, so the + # parameter name lives in one place and a rename there is + # followed here. template = self._item_template(collection_paths, base) self.assertIsNotNone( template, @@ -1389,7 +1452,17 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): for key, path_item in collection_paths.items(): if 'x-sovd-name' not in path_item: continue - built_items[collection] += 1 + # The key a built item is published under names the item + # the request resolves to. + self.assertTrue( + key.startswith(f'{base}/'), + f'{key}: a built item published outside {base}/') + self.assertEqual( + self._item_half(key[len(base) + 1:]), + self._item_half(path_item['x-sovd-name']), + f'{key}: the built item names ' + f'{path_item["x-sovd-name"]}') + built_items[(entity_type, collection, 'collection')] += 1 for method, operation in path_item.items(): if method not in HTTP_METHODS: continue @@ -1403,21 +1476,47 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): listing = requests.get(f'{self.BASE_URL}{base}', timeout=10) if listing.status_code != 200: continue - for entry in listing.json().get('items', []): - resource_id = entry['id'] + listed_ids = [entry['id'] + for entry in listing.json().get('items', [])] + # An operation id that names more than one operation answers + # 400 and gets no built item. The listing shows such an id + # once per operation it names, which is how it is told apart + # here without a second copy of the producer's rule. + ambiguous = {resource_id for resource_id in listed_ids + if listed_ids.count(resource_id) > 1} + for resource_id in listed_ids: scoped = requests.get( f'{self.BASE_URL}{base}/{resource_id}/docs', timeout=10) - if scoped.status_code != 200: - continue + # Every id the collection listed has a sub-document. + self.assertEqual( + scoped.status_code, 200, + f'{base}/{resource_id}/docs answered ' + f'{scoped.status_code}; the collection lists that id') key = f'{base}/{resource_id.lstrip("/")}' path_item = scoped.json().get('paths', {}).get(key, {}) + if resource_id in ambiguous: + self.assertNotIn( + 'x-sovd-name', path_item, + f'{key}: a built item for an id that names more ' + f'than one operation describes a request the ' + f'gateway refuses') + continue # `x-sovd-name` is written only by `PathBuilder`, so it is # what tells a *built* item from the projection that sits - # at the same key at this scope. Counting the key alone - # made this test unfalsifiable. - if 'x-sovd-name' not in path_item: - continue - built_items[collection] += 1 + # at the same key at this scope. The listing and the + # producer read the same cache, so every id listed here + # carries one. + self.assertIn( + 'x-sovd-name', path_item, + f'{key}: the collection lists this id and its scoped ' + f'document carries no built item for it') + self.assertEqual( + self._item_half(path_item['x-sovd-name']), + self._item_half(resource_id), + f'{key}: the built item names ' + f'{path_item["x-sovd-name"]}, the request named ' + f'{resource_id}') + built_items[(entity_type, collection, 'resource')] += 1 for method, operation in path_item.items(): if method not in HTTP_METHODS: continue @@ -1429,27 +1528,36 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): self.assertEqual( offenders, [], f'built items contradicting their route: {offenders[:12]}') - # A cache-derived item must *exist*, and a comparison must actually have - # happened. `compared` counts comparisons performed, not operations - # visited: a sibling that is missing is a miss, not a pass, so a guard - # that found no sibling can no longer satisfy this by counting the + # A cache-derived item must exist, and a comparison must have happened. + # `compared` counts comparisons performed: a missing sibling is a miss, + # so a guard that found no sibling cannot satisfy this by counting the # operations it skipped. - # Per collection, not in total. Both are built by the same code down - # different branches, so one can vanish entirely while the other keeps - # the count above zero - which is what happened when a built verb the - # sibling lacked made every *data* item get discarded and this stayed - # green on operations alone. - for collection, count in sorted(built_items.items()): + # Per type, collection and scope. Data and operations are built by the + # same code down different branches, and the two scopes are separate + # branches again, so any one of them can stop publishing while a merged + # count stays above zero. + for (entity_type, collection, scope), count in sorted(built_items.items()): self.assertGreater( count, 0, - f'no {collection} sub-document published a cache-derived item ' - f'(none carried x-sovd-name); every comparison over ' - f'{collection} was vacuous') + f'no {entity_type} {collection} sub-document published a ' + f'cache-derived item at {scope} scope (none carried ' + f'x-sovd-name); every comparison over {entity_type} ' + f'{collection} at {scope} scope was vacuous') self.assertGreater( compared, 0, 'no built operation was compared against a sibling; the guard ran ' 'over nothing') + @staticmethod + def _item_half(item_id): + """Return the item half of a possibly member-qualified id, no leading slash. + + A short name more than one member of an entity carries is addressed + ``:``; ``x-sovd-name`` carries the item half alone. + """ + _, sep, item = item_id.partition(':') + return (item if sep else item_id).lstrip('/') + @staticmethod def _item_template(collection_paths, base): """Return the templated item route under `base`, or None. @@ -1470,9 +1578,9 @@ def _framework_mismatch(self, collection_paths, template, key, method, operation): """Compare one built operation against its templated sibling. - Returns ``(problems, compared)``. A missing sibling is reported rather - than skipped: it used to return no problems, so a lookup that found - nothing counted as a pass everywhere it was called. + Returns ``(problems, compared)``. A missing sibling is reported as a + problem, so a lookup that finds nothing is a miss everywhere this is + called. """ sibling = collection_paths.get(template, {}).get(method) if sibling is None: @@ -1483,11 +1591,9 @@ def _framework_mismatch(self, collection_paths, template, key, method, problems.append( f'{method.upper()} {key}: security ' f'{operation.get("security")} != {sibling.get("security")}') - # Every status, 2xx included. An earlier version carved 2xx out as - # "the payload, meant to differ", which is true of a request body and - # false of a response: the gateway envelopes every read - `DataValue`, - # `OperationDetail` - so a built 200 was a second, contradictory answer - # for one route rather than a more specific one. + # Every status, 2xx included: the gateway envelopes every read - + # `DataValue`, `OperationDetail` - so a built 200 is a second answer + # for one route, and it has to be the same answer. built_statuses = set(operation.get('responses', {})) sibling_statuses = set(sibling.get('responses', {})) if built_statuses != sibling_statuses: From d54e8c92beb2043f42cbde1896f6cc23781f97a0 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 09:56:59 +0200 Subject: [PATCH 20/25] docs(opcua): describe the binding file, its rules and the cases the identity check covers The binding section of the README covers: - the path and the format of the file - the rule that only discovery uses it - how to clear it - the start-up sweep and the rescan behaviour - who owns a reconnect, and the cases the end-to-end tests cover - the probing cadence at a refused address - the last eight refused URIs - one file per plugin instance - the mount that a re-created container needs - the configuration and environment rows --- .../ros2_medkit_opcua/README.md | 101 +++++++++++++++--- 1 file changed, 84 insertions(+), 17 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 030f080df..2f1b96fd5 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -661,6 +661,7 @@ ros2_medkit_gateway: | `comms_lost_severity` | `ERROR` | SOVD severity bucket for the `PLC_COMMS_LOST` fault | | `fault_service_timeout_ms` | `5000` | How long a fault-store read may stay outstanding before the `PLC_COMMS_LOST` decision is owed again (out of range [100, 600000] ms is refused with a warning and the previous value is kept) | | `discovery.enabled` | `false` | Opt-in read-only PLC network discovery (auto endpoint). See below | +| `discovery.binding_file` | `/var/lib/ros2_medkit/opcua/binding` | Where the bound server's `ApplicationUri` is kept, so the binding survives a restart. Read and written only while `endpoint_url` is unset. Empty disables persistence and makes every process's first adoption unconstrained. One file per plugin instance. Env: `OPCUA_DISCOVERY_BINDING_FILE` | ### OPC-UA client security (SecurityPolicy, certificates, user auth) @@ -717,10 +718,13 @@ plugins.opcua.discovery: # or set it to 0 to keep discovery on but never re-scan (start-up scan only). interval_s: 30 anonymous_none_only: true # only auto-connect None/Anonymous servers + # where the bound server's ApplicationUri is kept across restarts; "" disables + binding_file: /var/lib/ros2_medkit/opcua/binding ``` Environment overrides (Docker / appliance): `OPCUA_DISCOVERY_ENABLED`, -`OPCUA_DISCOVERY_SUBNETS` (comma-separated CIDRs), `OPCUA_DISCOVERY_INTERVAL_S`. +`OPCUA_DISCOVERY_SUBNETS` (comma-separated CIDRs), `OPCUA_DISCOVERY_INTERVAL_S`, +`OPCUA_DISCOVERY_BINDING_FILE`. Leaving `interval_s` (and `OPCUA_DISCOVERY_INTERVAL_S`) unset means "no cadence stated" and takes the 30 s default. An explicit `0` is honoured as written and turns the recurring sweep off. A negative value is refused with a warning and @@ -789,31 +793,93 @@ How it works: The **binding** is the OPC-UA `ApplicationUri` of the server the plugin actually held a session with, read off that session (the `ServerArray`, whose first entry -is that URI). It is not persisted, so it exists only for the life of the process. - -- A re-scan looks for **that server and no other**, at any address: the bound - `ApplicationUri` is an input to the selection, so a foreign server does not win - by sorting lower. A sweep that finds no hit carrying it selects nothing, the - endpoint stands and the `PLC_COMMS_LOST` fault stands with it. -- A **different server at the bound address** is caught when the session comes +is that URI). On the discovery path it is written to `discovery.binding_file` +(default `/var/lib/ros2_medkit/opcua/binding`) the moment it is established and +read back in `configure()` before the first sweep, so it **is kept across +restarts**: a process that comes back looks for the server it was bound to and +no other. With `endpoint_url` configured the file is neither read nor written: +the operator has named the server, and re-pointing `endpoint_url` at a +replacement and restarting is the whole gesture. + +- The **start-up sweep and every re-scan look for that server and no other**, + at any address: the bound `ApplicationUri` is an input to the selection, so a + foreign server does not win by sorting lower. A sweep that finds no hit + carrying it selects nothing, the endpoint stands and the `PLC_COMMS_LOST` + fault stands with it. +- A **different server at the bound address** is caught when a session comes up: the live `ApplicationUri` is read, the mismatch is logged at WARN, the session is dropped, no link-state clear is sent, and the reconnect loop keeps - trying. Both reports are once per distinct URI per outage, and the list is - cleared as soon as the bound server is reached again. + trying. The check runs on every session the reconnect arm opens. A session + open62541 re-opens on its own after a channel drop is caught wherever + `run_iterate` runs, which is every shape with native alarms on (the event + pump and the poll cycle both iterate): after each iterate the client reads + its session state, and a session that fell below ACTIVATED after having been + active is disconnected explicitly, so the next session is the arm's, checked + against the binding, with the native alarm subscription re-created on it. + That is what makes a swap visible where nothing reads a value: the + config-less deployment with native alarms and no node map, at any poll + cadence. open62541 re-opens a session only from `run_iterate`, so with native + alarms off nothing is re-opened underneath: a node map's scalar read fails + with a connection-closed code, the client is marked down, and the arm's next + session is checked the same way. The re-created subscription carries device + alarms; that is proven for a swap followed by the bound server's return and + for a plain reboot of the bound server, both config-less with native alarms. +- While a foreign server answers at the bound address, the reconnect arm probes + that address at the backoff cadence (capped at `discovery.interval_s` while + re-scanning is on), refuses the foreign server each time, and adopts the + bound server the moment it answers there again. The plugin remembers the last + eight refused URIs and logs a URI once while it is remembered; the list is + cleared as soon as the bound server is reached. - A PLC that **moved** - new address, same `ApplicationUri` - is re-adopted, which is what the re-scan is for. -- **Replacing a PLC is a recommissioning**: restart the plugin against the new - one. Adopting a different PLC silently would re-point every SOVD entity at - hardware nobody asked for and would clear the outage as if the link had healed. +- **Replacing a PLC is a recommissioning**. On the discovery path: remove the + binding file (or set `discovery.binding_file` to an empty string) and + restart; the refusal WARN names the file. With `endpoint_url` configured: + point it at the new PLC and restart. Adopting a different PLC silently would + re-point every SOVD entity at hardware nobody asked for and would clear the + outage as if the link had healed. + +The binding is kept across restarts; removing the file and restarting clears +it. The binding that was loaded, and the file it came from, are logged at INFO +on start. + +A persisted URI naming a server that is nowhere on the subnet keeps the plugin +disconnected: it re-scans at the cadence, reports a foreign server once while +it is among the last eight refused, and holds the outage. There is no expiry - +an identity that moved silently +is the thing this exists to catch. Three cases have no binding, and in each the next adoption is unconstrained: -1. Nothing has been connected yet - the gateway that started before its PLC, and - every process after a restart, because the binding is never persisted. -2. An operator-configured `endpoint_url`, which runs no discovery at all. +1. Nothing has ever been connected and no binding file was found. +2. An operator-configured `endpoint_url`, which runs no discovery at all and + neither reads nor writes the binding file. 3. A server that publishes **no `ApplicationUri`**: there is nothing to bind to, so after a drop a re-scan accepts whichever server answers. This is logged at - WARN when such a server is adopted. + WARN when such a server is adopted, and nothing is written to the file. Such + a server is adoptable only while no binding is held; with a binding it is + refused like any other mismatch. + +The file holds one line, the URI. A UTF-8 byte-order mark and surrounding +whitespace are stripped, a CRLF line ending is accepted, and anything after the +first line is ignored. A first line holding a NUL or another control character +is refused with a WARN naming the file and reads as no binding. A URI holding a +line break is refused at the write with a WARN, since it would read back +truncated. The file is written through a temporary in the same directory that +is flushed to disk before it is renamed over the path; a path that is a symlink +is followed, so its target is what gets replaced. A binding that cannot be +written - a directory that cannot be created or is not writable - is reported +at WARN naming the path. The session is unaffected; what is lost is the next +process's constraint. + +The file belongs to **one plugin instance**. A second gateway on the same host, +or a second instance of this plugin in one gateway, bound to a different PLC +must set its own `discovery.binding_file`; two instances on the default path +overwrite each other's binding. The shipped image creates +`/var/lib/ros2_medkit` for the fault manager database and declares no volume +for it, so a binding that has to outlive a re-created container needs that +directory mounted (`docker restart` keeps the writable layer; a new +`docker run` starts without it). Re-scanning stops as soon as a session is up, and never starts at all when an `endpoint_url` is configured. @@ -965,6 +1031,7 @@ Write operations use the `set_` prefix convention: | `OPCUA_REQUIRE_CONFIRM_FOR_CLEAR` | `0`/`false`/`no`/`off` to clear native alarms on Acknowledge alone (Confirm-less servers) | | `OPCUA_COMMS_LOST_ENABLED` | `0`/`false`/`no`/`off` to disable the `PLC_COMMS_LOST` fault | | `OPCUA_COMMS_LOST_DEBOUNCE_MS` | Continuous down time (ms) before `PLC_COMMS_LOST` is raised. A value above 3600000 ms is clamped, with a warning, as on the JSON path; a non-numeric or negative value is refused with a warning and the existing value is kept | +| `OPCUA_DISCOVERY_BINDING_FILE` | Path the bound server's `ApplicationUri` is kept in while `endpoint_url` is unset (empty disables persistence) | ## Hardware Deployment From 070e158921909295509d9241da6e1bec3ba32175 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 11:43:45 +0200 Subject: [PATCH 21/25] fix(opcua): create, dispatch and destroy the fault-service clients on the plugin's own threads The ReportFault, ClearFault and GetFault clients live in a callback group that is not added to an executor together with the node. A single-threaded executor owned by the plugin serves that group from a thread the plugin starts in set_context. shutdown stops the poller, stops and joins that thread, and then drops all three clients on the calling thread, so a client destructor never runs on a gateway executor thread while another plugin or the gateway creates an entity on the same node. The GetFault answer is parked from the plugin's thread under the probe state's mutex, as before. The client thread also ends once the node's context has shut down. set_context is documented as called once per plugin instance. A new end-to-end case builds two plugin instances in sequence on one node while the gateway's executor spins, and checks that the first shutdown joins the client thread and that the second instance's clients reach the fault services. --- .../ros2_medkit_opcua/README.md | 1 + .../ros2_medkit_opcua/design/index.rst | 6 + .../ros2_medkit_opcua/opcua_plugin.hpp | 35 ++++++ .../ros2_medkit_opcua/src/opcua_plugin.cpp | 119 +++++++++++++++--- .../test/test_opcua_identity.cpp | 61 +++++++++ 5 files changed, 206 insertions(+), 16 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 2f1b96fd5..2e3e3589a 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -1221,6 +1221,7 @@ keeps a history. - **Type-aware writes** - Plugin reads the OPC-UA node's data type before writing to avoid type mismatches (e.g., writing float32 to a REAL node, not float64). - **Node map driven** - All entity mapping is in YAML config, not code. Same plugin binary works with any PLC by changing the config file. - **Env var overrides** - `OPCUA_ENDPOINT_URL` and `OPCUA_NODE_MAP_PATH` override YAML config for Docker deployment flexibility. +- **Fault-service clients on the plugin's own thread** - the `ReportFault`, `ClearFault` and `GetFault` clients live in a callback group the gateway's executor never collects and are pumped by a thread the plugin owns, so they are created, dispatched and destroyed on the plugin's threads; a plugin shut down while another creates an entity on the same node never runs a client destructor on a gateway executor thread. - **Read-only is a build property, not a setting** - the shipped binary contains no OPC-UA write path, and CI proves it by inspecting the object rather than by reading a configuration value. A setting can be flipped on a running box; an absent symbol cannot. ## License diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst b/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst index b7da7bac0..f84d6616d 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/design/index.rst @@ -75,6 +75,12 @@ The plugin supports two data paths, selected by the ``prefer_subscriptions`` con (``poll_interval_ms``, default 1000 ms). Simple, predictable, no event loop needed, sufficient for diagnostic use cases where latency below one second does not add value. +Alongside the poll thread and the event pump, the plugin runs a client executor thread of +its own: the ``ReportFault``, ``ClearFault`` and ``GetFault`` clients live in a callback +group the gateway's executor never collects, so their creation, response dispatch and +destruction all happen on the plugin's threads, and a plugin shut down while another creates +an entity on the same node never runs a client destructor on a gateway executor thread. + **Subscription** - registers OPC-UA monitored items and receives change notifications at ``subscription_interval_ms``. Lower CPU cost on large node sets, but requires a running OPC-UA client event loop. Kept as an opt-in path because many PLC servers have limits on the number diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp index d294bdfb0..24a5d007b 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/include/ros2_medkit_opcua/opcua_plugin.hpp @@ -43,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -103,6 +104,10 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return "opcua"; } void configure(const nlohmann::json & config) override; + /// Precondition: called at most once per instance. It creates the + /// fault-service clients and starts the thread that pumps them, and a second + /// call would assign over that running std::thread, which is std::terminate. + /// The gateway makes that one call, through PluginManager::set_context. void set_context(ros2_medkit_gateway::PluginContext & context) override; std::vector get_routes() override; void shutdown() override; @@ -182,6 +187,15 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, return last_binding_refusal_; } + /// Whether the three fault-service clients see their services. Read by tests + /// that build two plugin instances in sequence on one node. + bool fault_services_ready_for_test() const; + /// Whether the thread that pumps the fault-service clients is running. False + /// before set_context() and after shutdown() has joined it. + bool fault_executor_thread_running_for_test() const { + return fault_executor_thread_.joinable(); + } + /// The address-space walk configuration after configure() has merged the /// node map and the ROS parameters. Read by tests that need to see which /// source supplied a setting, which no REST response exposes. @@ -868,6 +882,27 @@ class OpcuaPlugin : public ros2_medkit_gateway::GatewayPlugin, // ROS 2 service clients for fault reporting struct FaultClients; std::unique_ptr fault_clients_; + // The callback group the three clients live in, made with + // automatically_add_to_executor_with_node=false so the gateway's executor + // never collects them. rclcpp::AnyExecutable holds a strong reference to the + // client it dispatches, so a client in the node's default group can have its + // last reference dropped, and ~Client run against the node's entity registry, + // on a gateway executor thread while another plugin or the gateway creates an + // entity on the same node. With the group on a private executor, creation + // (set_context), response dispatch (fault_executor_thread_) and destruction + // (shutdown(), after that thread is joined) all happen on this plugin's + // threads. Response callbacks therefore run on fault_executor_thread_. + rclcpp::CallbackGroup::SharedPtr fault_client_group_; + std::unique_ptr fault_executor_; + std::thread fault_executor_thread_; + std::atomic fault_executor_stop_{false}; + // Starts fault_executor_thread_. Precondition: the thread is not running, + // which set_context()'s once-per-instance contract guarantees; context_ and + // fault_executor_ are set before the call. + void start_fault_executor(); + // Raises the stop flag, cancels the current wait and joins the thread. + // Idempotent. + void stop_fault_executor(); // Ordered buffer of pending fault report/clear dispatches. ReportFault / // ClearFault are fire-and-forget, so a report sent before the fault_manager diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index e4b5d9a6b..9990d4000 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -17,6 +17,14 @@ #include "ros2_medkit_opcua/device_identity.hpp" #include +// create_client() takes its callback group with a rclcpp::QoS from rclcpp 28 +// (Jazzy) on; earlier releases offer the rmw_qos_profile_t form. Humble ships +// no , so the header's absence identifies it. +#if defined(__has_include) +#if __has_include() +#include +#endif +#endif #include #include #include @@ -209,6 +217,26 @@ std::vector parse_json_ns_list(const nlohmann::json & arr, const char } } // namespace +namespace { + +/// create_client() with an explicit callback group, in the form the linked +/// rclcpp wants. +template +typename rclcpp::Client::SharedPtr create_client_in_group(rclcpp::Node * node, const std::string & name, + const rclcpp::CallbackGroup::SharedPtr & group) { +#if defined(RCLCPP_VERSION_MAJOR) && RCLCPP_VERSION_MAJOR >= 28 + return node->create_client(name, rclcpp::ServicesQoS(), group); +#else + return node->create_client(name, rmw_qos_profile_services_default, group); +#endif +} + +/// How long one pass of the client executor waits for work before it re-reads +/// the stop flag. Bounds shutdown latency only: a response wakes the wait. +constexpr std::chrono::milliseconds kFaultExecutorWait{100}; + +} // namespace + struct OpcuaPlugin::FaultClients { rclcpp::Client::SharedPtr report; rclcpp::Client::SharedPtr clear; @@ -662,10 +690,21 @@ void OpcuaPlugin::set_context(PluginContext & context) { auto * node = ctx_->node(); if (node) { - fault_clients_->report = node->create_client("/fault_manager/report_fault"); - fault_clients_->clear = node->create_client("/fault_manager/clear_fault"); - fault_clients_->get_fault = node->create_client("/fault_manager/get_fault"); + // The clients live in a group the gateway's executor never collects and are + // pumped by a thread this plugin owns, so every reference to them stays on + // this plugin's threads (see fault_client_group_). + fault_client_group_ = node->create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive, + /*automatically_add_to_executor_with_node=*/false); + fault_executor_ = std::make_unique(); + fault_executor_->add_callback_group(fault_client_group_, node->get_node_base_interface()); + fault_clients_->report = create_client_in_group( + node, "/fault_manager/report_fault", fault_client_group_); + fault_clients_->clear = create_client_in_group( + node, "/fault_manager/clear_fault", fault_client_group_); + fault_clients_->get_fault = + create_client_in_group(node, "/fault_manager/get_fault", fault_client_group_); context_ = node->get_node_base_interface()->get_context(); + start_fault_executor(); } run_startup_discovery(); @@ -862,13 +901,13 @@ void OpcuaPlugin::shutdown() { if (poller_) { poller_->stop(); } - // The poll thread is joined, so nothing else drives the decision. Dropping the - // client releases the plugin's reference to it and takes the outstanding - // request out of its pending map; it does NOT recall a callback the executor - // has already taken off the wait set. Bumping the generation makes such a - // callback a no-op even while the plugin is alive, and once the plugin is - // destroyed the weak_ptr it holds stops locking at all - which is the part a - // mutex here could never provide. + // The poll thread is joined, so nothing drives the decision; the client + // executor thread is joined next, so no response callback of ours runs after + // this returns and the clients can be dropped on this thread. The generation + // bump keeps an answer that was parked before the join from being read by a + // later probe, and the weak_ptr the callback holds is what makes a callback + // outlive-safe should the state block go before the client does. + stop_fault_executor(); comms_lost_decision_owed_.store(false); comms_lost_probe_in_flight_ = false; { @@ -877,14 +916,62 @@ void OpcuaPlugin::shutdown() { comms_lost_probe_state_->answered = false; comms_lost_probe_state_->sources.clear(); } - if (fault_clients_ && fault_clients_->get_fault) { - fault_clients_->get_fault->remove_pending_request(comms_lost_probe_request_id_); + // Dropping a client takes its outstanding requests out of its pending map. + // All three go here, on the thread that called shutdown(), so ~Client never + // runs on an executor thread against a node another plugin may be creating an + // entity on. + if (fault_clients_) { + if (fault_clients_->get_fault) { + fault_clients_->get_fault->remove_pending_request(comms_lost_probe_request_id_); + } fault_clients_->get_fault.reset(); + fault_clients_->clear.reset(); + fault_clients_->report.reset(); } + fault_executor_.reset(); + fault_client_group_.reset(); client_->disconnect(); log_info("OPC-UA plugin shutdown complete"); } +void OpcuaPlugin::start_fault_executor() { + fault_executor_stop_.store(false); + fault_executor_thread_ = std::thread([this]() { + // Runs until shutdown() raises the stop flag or the node's context is shut + // down, whichever comes first; a context shut down ahead of the plugin + // leaves nothing for the clients to dispatch. One wait per pass, bounded, + // so a stop requested before a wait began is still seen within + // kFaultExecutorWait. An exception out of the executor ends the thread too, + // and shutdown() joins it as usual. + while (!fault_executor_stop_.load() && rclcpp::ok(context_)) { + try { + fault_executor_->spin_once(kFaultExecutorWait); + } catch (const std::exception & e) { + RCLCPP_WARN(opcua_plugin_logger(), "fault-service client executor stopped: %s", e.what()); + break; + } catch (...) { + break; + } + } + }); +} + +void OpcuaPlugin::stop_fault_executor() { + fault_executor_stop_.store(true); + if (fault_executor_) { + fault_executor_->cancel(); + } + if (fault_executor_thread_.joinable()) { + fault_executor_thread_.join(); + } +} + +bool OpcuaPlugin::fault_services_ready_for_test() const { + return fault_clients_ && fault_clients_->report && fault_clients_->clear && fault_clients_->get_fault && + fault_clients_->report->service_is_ready() && fault_clients_->clear->service_is_ready() && + fault_clients_->get_fault->service_is_ready(); +} + // -- IntrospectionProvider -- IntrospectionResult OpcuaPlugin::introspect(const IntrospectionInput & /*input*/) { @@ -1624,10 +1711,10 @@ void OpcuaPlugin::drive_comms_lost_decision() { try { auto future = fault_clients_->get_fault->async_send_request( request, [weak_state, generation](rclcpp::Client::SharedFuture answer) { - // Executor thread. It parks the answer and nothing else: the decision - // reads component ids, which only the poll thread may do. The state is - // reached through a weak_ptr, so a callback the executor had already - // taken when the plugin went away touches nothing. + // The plugin's client executor thread. It parks the answer and nothing + // else: the decision reads component ids, which only the poll thread + // may do. The state is reached through a weak_ptr, so a callback that + // outlives the plugin's state block touches nothing. const auto state = weak_state.lock(); if (!state) { return; diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 154902f94..6ad7e3728 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -2215,6 +2215,67 @@ TEST_F(OpcuaIdentityE2ETest, ASwapUnderTheShippedPollCadenceIsCaught) { EXPECT_TRUE(delivered) << "a device alarm fired on the returned server never reached the fault store"; } +// The restart shape on one node: a second plugin instance is built on the node +// the first one used while the gateway's executor spins. The fault-service +// clients live in a callback group that executor never collects and are pumped +// by a thread the plugin owns, so the first instance's clients are destroyed on +// the thread that calls shutdown(), after their executor thread is joined, and +// the second instance creates its own on that same thread. A data race between +// the two is ThreadSanitizer's to report, in the sanitizer build. Every build +// pins the observable half: the first shutdown returns with its executor thread +// joined, and the second instance's clients resolve the services. +TEST_F(OpcuaIdentityE2ETest, ASecondPluginInstanceOnTheSameNodeReachesTheFaultServices) { + ScopedRclcpp rclcpp_scope; + auto node = std::make_shared("opcua_identity_two_instances"); + auto fault_manager = std::make_shared("opcua_identity_two_instances_faultmgr"); + + FaultStoreStub store(fault_manager); + store.open_reports(); + store.open_clears(); + store.open_reads(); + + rclcpp::executors::MultiThreadedExecutor executor; + executor.add_node(node); + executor.add_node(fault_manager); + ScopedExecutorSpin spin(executor); + + RealNodePluginContext ctx(node.get()); + nlohmann::json config; + config["endpoint_url"] = endpoint_; + config["poll_interval_ms"] = 100; + config["discovery"] = nlohmann::json{{"binding_file", ""}}; + + const auto services_ready_within = [](const OpcuaPlugin & plugin, std::chrono::seconds budget) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (!plugin.fault_services_ready_for_test() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + return plugin.fault_services_ready_for_test(); + }; + + { + OpcuaPlugin first; + first.configure(config); + first.set_context(ctx); + ASSERT_TRUE(first.fault_executor_thread_running_for_test()) << "set_context started no client executor thread"; + ASSERT_TRUE(services_ready_within(first, std::chrono::seconds(10))) + << "the first instance's fault-service clients never saw the store"; + first.shutdown(); + EXPECT_FALSE(first.fault_executor_thread_running_for_test()) + << "shutdown returned with the client executor thread still running"; + } + + OpcuaPlugin second; + second.configure(config); + second.set_context(ctx); + const bool second_ready = services_ready_within(second, std::chrono::seconds(10)); + second.shutdown(); + spin.stop(); + + EXPECT_TRUE(second_ready) << "the second instance's fault-service clients never saw the store"; + EXPECT_FALSE(second.fault_executor_thread_running_for_test()); +} + // A PLC reboot is an outage the same server ends, so the identity check has // nothing to refuse and the arm reconnects to the server it is bound to. The // alarm subscription the arm re-creates there carries device alarms: the From aeb986d31d4971bee8c0097c49bdf88e3ca4f130 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 11:43:56 +0200 Subject: [PATCH 22/25] test(gateway): shut rclcpp down after every lifecycle handler suite EntityDetailStatusLinkTest and LifecycleHandlersWithProviderTest initialise rclcpp when it is not running, and their TearDownTestSuite now shuts it down the way LifecycleHandlersTest does. The test process exits with no rclcpp context left initialised, so static destruction does not run alongside live middleware threads. --- src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp b/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp index 9cbea1b35..08e611979 100644 --- a/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp @@ -329,6 +329,9 @@ class EntityDetailStatusLinkTest : public ::testing::Test { } static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } } void SetUp() override { @@ -471,6 +474,9 @@ class LifecycleHandlersWithProviderTest : public ::testing::Test { } static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } } void SetUp() override { From 09deeacacf23967594028a20e735be21ba2250db Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Wed, 16 Sep 2026 12:10:20 +0200 Subject: [PATCH 23/25] opcua: take the fault-service answer and config values by reference, and name the test stubs' parameters The GetFault answer and the fault_service_timeout_ms config value are read through const references, and the persisted-binding and rescan results are returned from non-const locals so they are moved out. The severity band order check is written as the direct comparison of the integer bands. The GetFault response callback keeps its future by value, because rclcpp offers the response-callback overload of async_send_request only to a callable taking exactly SharedFuture; the check is suppressed at that line with the reason next to it. In the identity tests, the stub definitions name every parameter, the child process fixture and the rclcpp scope guard are neither copyable nor movable, and the fault store stub's service callbacks take their arguments by const reference. --- .../ros2_medkit_opcua/src/opcua_plugin.cpp | 18 +++-- .../test/test_opcua_identity.cpp | 67 ++++++++++++------- 2 files changed, 52 insertions(+), 33 deletions(-) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp index 9990d4000..a3edbb743 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/src/opcua_plugin.cpp @@ -390,7 +390,7 @@ void OpcuaPlugin::configure(const nlohmann::json & config) { } } if (config.contains("fault_service_timeout_ms")) { - const auto raw = config["fault_service_timeout_ms"]; + const auto & raw = config["fault_service_timeout_ms"]; if (raw.is_number_integer()) { const int64_t ms = raw.get(); if (ms >= 100 && ms <= 600000) { @@ -1418,8 +1418,8 @@ void OpcuaPlugin::apply_auto_alarms_param(const nlohmann::json & value, AutoAlar warn("plugins.opcua.auto_alarms.severity_bands: unknown key '" + key + "' - ignored"); } } - if (!(cfg.severity_bands.critical_min >= cfg.severity_bands.error_min && - cfg.severity_bands.error_min >= cfg.severity_bands.warning_min)) { + if (cfg.severity_bands.critical_min < cfg.severity_bands.error_min || + cfg.severity_bands.error_min < cfg.severity_bands.warning_min) { warn( "plugins.opcua.auto_alarms.severity_bands must satisfy critical >= error >= warning - " "resetting to the default bands (801/501/201)"); @@ -1708,9 +1708,13 @@ void OpcuaPlugin::drive_comms_lost_decision() { std::weak_ptr weak_state = comms_lost_probe_state_; auto request = std::make_shared(); request->fault_code = kCommsLostFaultCode; + // rclcpp selects the response-callback overload of async_send_request only for + // a callable whose parameter list is exactly (SharedFuture); a const reference + // parameter matches no overload. The future is therefore taken by value. + using GetFaultFuture = rclcpp::Client::SharedFuture; try { auto future = fault_clients_->get_fault->async_send_request( - request, [weak_state, generation](rclcpp::Client::SharedFuture answer) { + request, [weak_state, generation](GetFaultFuture answer) { // NOLINT(performance-unnecessary-value-param) // The plugin's client executor thread. It parks the answer and nothing // else: the decision reads component ids, which only the poll thread // may do. The state is reached through a weak_ptr, so a callback that @@ -1720,7 +1724,7 @@ void OpcuaPlugin::drive_comms_lost_decision() { return; } try { - const auto response = answer.get(); + const auto & response = answer.get(); std::lock_guard lock(state->mutex); if (state->generation != generation) { return; // answer to a probe that has already been given up on @@ -2577,7 +2581,7 @@ std::string OpcuaPlugin::write_persisted_binding(const std::string & path, const // its name, so a power loss after the rename finds the URI on disk. Without // it the name can outlive the bytes, leaving an empty file that reads as // "never bound" and lets the next process adopt whichever server answers. - const std::string sync_failure = fsync_path(tmp.string(), /*is_directory=*/false); + std::string sync_failure = fsync_path(tmp.string(), /*is_directory=*/false); if (!sync_failure.empty()) { std::error_code ignored; std::filesystem::remove(tmp, ignored); @@ -2678,7 +2682,7 @@ std::optional OpcuaPlugin::rescan_endpoint_for_reconnect() { const int interval_s = effective_rescan_interval_s(discovery_config_, endpoint_configured_); std::string candidate_uri; - const auto chosen = rescan_guarded( + auto chosen = rescan_guarded( interval_s, []() { return std::chrono::steady_clock::now(); diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp index 6ad7e3728..b57e8fc65 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/test/test_opcua_identity.cpp @@ -80,10 +80,10 @@ namespace ros2_medkit_gateway { PluginRequest::PluginRequest(const void * impl) : impl_(impl) { } -std::string PluginRequest::path_param(size_t) const { +std::string PluginRequest::path_param(size_t /*index*/) const { return {}; } -std::string PluginRequest::header(const std::string &) const { +std::string PluginRequest::header(const std::string & /*name*/) const { return {}; } const std::string & PluginRequest::path() const { @@ -94,15 +94,16 @@ const std::string & PluginRequest::body() const { static const std::string empty; return empty; } -std::string PluginRequest::query_param(const std::string &) const { +std::string PluginRequest::query_param(const std::string & /*name*/) const { return {}; } PluginResponse::PluginResponse(void * impl) : impl_(impl) { } -void PluginResponse::send_json(const nlohmann::json &) { +void PluginResponse::send_json(const nlohmann::json & /*data*/) { } -void PluginResponse::send_error(int, const std::string &, const std::string &, const nlohmann::json &) { +void PluginResponse::send_error(int /*status*/, const std::string & /*error_code*/, const std::string & /*message*/, + const nlohmann::json & /*parameters*/) { } // -- FakePluginContext: node() is null (no ROS graph), just enough for @@ -119,35 +120,38 @@ class FakePluginContext : public RosPluginContext { auto it = entities.find(id); return it != entities.end() ? std::optional(it->second) : std::nullopt; } - std::vector get_child_apps(const std::string &) const override { + std::vector get_child_apps(const std::string & /*component_id*/) const override { return {}; } - nlohmann::json list_entity_faults(const std::string &) const override { + nlohmann::json list_entity_faults(const std::string & /*entity_id*/) const override { // Contract: a bare JSON array of fault objects (empty for this fake). return nlohmann::json::array(); } - std::optional validate_entity_for_route(const PluginRequest &, PluginResponse &, + std::optional validate_entity_for_route(const PluginRequest & /*req*/, PluginResponse & /*res*/, const std::string & entity_id) const override { return get_entity(entity_id); } - void register_capability(SovdEntityType, const std::string &) override { + void register_capability(SovdEntityType /*entity_type*/, const std::string & /*capability_name*/) override { } - void register_entity_capability(const std::string &, const std::string &) override { + void register_entity_capability(const std::string & /*entity_id*/, const std::string & /*capability_name*/) override { } - std::vector get_type_capabilities(SovdEntityType) const override { + std::vector get_type_capabilities(SovdEntityType /*entity_type*/) const override { return {}; } - std::vector get_entity_capabilities(const std::string &) const override { + std::vector get_entity_capabilities(const std::string & /*entity_id*/) const override { return {}; } - LockAccessResult check_lock(const std::string &, const std::string &, const std::string &) const override { + LockAccessResult check_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::string & /*collection*/) const override { return {true, "", "", ""}; } - tl::expected acquire_lock(const std::string &, const std::string &, - const std::vector &, int) override { + tl::expected acquire_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::vector & /*scopes*/, + int /*expiration_seconds*/) override { return tl::make_unexpected(LockError{"not supported", "", 409, std::nullopt}); } - tl::expected release_lock(const std::string &, const std::string &) override { + tl::expected release_lock(const std::string & /*entity_id*/, + const std::string & /*client_id*/) override { return tl::make_unexpected(LockError{"not supported", "", 409, std::nullopt}); } IntrospectionInput get_entity_snapshot() const override { @@ -157,9 +161,9 @@ class FakePluginContext : public RosPluginContext { return nlohmann::json::object(); } void register_sampler( - const std::string &, - const std::function(const std::string &, const std::string &)> &) - override { + const std::string & /*collection*/, + const std::function(const std::string &, const std::string &)> & + /*fn*/) override { } ResourceChangeNotifier * get_resource_change_notifier() override { return nullptr; @@ -197,9 +201,17 @@ int reserve_local_port() { // prints the "READY " handshake line on stdout. SIGTERM on teardown. class AlarmServer { public: + AlarmServer() = default; ~AlarmServer() { stop(); } + // Owns a child process and its pipe descriptors, so a member-wise copy or move + // would leave two objects stopping the same process and closing the same + // descriptors. + AlarmServer(const AlarmServer &) = delete; + AlarmServer & operator=(const AlarmServer &) = delete; + AlarmServer(AlarmServer &&) = delete; + AlarmServer & operator=(AlarmServer &&) = delete; bool start(const std::string & binary, int port, const std::vector & extra_args = {}) { int pipefd[2]; @@ -687,6 +699,8 @@ struct ScopedRclcpp { } ScopedRclcpp(const ScopedRclcpp &) = delete; ScopedRclcpp & operator=(const ScopedRclcpp &) = delete; + ScopedRclcpp(ScopedRclcpp &&) = delete; + ScopedRclcpp & operator=(ScopedRclcpp &&) = delete; }; // Spins an executor on its own thread and guarantees cancel -> join on every @@ -893,8 +907,9 @@ class FaultStoreStub { void open_reports() { report_srv_ = node_->create_service( - "/fault_manager/report_fault", [this](const std::shared_ptr req, - std::shared_ptr res) { + "/fault_manager/report_fault", + [this](const std::shared_ptr & req, + const std::shared_ptr & res) { { std::lock_guard lock(mutex_); reported_.push_back(req->fault_code); @@ -909,8 +924,8 @@ class FaultStoreStub { void open_clears() { clear_srv_ = node_->create_service( - "/fault_manager/clear_fault", [this](const std::shared_ptr req, - std::shared_ptr res) { + "/fault_manager/clear_fault", [this](const std::shared_ptr & req, + const std::shared_ptr & res) { { std::lock_guard lock(mutex_); cleared_.push_back(*req); @@ -923,8 +938,8 @@ class FaultStoreStub { void open_reads(bool answer_immediately = true) { answer_reads_.store(answer_immediately); read_srv_ = node_->create_service( - "/fault_manager/get_fault", [this](const std::shared_ptr header, - const std::shared_ptr req) { + "/fault_manager/get_fault", [this](const std::shared_ptr & header, + const std::shared_ptr & req) { if (answer_reads_.load()) { answer_read(*header, req->fault_code); return; @@ -1666,7 +1681,7 @@ TEST_F(OpcuaIdentityE2ETest, AFaultRaisedUnderTheStandInHealsAfterTheDeviceNames ASSERT_TRUE(wait_until_connectable()); const auto heal_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(45); - while (store.sources_of(kCommsLostFaultCode).size() != 0 && std::chrono::steady_clock::now() < heal_deadline) { + while (!store.sources_of(kCommsLostFaultCode).empty() && std::chrono::steady_clock::now() < heal_deadline) { std::this_thread::sleep_for(std::chrono::milliseconds(100)); } From 1a28ad8c68c2b03ba00ecd78087b8a89d476d96d Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Fri, 18 Sep 2026 13:55:35 +0200 Subject: [PATCH 24/25] fix(gateway): hide a graph leftover only for a node discovery saw running The DDS RMWs take a participant's nodes from the ros_discovery_info topic on a listener thread, and its enclave from participant discovery. When the listener thread takes a discovery sample only after its participant was removed, the sample re-creates the participant entry and nothing removes it again: the graph keeps listing the node, with an empty enclave and none of its endpoints, for as long as the gateway runs. An empty enclave alone does not identify such a leftover, because nodes behind a DDS router and nodes whose participant was created outside rcl read the same way. The node-list reader leaves an entry out only for a name it saw running and that has departed, when no other entry of that name has an enclave and the graph resolves no publisher or subscriber for the node. Names it never saw running are listed. The names running on the previous read are that read's own list and cost nothing. A name that ran on the previous read and does not run now has departed. The reader forgets a departed name when it runs again, or on the first read that finds no entry of it more than 10 s after the first read that found none; while a leftover of it is listed, that time does not start. A late sample that arrives before the read that forgets the name stays hidden, however late it comes. The 10 s cover the listener thread's delay in taking a sample it already holds. The reader remembers at most 1024 departed names; past that it forgets the names no entry has listed for longest first, then the names that departed longest ago. Endpoints are read with the publisher and subscriber queries without demangling, which on these RMWs also return service and client endpoints. Only rclcpp's error for a node the graph no longer lists counts as no endpoints; any other error, such as the one a shut-down context raises, reaches the caller. Discovery owns one reader and the startup peer count reads through it, so both count the same nodes. Discovery logs a warning when it starts leaving a node out. The check for the gateway's in-process helper nodes moves into the same header-only file, so the discovery plugins can use it. parameter_beacon reads the graph through a reader of its own, and skips the gateway's own node and its helper nodes, which carry no beacon. It keeps one parameter client per poll target across cycles: an rclcpp client for list_parameters and one for get_parameters, spun on the parameter client's own executor for the length of a call. Every call gives up after param_timeout_sec and removes the unanswered request from its client, so a node whose parameter services never answer neither stalls polling nor leaves requests pending. The waits for the list and get services share one deadline, and each wait gets the time left, at least zero: rclcpp waits forever when it is given a negative time. A timeout puts the node on backoff and any answer clears it, including a get answer without values: rclpy answers so when a typed parameter has no value, and the plugin then stores no hint. A node that is no longer a target, also when no node is, loses its client and its backoff. Only the poll thread spins the plugin's node. The poll thread destroys the clients, and so does shutdown() after it has joined that thread. The plugin's node joins rclcpp's graph listener when it is created: a first join after rclcpp shuts down fails half-way, and the node's destructor then terminates the process. A try/catch around reset() cannot stop that, because reset() and the destructors it runs are noexcept, so both beacons drop the try/catch they had there in shutdown(). parameter_beacon's destructor still catches what shutdown() throws, because locking clients_mutex_ can throw std::system_error. topic_beacon's shutdown() throws nothing, and its destructor calls it directly. Every duration parameter_beacon reads takes its minimum up to 2147483647 s. Fast DDS keeps the seconds of a wait in a signed 32-bit field, and a longer param_timeout_sec makes the poll thread spin. A larger value, +inf included, becomes the maximum; NaN, -inf and a smaller value become the minimum. Each replacement is logged with the value that replaced it. topic_beacon applies the same rules to beacon_ttl_sec and beacon_expiry_sec (up to 2147483647 s) and to max_messages_per_second (1 to 10000), so NaN no longer drops every beacon or keeps hints forever, and +inf no longer removes the rate limit. Both beacons read max_hints as a 64-bit integer before they narrow it. A 64-bit integer below 1 becomes 1, and one above 2147483647 becomes 2147483647, each with a warning, so a value such as 4294967297 no longer wraps to a capacity of 1. The parameter parser reads an integer that does not fit in 64 bits as a double. A double, such as 1e12 or NaN, is refused with a warning and the default of 10000 stays. --- docs/config/discovery-options.rst | 41 +++ docs/config/server.rst | 87 ++++- .../ros2_medkit_param_beacon/README.md | 31 +- .../ros2_medkit_param_beacon/design/index.rst | 25 +- .../param_beacon_plugin.hpp | 26 +- .../parameter_client_interface.hpp | 89 ++++- .../src/param_beacon_plugin.cpp | 191 ++++++----- .../ros2_medkit_topic_beacon/README.md | 11 + .../ros2_medkit_topic_beacon/design/index.rst | 15 +- .../topic_beacon_plugin.hpp | 20 +- .../src/topic_beacon_plugin.cpp | 93 +++-- src/ros2_medkit_gateway/README.md | 2 +- .../discovery/discovery_manager.hpp | 4 + .../ros2_medkit_gateway/gateway_node.hpp | 49 +-- .../providers/ros2_runtime_introspection.hpp | 12 +- .../ros2_common/graph_node_list.hpp | 320 ++++++++++++++++++ .../src/discovery/discovery_manager.cpp | 4 + src/ros2_medkit_gateway/src/gateway_node.cpp | 46 +-- .../providers/ros2_runtime_introspection.cpp | 21 +- .../ros2_medkit_graph_watchdog/README.md | 13 + 20 files changed, 849 insertions(+), 251 deletions(-) create mode 100644 src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp diff --git a/docs/config/discovery-options.rst b/docs/config/discovery-options.rst index 4df39077e..59374a167 100644 --- a/docs/config/discovery-options.rst +++ b/docs/config/discovery-options.rst @@ -514,6 +514,17 @@ Configuration # Default: 100 plugins.topic_beacon.max_messages_per_second: 100 +``beacon_ttl_sec`` takes 0.1 to 2147483647 s, ``beacon_expiry_sec`` 1.0 to 2147483647 s, and +``max_messages_per_second`` 1 to 10000. A value above its maximum, including ``.inf`` in a +parameter file, becomes the maximum. NaN, ``-.inf`` and a value below the minimum become the +minimum. The gateway logs a warning naming the key, the value and what replaced it. Messages +over the rate limit are dropped without a log. + +``max_hints`` takes an integer from 1 to 2147483647. A 64-bit integer outside that range +becomes the nearer bound, with the same warning. The parameter parser reads an integer that +does not fit in 64 bits as a double. A double, such as ``1.0e12`` or ``.nan``, is refused with +a warning and the default 10000 is used. + Beacon Lifecycle ^^^^^^^^^^^^^^^^ @@ -669,6 +680,36 @@ Configuration # Default: 10000 plugins.parameter_beacon.max_hints: 10000 +Every duration takes its minimum - 0.1 s, and 1.0 s for ``beacon_expiry_sec`` - up to +2147483647 s. Fast DDS keeps the seconds of a wait in a 32-bit signed integer, and a longer +``param_timeout_sec`` would make the poll thread spin. A value above the maximum, including +``.inf`` in a parameter file, becomes the maximum. NaN, ``-.inf`` and a value below the +minimum become the minimum. The gateway logs a warning naming the key, the value and what +replaced it. + +``max_hints`` takes an integer from 1 to 2147483647. A 64-bit integer outside that range +becomes the nearer bound, with the same warning. The parameter parser reads an integer that +does not fit in 64 bits as a double. A double, such as ``1.0e12`` or ``.nan``, is refused with +a warning and the default 10000 is used. + +In ``runtime_only`` and ``manifest_only`` mode the plugin reads its poll targets from the +ROS graph. In ``hybrid`` mode the merge pipeline passes it the discovered Apps, but the +gateway's refresh then calls it again with no Apps, which clears them. So a poll cycle mostly +reads the graph there too; only a cycle that starts between the two calls polls the nodes of +the online Apps that discovery bound to a node. A graph read skips hidden nodes (a name +starting with ``_``), the gateway's own node and its helper nodes (``_sub``, +``_fault_clients`` and ``_lifecycle_state_reader``): they carry no beacon. + +A parameter request that gets no answer within ``param_timeout_sec`` - waiting for the +service, listing parameters or getting their values - is given up and removed from its +client, and the node is skipped for the next 1, 2, 4 and then 8 poll cycles while it keeps +timing out. A node that answers is polled on every cycle, also when its answer carries no +values: rclpy answers so when one of the parameters asked for is declared with a type and no +value, and the plugin then stores no hint for the node. None of this is logged. + +Each node keeps one parameter client across poll cycles. A node that is no longer a poll +target, also when no node is, loses its client and its skip count. + Parameter Naming ^^^^^^^^^^^^^^^^ diff --git a/docs/config/server.rst b/docs/config/server.rst index 0171b268a..8f5410c7d 100644 --- a/docs/config/server.rst +++ b/docs/config/server.rst @@ -480,10 +480,12 @@ How long a departed node keeps being listed Once a node has actually left the ROS graph, ``GET /apps`` stops listing it within roughly one refresh: at most ``discovery.refresh_debounce_ms`` plus the 100 ms graph -poll, or ``refresh_interval_ms``, whichever comes first. Nothing is retained behind -that - every refresh rebuilds the entity set from a live read of the graph - so this -is the whole of the gateway's share, and it is the only part of a departure the -gateway can be held to. +poll, or ``refresh_interval_ms``, whichever comes first. Every refresh rebuilds the +entity set from a live read of the graph, so this is the whole of the gateway's share, +and it is the only part of a departure the gateway can be held to. What the gateway +remembers between reads - the names it saw running, described below - decides only +whether an entry the graph still lists after its participant left is exposed; it never +keeps listing a node the graph no longer lists. Before that point nothing here is promised, and the difference is not small: @@ -504,6 +506,83 @@ rather than assume a deadline. Anything that measures a departure should establi that the process has actually exited first, and only then hold the gateway to the figure above. +The graph itself can also keep a node after its process has exited, for as long as the +gateway runs. The DDS RMWs take a participant's node list from discovery messages on an +internal topic, read on a listener thread of their own. When that thread takes a message +only after the participant was removed, the message puts the node back, with an empty +enclave and none of its endpoints, and nothing removes it again. This page calls such an +entry a leftover. + +An empty enclave on its own does not make a leftover. A node on the far side of a DDS +router, or a node whose participant was created outside rcl (a micro-ROS agent's, for +example), also reads an empty enclave, and the router's far side resolves no endpoints +either. So the gateway leaves an entry out only for a node it has itself seen running. +Per node name, on every read of the graph: + +* an entry with an enclave is listed, and the name counts as seen running; +* an entry without an enclave is dropped when another entry of the same name has one, so a + restarted node is listed once; +* an entry without an enclave for a name the gateway has seen running is left out when the + graph resolves no publisher or subscriber for the node (a service or client is a request + and a reply endpoint, so it counts too); +* an entry without an enclave for a name the gateway has never seen running is listed. + +A leftover is left out of ``GET /apps`` and ``GET /apps/{id}``, out of the Functions +derived from namespaces, and out of the bare-name collision check that would otherwise +rename a live node sharing its name. The read that starts leaving a node out logs: + +.. code-block:: text + + [WARN] [ros2_medkit_gateway]: Node '/ns/name' is not exposed: this gateway saw it running, and the ROS graph still lists it after its participant left, with no endpoints + +Later reads that keep leaving it out stay quiet. The warning comes back only after a read +that did not leave the node out. + +Which names the gateway saw running comes from the reads themselves. The names running on +the previous read are that read's own list, so they cost nothing however many nodes run. A +name that ran on the previous read and does not run on this one - the graph lists no entry +of it, or only entries without an enclave - has departed, and the gateway remembers it until +it runs again or until it is forgotten on a read: the first read that finds no entry of it +more than 10 seconds after the first read that found none. While an entry of it without an +enclave is listed, those 10 seconds do not start. They cover the gap between the +participant's removal and the late message: the time the listener thread takes to process a +message it already holds, measured under a millisecond on an idle host and up to 0.64 s with +its process and eight busy threads sharing two cores. + +The name is forgotten only on a read. Reads come from graph changes, the refresh backstop +(``refresh_interval_ms``), the gateway's start and, in ``runtime_only`` mode, a request for an +App or a Function that a refresh removed from the entity cache while the request ran. A late +message that arrives before the read that forgets the name stays hidden, however long after +the 10 seconds it comes: the late message itself changes the graph, and the read that follows +finds its entry. Only a late message that arrives after that read is listed, like a node the +gateway never saw running. + +The gateway remembers at most 1024 departed names. Past that it forgets first the names the +graph has listed no entry of for longest, and then the names that departed longest ago; a +forgotten name whose only entries are leftovers is listed again, without a warning. So once +1024 leftovers are hidden, a node that departs with no entry of it left is the first name +forgotten, and a late message for it is listed. Every remembered leftover costs two endpoint +queries on each read, so the cap also bounds that work. + +A node that ran on this host and then appears only behind a DDS router, as an entry without +an enclave, is hidden like a leftover if it appears before the gateway forgets the name or +while a leftover of it is still listed. + +The startup discovery summary counts peer nodes through the same memory, so in +``runtime_only`` mode, and in ``hybrid`` mode with the runtime layer enabled, it counts what +discovery would list at that moment. When discovery does not read the graph itself - in +``manifest_only`` mode, or in ``hybrid`` mode with ``discovery.runtime.enabled: false`` - the +summary's read is the first, so no node has been seen running: it counts every entry the +graph lists, except an entry without an enclave next to an entry of the same name with one. + +The ``param_beacon`` plugin reads the graph itself only when discovery gives it no poll +targets, and then through a memory of its own: a node its reads saw running is not polled +once only a leftover of it is listed. It logs nothing about it. + +A node restarted while a leftover of its previous instance is still listed can show no +services, and so no operations, because the graph answers per-node queries from whichever +of the two participants sorts first. + Thread Pools ------------ diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md index a7fe897cb..1fcd41e2d 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md @@ -11,8 +11,35 @@ through standard ROS 2 parameters. 3. Entity metadata is mapped into the SOVD hierarchy via `BeaconEntityMapper` 4. Results are exposed at the `x-medkit-param-beacon` vendor extension endpoint -In non-hybrid discovery mode, the plugin discovers poll targets automatically from the -ROS 2 graph. In hybrid mode, targets come from the manifest. +In `runtime_only` and `manifest_only` discovery the plugin reads its poll targets from the +ROS 2 graph. In `hybrid` mode the merge pipeline passes it the discovered Apps, but the +gateway's refresh then calls it again with no Apps, which clears them, so a poll cycle mostly +reads the graph there too. A graph read skips hidden nodes (a name starting with `_`), the +gateway's own node and its helper nodes (`_sub`, `_fault_clients`, +`_lifecycle_state_reader`): they carry no beacon. The plugin remembers what its own +reads saw: a node that ran on one of its reads is not polled once the graph lists only a +leftover of it, an entry with an empty enclave and no endpoints, which a node whose +participant has left can leave behind (see "How long a departed node keeps being listed" in +the gateway's `docs/config/server.rst`). + +Each node keeps one parameter client across poll cycles. A node that is no longer a target, +also when no node is, loses its client. A parameter request that gets no answer within +`param_timeout_sec` - waiting for the service, listing parameters or getting their values - +is given up and removed from its client, and the node is skipped for the next 1, 2, 4 and +then 8 poll cycles while it keeps timing out, so a node whose parameter services never answer +does not stall polling. A node that answers is polled on every cycle, also when its answer +carries no values: rclpy answers so when one of the parameters asked for is declared with a +type and no value, and the plugin then stores no hint. None of this is logged. + +Every duration below takes its minimum (0.1 s, `beacon_expiry_sec` 1.0 s) up to 2147483647 s, +the longest wait Fast DDS keeps without the poll thread spinning. A value above the maximum, +including `.inf` in a parameter file, becomes the maximum; NaN, `-.inf` and a value below the +minimum become the minimum. Each replacement is logged as a warning. + +`max_hints` takes an integer from 1 to 2147483647. A 64-bit integer outside that range becomes +the nearer bound. The parameter parser reads an integer that does not fit in 64 bits as a +double. A double, such as `1.0e12` or `.nan`, is refused and the default 10000 is used. Both +are logged as warnings. ## Configuration diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst index 3dd579cb3..e02376f5f 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst @@ -21,9 +21,9 @@ components. How It Works ------------ -1. During each poll cycle, the plugin retrieves the current list of ROS 2 nodes - from the gateway's entity cache -2. For each node, it creates (or reuses) an ``AsyncParametersClient`` and fetches +1. During each poll cycle, the plugin takes its targets from the last ``introspect()`` + input, or reads the ROS 2 graph when that input had none +2. For each node, it creates (or reuses) a ``RealParameterClient`` and fetches all parameters matching the configured prefix 3. Parameters are parsed into a ``BeaconHint``: ``entity_id``, ``stable_id``, ``function_ids``, ``metadata.*`` keys, etc. @@ -46,18 +46,25 @@ their parameters once at startup. Client Management ~~~~~~~~~~~~~~~~~ -The plugin maintains a cache of ``AsyncParametersClient`` instances keyed by node -FQN. Clients for nodes that disappear from the graph are evicted after a -configurable timeout. A lock ordering protocol (``nodes_mutex_`` then -``clients_mutex_`` then ``param_ops_mutex_``) prevents deadlocks between the -poll thread and the introspection callback. +The plugin keeps one ``RealParameterClient`` per node FQN across cycles. It holds an +``rclcpp::Client`` for ``list_parameters`` and one for ``get_parameters`` on the plugin's +own node, and spins that node on its own executor for the length of a call. A request that +gets no answer within ``param_timeout_sec`` is removed from its client with +``remove_pending_request()``, so a node that never answers leaves nothing pending. + +At the start of each cycle the poll thread drops the clients of nodes that are not among the +cycle's targets, also when there are none. Only the poll thread spins the plugin's node, and +only inside a call, so no executor holds a client when the poll thread, or ``shutdown()`` +after joining it, destroys one. A lock ordering protocol (``nodes_mutex_`` then +``clients_mutex_`` then ``param_ops_mutex_``) prevents deadlocks between the poll thread and +the introspection callback. Backoff and Budget ~~~~~~~~~~~~~~~~~~ Nodes that fail to respond (timeout, unavailable) accumulate a backoff counter. Subsequent poll cycles skip backed-off nodes with exponentially increasing skip -counts. A per-cycle time budget (default 10 seconds) prevents a few slow nodes +counts. Any answer clears the counter, including a get answer without values. A per-cycle time budget (default 10 seconds) prevents a few slow nodes from starving the rest of the poll targets. The start offset rotates each cycle so that all nodes eventually get polled even under budget pressure. diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp index 97e1bd5cf..020dd712f 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include #include #include @@ -39,6 +41,7 @@ #include "ros2_medkit_gateway/core/plugins/plugin_types.hpp" #include "ros2_medkit_gateway/core/providers/introspection_provider.hpp" #include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_param_beacon/parameter_client_interface.hpp" class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, @@ -46,6 +49,10 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public: ParameterBeaconPlugin() = default; ~ParameterBeaconPlugin() noexcept override; + ParameterBeaconPlugin(const ParameterBeaconPlugin &) = delete; + ParameterBeaconPlugin & operator=(const ParameterBeaconPlugin &) = delete; + ParameterBeaconPlugin(ParameterBeaconPlugin &&) = delete; + ParameterBeaconPlugin & operator=(ParameterBeaconPlugin &&) = delete; /// Constructor with injectable client factory (for testing). explicit ParameterBeaconPlugin(ros2_medkit_param_beacon::ParameterClientFactory factory) @@ -66,6 +73,9 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, ros2_medkit_beacon::BeaconHintStore & store() { return *store_; } + const rclcpp::Node::SharedPtr & param_node() const { + return param_node_; + } private: // Polling @@ -77,9 +87,17 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, // Client management std::shared_ptr get_or_create_client(const std::string & fqn); - void evict_stale_clients(); + /// Drop the client, and the backoff, of every node that is not among this cycle's targets. + void evict_stale_clients(const std::vector & targets); + /// Count one more timeout for the node and set how many cycles skip it. + void back_off(const std::string & fqn); // Config + /// Longest duration in seconds. Fast DDS keeps a wait's seconds in an int32; a longer wait spins. + static constexpr double kMaxSeconds = 2147483647.0; + /// max_hints takes 1 to kMaxHints. + static constexpr std::int64_t kMaxHints = 2147483647; + static constexpr std::size_t kDefaultMaxHints = 10000; std::string parameter_prefix_{"ros2_medkit.discovery"}; std::chrono::duration poll_interval_{5.0}; double poll_budget_sec_{10.0}; @@ -87,6 +105,8 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, // State ros2_medkit_gateway::RosPluginContext * ctx_{nullptr}; + /// The gateway node's FQN. Graph reads skip it and its helper nodes. + std::string gateway_fqn_; rclcpp::Node::SharedPtr param_node_; std::thread poll_thread_; std::atomic shutdown_requested_{false}; @@ -102,12 +122,14 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, std::map> clients_; ros2_medkit_param_beacon::ParameterClientFactory client_factory_; - // Serialization for SyncParametersClient operations (never hold while acquiring clients_mutex_) + // Serializes parameter client calls (never hold while acquiring clients_mutex_) std::mutex param_ops_mutex_; // Node list (shared between introspect and poll threads) mutable std::shared_mutex nodes_mutex_; std::vector poll_targets_; + // Reads the graph when poll_targets_ is empty. Hides leftovers of nodes its own reads saw running. + ros2_medkit_gateway::ros2_common::GraphNodeListReader graph_node_reader_; // Backoff tracking std::unordered_map backoff_counts_; diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp index 1524e6990..624f7cac8 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp @@ -14,15 +14,21 @@ #pragma once +#include #include +#include #include #include +#include #include #include #include +#include +#include #include #include +#include namespace ros2_medkit_param_beacon { @@ -34,34 +40,103 @@ class ParameterClientInterface { virtual bool wait_for_service(std::chrono::duration timeout) = 0; + /// Throws when no answer arrives in time. virtual rcl_interfaces::msg::ListParametersResult list_parameters(const std::vector & prefixes, uint64_t depth) = 0; + /// Throws when no answer arrives in time. An answer without values gives an empty vector. virtual std::vector get_parameters(const std::vector & names) = 0; }; -/// Production implementation wrapping rclcpp::SyncParametersClient. +/// Production implementation: a client per parameter service, spun on its own executor. +/// +/// Every call waits at most `timeout`. A request that gets no answer in time is removed from its client. class RealParameterClient : public ParameterClientInterface { public: - RealParameterClient(rclcpp::Node::SharedPtr node, const std::string & target_node) - : client_(std::make_shared(node, target_node)) { + RealParameterClient(const rclcpp::Node::SharedPtr & node, const std::string & target_node, + std::chrono::duration timeout) + : node_(node->get_node_base_interface()) + , list_client_(create_client(*node, target_node + "/list_parameters")) + , get_client_(create_client(*node, target_node + "/get_parameters")) + , timeout_(std::chrono::duration_cast(timeout)) { } bool wait_for_service(std::chrono::duration timeout) override { - return client_->wait_for_service(std::chrono::duration_cast(timeout)); + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::duration_cast(timeout); + if (!list_client_->wait_for_service( + std::max(std::chrono::steady_clock::duration::zero(), deadline - std::chrono::steady_clock::now()))) { + return false; + } + return get_client_->wait_for_service( + std::max(std::chrono::steady_clock::duration::zero(), deadline - std::chrono::steady_clock::now())); } rcl_interfaces::msg::ListParametersResult list_parameters(const std::vector & prefixes, uint64_t depth) override { - return client_->list_parameters(prefixes, depth); + auto request = std::make_shared(); + request->prefixes = prefixes; + request->depth = depth; + return call(*list_client_, request)->result; } std::vector get_parameters(const std::vector & names) override { - return client_->get_parameters(names); + auto request = std::make_shared(); + request->names = names; + const auto response = call(*get_client_, request); + std::vector parameters; + // A service that cannot get one of the names answers with no values. + if (response->values.size() != names.size()) { + return parameters; + } + parameters.reserve(names.size()); + for (std::size_t i = 0; i < names.size(); ++i) { + parameters.emplace_back(names[i], rclcpp::ParameterValue(response->values[i])); + } + return parameters; + } + + /// Drops the requests still waiting for an answer and returns how many there were. + std::size_t prune_pending_requests() { + return list_client_->prune_pending_requests() + get_client_->prune_pending_requests(); } private: - std::shared_ptr client_; + template + static typename rclcpp::Client::SharedPtr create_client(rclcpp::Node & node, const std::string & name) { +#if RCLCPP_VERSION_MAJOR >= 28 + return node.create_client(name, rclcpp::ParametersQoS()); +#else + return node.create_client(name, rmw_qos_profile_parameters); +#endif + } + + template + typename ServiceT::Response::SharedPtr call(rclcpp::Client & client, + const typename ServiceT::Request::SharedPtr & request) { + auto future = client.async_send_request(request); + executor_.add_node(node_); + rclcpp::FutureReturnCode code = rclcpp::FutureReturnCode::INTERRUPTED; + try { + code = executor_.spin_until_future_complete(future, timeout_); + } catch (...) { + executor_.remove_node(node_); + client.remove_pending_request(future); + throw; + } + executor_.remove_node(node_); + if (code != rclcpp::FutureReturnCode::SUCCESS) { + client.remove_pending_request(future); + throw std::runtime_error(std::string("No answer from ") + client.get_service_name() + " in time"); + } + return future.get(); + } + + rclcpp::node_interfaces::NodeBaseInterface::SharedPtr node_; + rclcpp::Client::SharedPtr list_client_; + rclcpp::Client::SharedPtr get_client_; + std::chrono::nanoseconds timeout_; + rclcpp::executors::SingleThreadedExecutor executor_; }; /// Factory function type for creating parameter clients. diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp index a00dc5ff5..c5a47243e 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp @@ -14,8 +14,16 @@ #include "ros2_medkit_param_beacon/param_beacon_plugin.hpp" +#include #include +#include +#include +#include +#include +#include +#include #include +#include #include #include @@ -35,10 +43,8 @@ using ros2_medkit_gateway::PluginContext; using ros2_medkit_gateway::SovdEntityType; ParameterBeaconPlugin::~ParameterBeaconPlugin() noexcept { - // On Lyrical (originally observed on Rolling), ~rclcpp::Node can throw - // graph_listener::NodeNotFoundError once rclcpp::shutdown() has invalidated - // the context. An exception escaping a destructor calls std::terminate(), - // so swallow it here. + // shutdown() can throw std::system_error when it locks clients_mutex_. An exception leaving this + // destructor terminates the process. try { shutdown(); } catch (...) { @@ -57,30 +63,58 @@ void ParameterBeaconPlugin::configure(const nlohmann::json & config) { auto beacon_ttl = config.value("beacon_ttl_sec", 15.0); auto beacon_expiry = config.value("beacon_expiry_sec", 300.0); - auto max_hints = static_cast(std::max(config.value("max_hints", 10000), 1)); - // Clamp to safe minimums - if (poll_interval_.count() < 0.1) { - log_warn("poll_interval_sec clamped from " + std::to_string(poll_interval_.count()) + " to 0.1"); - poll_interval_ = std::chrono::duration(0.1); - } - if (poll_budget_sec_ < 0.1) { - log_warn("poll_budget_sec clamped from " + std::to_string(poll_budget_sec_) + " to 0.1"); - poll_budget_sec_ = 0.1; - } - if (param_timeout_sec_ < 0.1) { - log_warn("param_timeout_sec clamped from " + std::to_string(param_timeout_sec_) + " to 0.1"); - param_timeout_sec_ = 0.1; - } - if (beacon_ttl < 0.1) { - log_warn("beacon_ttl_sec clamped from " + std::to_string(beacon_ttl) + " to 0.1"); - beacon_ttl = 0.1; - } - if (beacon_expiry < 1.0) { - log_warn("beacon_expiry_sec clamped from " + std::to_string(beacon_expiry) + " to 1.0"); - beacon_expiry = 1.0; - } - // max_hints already clamped to >= 1 via std::max above + // max_hints is checked as int64 before it narrows. An integer outside 1 to kMaxHints becomes the nearer + // bound. Any other value, a double included, is rejected and the default stays. + auto read_max_hints = [this, &config]() -> std::size_t { + const auto it = config.find("max_hints"); + if (it == config.end()) { + return kDefaultMaxHints; + } + if (!it->is_number_integer()) { + std::ostringstream message; + message << std::setprecision(12) << "max_hints "; + if (it->is_number_float()) { + message << it->get(); + } else { + message << it->dump(); + } + message << " is not an integer, using " << kDefaultMaxHints; + log_warn(message.str()); + return kDefaultMaxHints; + } + // Only an unsigned JSON integer can exceed int64; it is above kMaxHints either way. + constexpr auto kInt64Max = std::numeric_limits::max(); + const bool beyond_int64 = + it->is_number_unsigned() && it->get() > static_cast(kInt64Max); + const std::int64_t value = beyond_int64 ? kInt64Max : it->get(); + const std::int64_t clamped = std::clamp(value, 1, kMaxHints); + if (clamped != value) { + log_warn("max_hints clamped from " + it->dump() + " to " + std::to_string(clamped)); + } + return static_cast(clamped); + }; + const std::size_t max_hints = read_max_hints(); + + // A duration above kMaxSeconds, +inf included, becomes kMaxSeconds. NaN and a duration below the + // minimum become the minimum. + auto clamp_seconds = [this](const char * key, double value, double minimum) { + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite: it lets NaN through. + // NOLINTNEXTLINE(readability-simplify-boolean-expr) + if (!(std::isfinite(value) && value >= minimum && value <= kMaxSeconds)) { + const double clamped = value > kMaxSeconds ? kMaxSeconds : minimum; + std::ostringstream message; + message << std::setprecision(12) << key << " clamped from " << value << " to " << clamped; + log_warn(message.str()); + return clamped; + } + return value; + }; + poll_interval_ = std::chrono::duration(clamp_seconds("poll_interval_sec", poll_interval_.count(), 0.1)); + poll_budget_sec_ = clamp_seconds("poll_budget_sec", poll_budget_sec_, 0.1); + param_timeout_sec_ = clamp_seconds("param_timeout_sec", param_timeout_sec_, 0.1); + beacon_ttl = clamp_seconds("beacon_ttl_sec", beacon_ttl, 0.1); + beacon_expiry = clamp_seconds("beacon_expiry_sec", beacon_expiry, 1.0); // Config validation if (beacon_ttl <= poll_interval_.count()) { @@ -110,6 +144,7 @@ void ParameterBeaconPlugin::configure(const nlohmann::json & config) { void ParameterBeaconPlugin::set_context(PluginContext & context) { ctx_ = as_ros_plugin_context(context); + gateway_fqn_ = ctx_->node()->get_fully_qualified_name(); if (!store_) { store_ = std::make_unique(); @@ -121,12 +156,16 @@ void ParameterBeaconPlugin::set_context(PluginContext & context) { options.start_parameter_event_publisher(false); options.use_global_arguments(false); param_node_ = std::make_shared("_param_beacon_node", options); + // Join rclcpp's graph listener now. wait_for_service() would join it on first use, and a join + // after rclcpp shuts down fails half-way, so ~NodeGraph later throws and terminates the process. + param_node_->get_node_graph_interface()->get_graph_event(); // Set default client factory if not injected (tests inject mock factory) if (!client_factory_) { auto node = param_node_; - client_factory_ = [node](const std::string & target) { - return std::make_shared(node, target); + const std::chrono::duration timeout(param_timeout_sec_); + client_factory_ = [node, timeout](const std::string & target) { + return std::make_shared(node, target, timeout); }; } @@ -158,14 +197,8 @@ void ParameterBeaconPlugin::shutdown() { backoff_counts_.clear(); skip_remaining_.clear(); } - // ~rclcpp::Node can throw graph_listener::NodeNotFoundError on Lyrical - // (and Rolling) when the context was already torn down by rclcpp::shutdown(). Swallow - // it so the plugin_manager shutdown sequence (and the plugin destructor - // that calls back into us) does not abort the process. - try { - param_node_.reset(); - } catch (...) { - } + // With the default client factory, which holds a copy of the node, ~Node runs when client_factory_ is destroyed. + param_node_.reset(); } std::vector ParameterBeaconPlugin::get_routes() { @@ -256,33 +289,34 @@ void ParameterBeaconPlugin::poll_cycle() { // "rcl node's context is invalid" if the poll timer fires between // SIGINT handling and the executor stopping; swallow it so the // shutdown path isn't aborted by std::terminate. + // The reader leaves out the leftovers of nodes it saw running (see GraphNodeListReader). std::vector> names_and_ns; try { - names_and_ns = param_node_->get_node_graph_interface()->get_node_names_and_namespaces(); + names_and_ns = graph_node_reader_.read(*param_node_->get_node_graph_interface()).nodes; } catch (const std::runtime_error & ex) { - RCLCPP_DEBUG(param_node_->get_logger(), "get_node_names_and_namespaces threw during shutdown: %s", ex.what()); + RCLCPP_DEBUG(param_node_->get_logger(), "Reading the node list threw during shutdown: %s", ex.what()); return; } for (const auto & [name, ns] : names_and_ns) { - // Skip internal nodes (leading underscore) and the gateway - if (name.empty() || name[0] == '_' || name == "ros2_medkit_gateway") { + // Skip hidden nodes (leading underscore), the gateway and its helper nodes: they carry no beacon. + if (name.empty() || name[0] == '_') { + continue; + } + auto fqn = ros2_medkit_gateway::ros2_common::graph_node_fqn(name, ns); + if (fqn == gateway_fqn_ || ros2_medkit_gateway::ros2_common::is_own_gateway_helper_node(fqn, gateway_fqn_)) { continue; } - auto fqn = (ns == "/" ? "/" : ns + "/") + name; - targets.push_back(fqn); + targets.push_back(std::move(fqn)); } } + evict_stale_clients(targets); if (targets.empty()) { return; } - // Evict stale clients - evict_stale_clients(); - auto cycle_start = std::chrono::steady_clock::now(); auto n = targets.size(); - size_t polled = 0; for (size_t i = 0; i < n; ++i) { // Budget check @@ -302,7 +336,6 @@ void ParameterBeaconPlugin::poll_cycle() { } poll_node(fqn); - ++polled; } start_offset_ = (start_offset_ + 1) % n; @@ -318,39 +351,30 @@ void ParameterBeaconPlugin::poll_node(const std::string & fqn) { std::lock_guard ops_lock(param_ops_mutex_); if (!client->wait_for_service(std::chrono::duration(param_timeout_sec_))) { - // Timeout - apply backoff - auto & count = backoff_counts_[fqn]; - if (count < 100) { - ++count; // cap to prevent overflow - } - int skip = std::min(1 << std::min(count - 1, 3), 8); - skip_remaining_[fqn] = skip; + back_off(fqn); return; } - - // List parameters under prefix auto list_result = client->list_parameters({parameter_prefix_}, 0); - if (list_result.names.empty()) { - return; // No beacon parameters declared + std::vector params; + if (!list_result.names.empty()) { + params = client->get_parameters(list_result.names); } - // Fetch parameter values - auto params = client->get_parameters(list_result.names); + // The node answered, so it is not backed off. An answer without values stores no hint. + backoff_counts_.erase(fqn); + skip_remaining_.erase(fqn); - // Convert to BeaconHint auto hint = parse_parameters(fqn, params); if (hint.entity_id.empty()) { - return; // No entity_id parameter - skip + return; } - // Validate auto result = validate_beacon_hint(hint, limits_); if (!result.valid) { log_warn("Beacon hint rejected for '" + hint.entity_id + "': " + result.reason); return; } - // Store if (!store_->update(hint)) { if (!capacity_warned_) { log_warn("BeaconHintStore capacity reached (max_hints=" + std::to_string(store_->size()) + @@ -358,20 +382,18 @@ void ParameterBeaconPlugin::poll_node(const std::string & fqn) { capacity_warned_ = true; } } + } catch (const std::exception &) { + // No answer in time, or the node disappeared. + back_off(fqn); + } +} - // Reset backoff on success - backoff_counts_.erase(fqn); - skip_remaining_.erase(fqn); - - } catch (const std::exception & e) { - // Node disappeared or service error - apply backoff - auto & count = backoff_counts_[fqn]; - if (count < 100) { - ++count; // cap to prevent overflow - } - int skip = std::min(1 << std::min(count - 1, 3), 8); - skip_remaining_[fqn] = skip; +void ParameterBeaconPlugin::back_off(const std::string & fqn) { + auto & count = backoff_counts_[fqn]; + if (count < 100) { + ++count; // cap to prevent overflow } + skip_remaining_[fqn] = std::min(1 << std::min(count - 1, 3), 8); } BeaconHint ParameterBeaconPlugin::parse_parameters(const std::string & /*fqn*/, @@ -382,7 +404,7 @@ BeaconHint ParameterBeaconPlugin::parse_parameters(const std::string & /*fqn*/, std::string metadata_prefix = parameter_prefix_ + ".metadata."; for (const auto & param : params) { - auto param_name = param.get_name(); + const auto & param_name = param.get_name(); // Strip prefix to get the field name if (param_name.rfind(parameter_prefix_ + ".", 0) != 0) { @@ -444,19 +466,12 @@ ParameterBeaconPlugin::get_or_create_client(const std::string & fqn) { return client; } -void ParameterBeaconPlugin::evict_stale_clients() { - std::shared_lock nodes_lock(nodes_mutex_); +void ParameterBeaconPlugin::evict_stale_clients(const std::vector & targets) { + const std::unordered_set current(targets.begin(), targets.end()); std::lock_guard clients_lock(clients_mutex_); for (auto it = clients_.begin(); it != clients_.end();) { - bool found = false; - for (const auto & target : poll_targets_) { - if (target == it->first) { - found = true; - break; - } - } - if (!found) { + if (current.count(it->first) == 0) { backoff_counts_.erase(it->first); skip_remaining_.erase(it->first); it = clients_.erase(it); diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md index c3fcb99e8..3ca8b10d9 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md @@ -14,6 +14,17 @@ to enrich the SOVD entity tree. Hints transition through states: **active** (within TTL) -> **stale** (TTL expired, data still served with stale marker) -> **expired** (removed from store). +`beacon_ttl_sec` takes 0.1 to 2147483647 s, `beacon_expiry_sec` 1.0 to 2147483647 s and +`max_messages_per_second` 1 to 10000. A value above its maximum, including `.inf` in a +parameter file, becomes the maximum; NaN, `-.inf` and a value below the minimum become the +minimum. Each replacement is logged as a warning. Messages over the rate limit are dropped +without a log. + +`max_hints` takes an integer from 1 to 2147483647. A 64-bit integer outside that range becomes +the nearer bound. The parameter parser reads an integer that does not fit in 64 bits as a +double. A double, such as `1.0e12` or `.nan`, is refused and the default 10000 is used. Both +are logged as warnings. + ## Configuration ```yaml diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst index ccc797ef2..11bd54129 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst @@ -42,10 +42,17 @@ publish beacons. Rate Limiting ~~~~~~~~~~~~~ -A ``TokenBucket`` rate limiter (default 100 messages/second) protects the gateway -from beacon floods. The bucket refills continuously and drops excess messages with -a single log warning. The rate limiter is thread-safe, as the DDS callback may fire -from any executor thread. +A ``TokenBucket`` rate limiter (default 100 messages/second, at most 10000) protects the +gateway from beacon floods. The bucket refills continuously and drops excess messages +without a log. The rate limiter is thread-safe, as the DDS callback may fire from any +executor thread. + +``configure()`` replaces a value above its maximum, ``+inf`` included, by the maximum, and +NaN or a value below the minimum by the minimum, with a warning. The check is the in-range +test negated, so NaN fails it. ``max_hints`` is read as an int64 before it is narrowed; a +64-bit integer outside 1 to 2147483647 becomes the nearer bound, and a value that is not an +integer keeps the default, each with a warning. The parameter parser reads an integer that +does not fit in 64 bits as a double, so it keeps the default. Timestamp Back-Projection ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp index e5753100d..de63d6f87 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include #include #include @@ -77,11 +79,17 @@ class TokenBucket { class TopicBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public ros2_medkit_gateway::IntrospectionProvider { public: + TopicBeaconPlugin() = default; + ~TopicBeaconPlugin() noexcept override; + TopicBeaconPlugin(const TopicBeaconPlugin &) = delete; + TopicBeaconPlugin & operator=(const TopicBeaconPlugin &) = delete; + TopicBeaconPlugin(TopicBeaconPlugin &&) = delete; + TopicBeaconPlugin & operator=(TopicBeaconPlugin &&) = delete; + std::string name() const override; void configure(const nlohmann::json & config) override; void set_context(ros2_medkit_gateway::PluginContext & context) override; void shutdown() override; - ~TopicBeaconPlugin() noexcept override; std::vector get_routes() override; ros2_medkit_gateway::IntrospectionResult introspect(const ros2_medkit_gateway::IntrospectionInput & input) override; @@ -94,7 +102,15 @@ class TopicBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public ros2 } private: - void on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr & msg); + void on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg); + + /// Longest TTL and expiry in seconds, the same bound as parameter_beacon's durations. + static constexpr double kMaxSeconds = 2147483647.0; + /// Highest rate limit: one beacon per second for each of the store's default 10000 hints. + static constexpr double kMaxMessagesPerSecond = 10000.0; + /// max_hints takes 1 to kMaxHints. + static constexpr std::int64_t kMaxHints = 2147483647; + static constexpr std::size_t kDefaultMaxHints = 10000; std::string topic_{"/ros2_medkit/discovery"}; ros2_medkit_gateway::RosPluginContext * ctx_{nullptr}; diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp index c133cb9c4..75014c444 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp @@ -16,6 +16,12 @@ #include #include +#include +#include +#include +#include +#include +#include #include "ros2_medkit_beacon_common/beacon_response_builder.hpp" @@ -33,14 +39,7 @@ using ros2_medkit_gateway::PluginContext; using ros2_medkit_gateway::SovdEntityType; TopicBeaconPlugin::~TopicBeaconPlugin() noexcept { - // On Lyrical (originally observed on Rolling), ~rclcpp::Subscription can throw - // graph_listener::NodeNotFoundError once rclcpp::shutdown() has invalidated - // the context. An exception escaping a destructor calls std::terminate(), - // so swallow it here. - try { - shutdown(); - } catch (...) { - } + shutdown(); } std::string TopicBeaconPlugin::name() const { @@ -53,23 +52,57 @@ void TopicBeaconPlugin::configure(const nlohmann::json & config) { BeaconHintStore::Config store_config; auto beacon_ttl = config.value("beacon_ttl_sec", 10.0); auto beacon_expiry = config.value("beacon_expiry_sec", 300.0); - auto max_hints = static_cast(std::max(config.value("max_hints", 10000), 1)); auto max_mps = config.value("max_messages_per_second", 100.0); - // Clamp to safe minimums - if (beacon_ttl < 0.1) { - log_warn("beacon_ttl_sec clamped from " + std::to_string(beacon_ttl) + " to 0.1"); - beacon_ttl = 0.1; - } - if (beacon_expiry < 1.0) { - log_warn("beacon_expiry_sec clamped from " + std::to_string(beacon_expiry) + " to 1.0"); - beacon_expiry = 1.0; - } - // max_hints already clamped to >= 1 via std::max above - if (max_mps < 1.0) { - log_warn("max_messages_per_second clamped from " + std::to_string(max_mps) + " to 1.0"); - max_mps = 1.0; - } + // max_hints is checked as int64 before it narrows. An integer outside 1 to kMaxHints becomes the nearer + // bound. Any other value, a double included, is rejected and the default stays. + auto read_max_hints = [this, &config]() -> std::size_t { + const auto it = config.find("max_hints"); + if (it == config.end()) { + return kDefaultMaxHints; + } + if (!it->is_number_integer()) { + std::ostringstream message; + message << std::setprecision(12) << "max_hints "; + if (it->is_number_float()) { + message << it->get(); + } else { + message << it->dump(); + } + message << " is not an integer, using " << kDefaultMaxHints; + log_warn(message.str()); + return kDefaultMaxHints; + } + // Only an unsigned JSON integer can exceed int64; it is above kMaxHints either way. + constexpr auto kInt64Max = std::numeric_limits::max(); + const bool beyond_int64 = + it->is_number_unsigned() && it->get() > static_cast(kInt64Max); + const std::int64_t value = beyond_int64 ? kInt64Max : it->get(); + const std::int64_t clamped = std::clamp(value, 1, kMaxHints); + if (clamped != value) { + log_warn("max_hints clamped from " + it->dump() + " to " + std::to_string(clamped)); + } + return static_cast(clamped); + }; + const std::size_t max_hints = read_max_hints(); + + // A value above its maximum, +inf included, becomes the maximum. NaN and a value below the minimum + // become the minimum. + auto clamp = [this](const char * key, double value, double minimum, double maximum) { + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite: it lets NaN through. + // NOLINTNEXTLINE(readability-simplify-boolean-expr) + if (!(std::isfinite(value) && value >= minimum && value <= maximum)) { + const double clamped = value > maximum ? maximum : minimum; + std::ostringstream message; + message << std::setprecision(12) << key << " clamped from " << value << " to " << clamped; + log_warn(message.str()); + return clamped; + } + return value; + }; + beacon_ttl = clamp("beacon_ttl_sec", beacon_ttl, 0.1, kMaxSeconds); + beacon_expiry = clamp("beacon_expiry_sec", beacon_expiry, 1.0, kMaxSeconds); + max_mps = clamp("max_messages_per_second", max_mps, 1.0, kMaxMessagesPerSecond); store_config.beacon_ttl_sec = beacon_ttl; store_config.beacon_expiry_sec = beacon_expiry; @@ -99,7 +132,8 @@ void TopicBeaconPlugin::set_context(PluginContext & context) { // Create subscription on configured topic subscription_ = node->create_subscription( - topic_, rclcpp::QoS(100).reliable(), [this](const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr msg) { + topic_, rclcpp::QoS(100).reliable(), + [this](const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg) { on_beacon(msg); }); @@ -114,13 +148,8 @@ void TopicBeaconPlugin::shutdown() { if (shutdown_requested_.exchange(true)) { return; } - // ~rclcpp::Subscription can throw on Lyrical (and Rolling) when the rclcpp - // context was torn down before us; swallow so plugin_manager shutdown and - // the plugin destructor calling back into us do not abort the process. - try { - subscription_.reset(); - } catch (...) { - } + // The callback captures this. A callback the executor already took returns early in on_beacon(). + subscription_.reset(); } std::vector TopicBeaconPlugin::get_routes() { @@ -172,7 +201,7 @@ IntrospectionResult TopicBeaconPlugin::introspect(const IntrospectionInput & inp return result; } -void TopicBeaconPlugin::on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr & msg) { +void TopicBeaconPlugin::on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg) { if (shutdown_requested_.load()) { return; } diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 5c0f55990..c475170ff 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -7,7 +7,7 @@ HTTP gateway node for the ros2_medkit diagnostics system. The ROS 2 Medkit Gateway exposes ROS 2 system information and data through a RESTful HTTP API. It automatically discovers nodes in the ROS 2 system, organizes them into a SOVD-aligned entity hierarchy (Areas, Components, Apps, Functions), and provides endpoints to query and interact with them. **Key Features:** -- **Auto-discovery**: Automatically detects ROS 2 nodes and topics +- **Auto-discovery**: Automatically detects ROS 2 nodes and topics. A node the gateway saw running that the ROS graph still lists after its participant left, with no endpoints, is not exposed, and a warning is logged when that starts (see "How long a departed node keeps being listed" in `docs/config/server.rst`) - **SOVD entity model**: Areas, Components (host-level), Apps (ROS 2 nodes), and Functions (namespace-based logical grouping) - **REST API**: Standard HTTP/JSON interface - **Incremental entity cache**: Discovery refresh diffs add/remove/change and performs zero structural allocations in the cache layer at steady state (object-pool backed, fixed capacity reserved at init via `entity_cache.capacity`) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp index c650d0212..6ff15b3a0 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp @@ -27,6 +27,7 @@ #include "ros2_medkit_gateway/discovery/manifest/manifest_manager.hpp" #include "ros2_medkit_gateway/discovery/merge_pipeline.hpp" #include "ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include #include @@ -183,6 +184,9 @@ class DiscoveryManager : public ServiceActionResolver { */ std::vector discover_apps(); + /// The ROS graph's node list through runtime discovery's reader, without its leftovers. Logs nothing. + ros2_common::GraphNodeList read_graph_nodes(); + /** * @brief Discover all functions * @return Vector of discovered Function entities (empty in runtime-only mode) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index b935605ed..4836d63b7 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -64,6 +64,7 @@ #include "ros2_medkit_gateway/ros2/transports/ros2_topic_transport.hpp" #include "ros2_medkit_gateway/ros2/trigger_topic_subscriber.hpp" #include "ros2_medkit_gateway/ros2_common/callback_groups.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_gateway/trigger_fault_subscriber.hpp" namespace ros2_medkit_gateway { @@ -501,53 +502,7 @@ class GatewayNode : public rclcpp::Node { std::unique_ptr server_thread_; }; -/** - * @brief Is this node FQN one of the helper nodes the gateway runs in-process? - * - * True for the subscription executor's `_sub`, the fault-service - * transport's `_fault_clients`, and the lifecycle reader's - * `_lifecycle_state_reader`. None of these begins with '_', so the ROS 2 - * hidden-node convention does not cover them and the gateway would otherwise - * list its own plumbing as diagnosable Apps. They carry no parameters and no - * services of their own, so there is nothing to diagnose on them. - * - * False for the gateway node itself. The gateway IS a diagnosable App: its ROS - * parameters are served as that App's configurations, and callers read and - * write them at `/apps//configurations`. Excluding it would remove the - * only entity carrying, for instance, `aggregation.peer_auth_header`, and would - * make two gateways watching one graph disagree about that graph, because each - * would hide a different node. - * - * A fault_manager node sharing the process is NOT ours either: it is a - * separate, diagnosable component and stays visible. - * - * Exact matches only. A prefix test would also claim a genuine peer named - * `_monitor` or `2`, and dropping a real node is the worse error. - * - * Two FQN spellings are recognised, because the three creation sites do not - * agree on the namespace: the subscription node is created with the gateway's - * own namespace, while the fault-client and lifecycle-reader nodes are created - * from the gateway's node NAME alone and so take the process default. A - * node-specific namespace remap on the gateway (`-r :__ns:=/x`) moves - * the gateway and the subscription node and leaves the other two behind, which - * is why those two are also matched as `/`. - * - * CONTRACT, and the cost of that second spelling: a helper-named node in the - * root namespace is treated as plumbing whichever gateway created it. Two - * gateways that keep the default node name and differ only in namespace build - * the same literal `/_fault_clients` and - * `/_lifecycle_state_reader`, so the name cannot say whose it is, and - * each will claim the other's. What it is does not depend on who owns it - - * those nodes carry nothing to diagnose in either process - and the - * alternative is that every namespaced gateway serves and counts its own - * plumbing. The subscription node is exempt: it always follows its gateway's - * namespace, so a root-namespace one is provably another process's and stays - * visible. - * - * @param node_fqn Fully qualified node name to test ("/ns/node") - * @param self_fqn The gateway node's own FQN. An empty value matches nothing - */ -bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn); +using ros2_common::is_own_gateway_helper_node; /** * @brief Filter ROS 2 internal nodes from an app list diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp index 9a2a4eaff..cac62795e 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp @@ -21,6 +21,7 @@ #include "ros2_medkit_gateway/core/discovery/models/component.hpp" #include "ros2_medkit_gateway/core/discovery/models/function.hpp" #include "ros2_medkit_gateway/core/providers/introspection_provider.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_serialization/type_introspection.hpp" #include @@ -93,9 +94,13 @@ class Ros2RuntimeIntrospection : public IntrospectionProvider { // --------------------------------------------------------------------------- /// Discover the live nodes as Apps. Always queries the ROS 2 graph; do not - /// call from hot paths. + /// call from hot paths. A leftover (see ros2_common::GraphNodeListReader) is + /// not an App; the read that starts leaving it out logs it. std::vector discover_apps(); + /// The node list through discover_apps()'s reader, without its leftovers. Logs nothing. + ros2_common::GraphNodeList read_graph_nodes(); + /// Group nodes by namespace into Function entities (no graph query). std::vector discover_functions(const std::vector & apps); @@ -153,6 +158,11 @@ class Ros2RuntimeIntrospection : public IntrospectionProvider { std::map cached_topic_map_; bool topic_map_ready_{false}; + + /// Nodes this discovery saw running, shared by discover_apps() and read_graph_nodes(). + ros2_common::GraphNodeListReader graph_node_reader_; + /// Logs the leftovers discover_apps() leaves out. + ros2_common::LeftoverNodeReporter leftover_node_reporter_; }; } // namespace ros2_medkit_gateway::ros2 diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp new file mode 100644 index 000000000..2d78be2e3 --- /dev/null +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp @@ -0,0 +1,320 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +// Header-only: the discovery plugins and their tests use it without the gateway's libraries. + +namespace ros2_medkit_gateway::ros2_common { + +/// One entry of `get_node_names_with_enclaves()`: name, namespace, enclave. +using GraphNodeEntry = std::tuple; + +/// The ROS graph's node list as a GraphNodeListReader reads it. +struct GraphNodeList { + /// (name, namespace) of every entry kept, in graph order. + std::vector> nodes; + /// FQN of every node left out as a leftover, once each, in graph order. + std::vector leftovers; +}; + +/// Fully qualified name from the name and namespace the graph reports. +inline std::string graph_node_fqn(const std::string & name, const std::string & ns) { + if (ns.empty() || ns == "/") { + return "/" + name; + } + return ns + "/" + name; +} + +/** + * @brief Is this FQN one of the gateway's in-process helper nodes? + * + * True for `_sub`, `_fault_clients` and `_lifecycle_state_reader`, exact matches + * only. False for the gateway node itself, which is a diagnosable App. The last two take only the + * gateway's node name, so they also match as `/`; two gateways with one name in + * different namespaces both claim those root-namespace nodes. + * + * @param node_fqn Fully qualified node name to test ("/ns/node") + * @param self_fqn The gateway node's own FQN. An empty value matches nothing + */ +inline bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn) { + if (self_fqn.empty() || node_fqn.empty()) { + return false; + } + struct HelperNode { + const char * suffix; + bool follows_gateway_namespace; + }; + static constexpr std::array kHelperNodes{{ + {"_sub", true}, + {"_fault_clients", false}, + {"_lifecycle_state_reader", false}, + }}; + const auto last_slash = self_fqn.rfind('/'); + const std::string bare_name = last_slash == std::string::npos ? self_fqn : self_fqn.substr(last_slash + 1); + return std::any_of(kHelperNodes.begin(), kHelperNodes.end(), [&](const HelperNode & helper) { + if (node_fqn == self_fqn + helper.suffix) { + return true; + } + return !helper.follows_gateway_namespace && !bare_name.empty() && node_fqn == "/" + bare_name + helper.suffix; + }); +} + +/** + * @brief Whether the ROS graph resolves any publisher or subscriber for a node. + * + * On the DDS RMWs (rmw_dds_common), without demangling these also cover services and clients (`rq/`, `rr/` topics). + * A node the graph no longer lists has none. Other errors, such as a shut-down context, propagate. + */ +inline bool graph_node_has_endpoints(const rclcpp::node_interfaces::NodeGraphInterface & graph, + const std::string & name, const std::string & ns) { + try { + return !graph.get_publisher_names_and_types_by_node(name, ns, true).empty() || + !graph.get_subscriber_names_and_types_by_node(name, ns, true).empty(); + } catch (const rclcpp::exceptions::RCLError & error) { + if (error.ret == RCL_RET_NODE_NAME_NON_EXISTENT) { + return false; + } + throw; + } +} + +/** + * @brief Reads the ROS graph's node list, leaving out the leftovers of nodes it saw running. + * + * A leftover is an entry with an empty enclave and no endpoints that a late `ros_discovery_info` + * sample puts back after its participant left. Nothing removes it again. An empty enclave alone is + * not enough: micro-ROS and DDS-router nodes read the same, so only names this reader saw running + * are hidden. Per FQN, on every read: + * - an entry with an enclave is listed; + * - an entry without an enclave is dropped when another entry of the name has one; + * - an entry without an enclave for a departed name is left out when it has no endpoints; + * - any other entry without an enclave is listed. + * + * A name that ran on the previous read and does not run now has departed. It is forgotten when it + * runs again, or on the first read that finds no entry of it more than the hold after the first + * read that found none. A listed entry without an enclave stops that clock. At most `capacity` + * departed names are kept; the ones unlisted longest go first, then the ones departed longest ago. + * + * Thread-safe. A read holds the lock from the list query to the last endpoint query. + */ +class GraphNodeListReader { + public: + using Clock = std::chrono::steady_clock; + + /// How long after the first read without an entry a departed name is kept. It covers the listener + /// thread's delay in taking a late sample, which exceeds 0.5 s on a loaded host. + static constexpr std::chrono::seconds kDefaultHold{10}; + /// Most departed names kept. Each hidden leftover costs two endpoint queries per read. + static constexpr std::size_t kDefaultCapacity = 1024; + + GraphNodeListReader() = default; + GraphNodeListReader(Clock::duration hold, std::size_t capacity) : hold_(hold), capacity_(capacity) { + } + + /** + * @brief Read the node list, leaving out leftovers of nodes this reader saw running. + * + * Throws what the graph queries throw, for example once the context is shut down. + */ + GraphNodeList read(const rclcpp::node_interfaces::NodeGraphInterface & graph) { + std::lock_guard lock(mutex_); + auto entries = graph.get_node_names_with_enclaves(); + return filter_locked(entries, Clock::now(), [&graph](const std::string & name, const std::string & ns) { + return graph_node_has_endpoints(graph, name, ns); + }); + } + + /// The decision read() makes, for `entries` read at `now`. + /// @param has_endpoints `bool(const std::string & name, const std::string & ns)` + template + GraphNodeList filter(const std::vector & entries, Clock::time_point now, + HasEndpoints && has_endpoints) { + std::lock_guard lock(mutex_); + return filter_locked(entries, now, std::forward(has_endpoints)); + } + + /// How many departed names the reader keeps. + std::size_t remembered() const { + std::lock_guard lock(mutex_); + return departed_.size(); + } + + private: + /// A name the reader saw running that no entry with an enclave lists. + struct Departed { + /// The read that first found the name not running. + Clock::time_point departed_at; + /// First read of the current run that listed no entry of the name; empty while one is listed. + std::optional absent_since; + }; + + template + GraphNodeList filter_locked(const std::vector & entries, Clock::time_point now, + HasEndpoints && has_endpoints) { + std::unordered_set running; + std::unordered_set listed; + for (const auto & [name, ns, enclave] : entries) { + auto fqn = graph_node_fqn(name, ns); + if (!enclave.empty()) { + running.insert(fqn); + } + listed.insert(std::move(fqn)); + } + remember(std::move(running), listed, now); + + GraphNodeList result; + result.nodes.reserve(entries.size()); + std::unordered_map resolved; + for (const auto & [name, ns, enclave] : entries) { + if (!enclave.empty()) { + result.nodes.emplace_back(name, ns); + continue; + } + auto fqn = graph_node_fqn(name, ns); + if (running_.count(fqn) > 0) { + continue; + } + if (departed_.count(fqn) == 0) { + result.nodes.emplace_back(name, ns); + continue; + } + auto found = resolved.find(fqn); + if (found == resolved.end()) { + const bool has = has_endpoints(name, ns); + found = resolved.emplace(fqn, has).first; + if (!has) { + result.leftovers.push_back(fqn); + } + } + if (found->second) { + result.nodes.emplace_back(name, ns); + } + } + return result; + } + + void remember(std::unordered_set running, const std::unordered_set & listed, + Clock::time_point now) { + for (const auto & fqn : running) { + departed_.erase(fqn); + } + for (const auto & fqn : running_) { + if (running.count(fqn) == 0) { + departed_.try_emplace(fqn, Departed{now, std::nullopt}); + } + } + running_ = std::move(running); + for (auto it = departed_.begin(); it != departed_.end();) { + auto & departed = it->second; + if (listed.count(it->first) > 0) { + departed.absent_since.reset(); + } else if (!departed.absent_since) { + departed.absent_since = now; + } else if (now - *departed.absent_since > hold_) { + it = departed_.erase(it); + continue; + } + ++it; + } + if (departed_.size() <= capacity_) { + return; + } + // Drop the names unlisted longest first, then the ones that departed longest ago. + std::vector::iterator> order; + order.reserve(departed_.size()); + for (auto it = departed_.begin(); it != departed_.end(); ++it) { + order.push_back(it); + } + const auto excess = static_cast(departed_.size() - capacity_); + std::nth_element(order.begin(), order.begin() + excess, order.end(), [](const auto & lhs, const auto & rhs) { + const auto & a = lhs->second; + const auto & b = rhs->second; + if (a.absent_since.has_value() != b.absent_since.has_value()) { + return a.absent_since.has_value(); + } + if (a.absent_since) { + return *a.absent_since < *b.absent_since; + } + return a.departed_at < b.departed_at; + }); + for (auto it = order.begin(); it != order.begin() + excess; ++it) { + departed_.erase(*it); + } + } + + Clock::duration hold_{kDefaultHold}; + std::size_t capacity_{kDefaultCapacity}; + mutable std::mutex mutex_; + /// Names an entry with an enclave listed on the latest read. + std::unordered_set running_; + /// Names seen running that no entry with an enclave lists since, at most `capacity_`. + std::unordered_map departed_; +}; + +/** + * @brief Logs a WARN for each leftover on the first read of a run of reads that leave it out. + * + * Thread-safe: the refresh and HTTP handlers both read the node list. + */ +class LeftoverNodeReporter { + public: + explicit LeftoverNodeReporter(rclcpp::Logger logger) : logger_(std::move(logger)) { + } + + /// Log the leftovers `list` leaves out that the previous read did not. Returns their FQNs. + std::vector report(const GraphNodeList & list) { + std::vector newly_left_out; + std::set current(list.leftovers.begin(), list.leftovers.end()); + std::lock_guard lock(mutex_); + for (const auto & fqn : list.leftovers) { + if (reported_.count(fqn) == 0) { + newly_left_out.push_back(fqn); + RCLCPP_WARN(logger_, + "Node '%s' is not exposed: this gateway saw it running, and the ROS graph still lists it " + "after its participant left, with no endpoints", + fqn.c_str()); + } + } + reported_ = std::move(current); + return newly_left_out; + } + + private: + rclcpp::Logger logger_; + std::mutex mutex_; + std::set reported_; +}; + +} // namespace ros2_medkit_gateway::ros2_common diff --git a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp index 03e4af988..ec7304474 100644 --- a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp +++ b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp @@ -290,6 +290,10 @@ std::vector DiscoveryManager::discover_apps() { return runtime_introspection_->discover_apps(); } +ros2_common::GraphNodeList DiscoveryManager::read_graph_nodes() { + return runtime_introspection_->read_graph_nodes(); +} + std::vector DiscoveryManager::discover_functions() { if (config_.mode == DiscoveryMode::MANIFEST_ONLY && manifest_manager_ && manifest_manager_->is_manifest_active()) { return manifest_manager_->get_functions(); diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index 8da9a4aa9..b9fec6ca1 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -15,7 +15,6 @@ #include "ros2_medkit_gateway/gateway_node.hpp" #include -#include #include #include #include @@ -1598,47 +1597,6 @@ GatewayNode::GatewayNode(const rclcpp::NodeOptions & options) : Node("ros2_medki }); } -bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn) { - if (self_fqn.empty() || node_fqn.empty()) { - return false; - } - // The helper nodes the gateway creates inside its own process. Each one's FQN - // is fixed by how its creation site builds the node, which is not the same - // for all three: - // "_sub" Ros2SubscriptionExecutor passes the gateway's - // own namespace (ros2_subscription_executor.cpp), - // so this one always shares it. - // "_fault_clients" Ros2FaultServiceTransport and - // "_lifecycle_state_reader" Ros2LifecycleStateReader build their node from - // the gateway's node NAME alone, so they take - // whatever namespace the process defaults to. - // Usually that is the gateway's namespace too and all three spellings - // coincide. They come apart when only the gateway is moved - a node-specific - // remap, `-r :__ns:=/x` - which leaves the last two where the - // process default put them. Both spellings are then ours, so both are - // matched for those two; `_sub` is matched only in the gateway's namespace, - // because a root-namespace `_sub` provably belongs to another process. - struct HelperNode { - const char * suffix; - bool follows_gateway_namespace; - }; - static constexpr std::array kHelperNodes{{ - {"_sub", true}, - {"_fault_clients", false}, - {"_lifecycle_state_reader", false}, - }}; - const auto last_slash = self_fqn.rfind('/'); - const std::string bare_name = last_slash == std::string::npos ? self_fqn : self_fqn.substr(last_slash + 1); - // Exact matches only: a prefix test would also claim a genuine peer named - // "_monitor" or "2", and hiding a real node is the worse error. - return std::any_of(kHelperNodes.begin(), kHelperNodes.end(), [&](const HelperNode & helper) { - if (node_fqn == self_fqn + helper.suffix) { - return true; - } - return !helper.follows_gateway_namespace && !bare_name.empty() && node_fqn == "/" + bare_name + helper.suffix; - }); -} - size_t GatewayNode::count_peer_nodes(const std::vector> & nodes_and_namespaces, const std::string & self_fqn) { size_t count = 0; @@ -1688,8 +1646,8 @@ void GatewayNode::log_startup_summary() { ++topic_count; } } - peer_node_count = - count_peer_nodes(get_node_graph_interface()->get_node_names_and_namespaces(), get_fully_qualified_name()); + // Read through discovery's own reader, so a leftover discovery leaves out is not a peer either. + peer_node_count = count_peer_nodes(discovery_mgr_->read_graph_nodes().nodes, get_fully_qualified_name()); } catch (const std::exception & e) { RCLCPP_DEBUG(get_logger(), "Startup summary: graph query failed: %s", e.what()); } diff --git a/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp b/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp index 66381e166..f2eab8339 100644 --- a/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp +++ b/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp @@ -49,7 +49,12 @@ bool Ros2RuntimeIntrospection::is_internal_service(const std::string & service_p service_path.find("/_action/") != std::string::npos; } -Ros2RuntimeIntrospection::Ros2RuntimeIntrospection(rclcpp::Node * node) : node_(node) { +Ros2RuntimeIntrospection::Ros2RuntimeIntrospection(rclcpp::Node * node) + : node_(node), leftover_node_reporter_(node->get_logger()) { +} + +ros2_common::GraphNodeList Ros2RuntimeIntrospection::read_graph_nodes() { + return graph_node_reader_.read(*node_->get_node_graph_interface()); } void Ros2RuntimeIntrospection::set_config(const RuntimeConfig & config) { @@ -86,14 +91,14 @@ std::vector Ros2RuntimeIntrospection::discover_apps() { auto node_graph = node_->get_node_graph_interface(); std::vector> names_and_namespaces; try { - names_and_namespaces = node_graph->get_node_names_and_namespaces(); + // Leftovers go before the bare-name collision count, so they never rename a live node. + auto node_list = graph_node_reader_.read(*node_graph); + leftover_node_reporter_.report(node_list); + names_and_namespaces = std::move(node_list.nodes); } catch (const std::runtime_error & ex) { - // rclcpp throws "rcl node's context is invalid" when get_node_names_* - // is called after rclcpp::shutdown (e.g. refresh timer fires once - // between SIGINT handling and the executor stopping). Swallow and - // return empty so ~GatewayNode's shutdown path isn't aborted mid-run - // by std::terminate; callers handle empty gracefully. - RCLCPP_DEBUG(node_->get_logger(), "get_node_names_and_namespaces threw during shutdown: %s", ex.what()); + // A graph query throws once rclcpp is shut down, e.g. a refresh between SIGINT and the + // executor stopping. Return empty so the shutdown path does not end in std::terminate. + RCLCPP_DEBUG(node_->get_logger(), "Reading the node list threw during shutdown: %s", ex.what()); return {}; } diff --git a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md index 117a403ff..f675aea59 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/README.md @@ -85,6 +85,19 @@ manifest-declared node immortal. A managed lifecycle node that merely deactivate so a deactivation is never mistaken for a death either; that is `lifecycle_expectation`'s concern, not this one's (see "The boundary with `lifecycle_expectation`" below). +When the ROS graph keeps a leftover of a node after its participant left - an entry with an +empty enclave and no endpoints - the App leaves the runtime-only snapshot only while the +gateway remembers the node: a read of the graph saw it running, the late discovery sample that +left the leftover arrived before the gateway forgot the node, and the node was not among the +names forgotten once the gateway remembers more than 1024 departed nodes. The gateway forgets +a node on the first read that finds no entry of it more than 10 s after the first read that +found none; a late sample that arrives before that read stays hidden, however late. Reads come +from refreshes, the gateway's start and, in `runtime_only` mode, a request for an App or a +Function that a refresh removed from the entity cache while the request ran. +Otherwise the leftover stays in the snapshot as an online App, and this detector does not see +the node die (see "How long a departed node keeps being listed" in the gateway's +`docs/config/server.rst`). + Tracking is keyed on the STABLE fqn (`App::effective_fqn()`), never `App::id`: an id is recomputed every sweep and only gains a namespace prefix once a same-bare-name collision currently exists anywhere in the graph, so a live node's id can change out from under a key From f6dc89846890e352896a91609666e9ee686b4461 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Fri, 18 Sep 2026 13:55:35 +0200 Subject: [PATCH 25/25] test: reproduce graph leftovers of real nodes and check what each reader exposes ghost_node_injector gains --leftover: it runs a node in an rclcpp context of its own, captures the discovery message that node's participant publishes, removes the participant on request and, after a given delay or on a publish command, publishes the captured message again, the way a late sample re-creates the node. The participant GID is derived from the node's publisher GID and has to match a received discovery message before the node is reported ready. --announce adds more nodes to that participant's message. --ghost and --backed GIDs now sort before those of real participants, and their messages go out one per node in command-line order. test_graph_leftover_nodes checks discovery on one gateway: the leftover of a listed node is hidden when its late sample arrives inside the hold; nodes never seen running without an enclave, next to a ghost of their name, and a running node without endpoints are listed; a restarted node is listed once; a leftover does not rename a live node sharing its bare name; a node seen running that stays behind an endpoint is listed; 50 leftovers leave live nodes and a node started afterwards listed. The endpoint case stops the node only once GET /apps lists a ghost sent after the backed entry of its name, so the gateway's graph holds that entry when the node exits. The last case checks that leftovers the earlier cases hid are still hidden once they have been listed for longer than the hold, and that a late sample for a node that departed with no entry of it left is listed once a refresh ran past the hold. Every case first checks that the test's own graph lists the leftover, and cases that rely on the gateway's graph holding it check the gateway's warning. test_graph_leftover_nodes_scale announces more running nodes than discovery remembers and checks that a node departing next to them is still hidden. It then leaves all of them over and checks how many stay hidden, that a node departing with no entry of it left is the first name forgotten past the capacity, and that a new node is listed. test_graph_leftover_nodes_startup checks the startup peer count, and that parameter_beacon polls a node it never saw running and does not poll the leftover of a node it polled, after checking that the plugin's node shares the gateway's participant and that discovery warned about the leftover. test_param_beacon_out_of_range_config loads parameter_beacon once per sweep point: NaN, -inf, below the minimum, the minimum, the maximum, just above it, +inf and 3e9, plus one instance with a working configuration. It checks every logged clamp and that in-range values log none. Instances at the low points keep polling past nodes that never answer; instances whose timeout sits at the high points send one request to such a node and wait there, and no gateway thread uses more than half a core meanwhile; instances whose interval sits at the high points run at most one cycle. The working instance polls an rclpy node whose typed, unset parameter makes its get answers carry no values on every cycle, backs off a node whose get requests go unanswered, and keeps the request writer of its client for a node that never answers across cycles. The test counts a list answer when it is sent and a get request when a polling thread takes it, so it compares the two only on samples where every list answer has its get counted, and fails when the two counts do not settle to the same number. test_topic_beacon_config_bounds sweeps the same points over beacon_ttl_sec, beacon_expiry_sec and max_messages_per_second, checks every logged clamp, that each instance stores a beacon, and that a burst of ten is cut to one or two at 1 per second and passes whole at 10000 per second. Both tests load one more instance per max_hints point: 0, -5, 1, 2147483647, 2147483648, 2^32 + 1, 1e12 and NaN. They check each warning, that the instances whose max_hints ends at 1 keep one of two hints and are the only ones that reach capacity, and that the others keep both. The instance each test loads first answers the beacon endpoint and has NaN TTL and expiry. The tests give it one beacon for a node the gateway lists, stop refreshing it and check that the endpoint reports it stale and then removes it. topic_beacon's test checks that the removal comes no earlier than 1 s and within a few seconds. parameter_beacon's test stops refreshing while the beacon is active and times each change from the node's last get answer. The beacon must go stale between 0.3 s, its TTL, and 0.3 s later, and be removed between 1 s, its expiry, and 0.3 s later. The 0.3 s margin grows with the test time scale. The unit tests cover the reader's decisions, the hold, the eviction order and that running names do not count against the capacity, and graph_node_has_endpoints against a real rcl node before and after its context is shut down. The parameter_beacon unit tests run the real parameter client against parameter services that do not answer, answer without values or answer normally: a timed-out request leaves nothing pending, a node keeps one client across cycles, a graph read without targets drops every client, the gateway's helper nodes are not polled, a wait with no time left returns within 1 s for a service that does not exist, and the plugin's node can be destroyed after rclcpp shuts down. The leftover tests scale their default wait for the gateway's warning with the test time scale. The graph_watchdog ghost_departure scenario runs the watched node inside the fixture and expects GRAPH_NODE_DISAPPEARED once it leaves a leftover the gateway warned about. test_node_death_integration's X2 case attributes a fault report to a tick only after the fake fault service has recorded it. --- .../ros2_medkit_param_beacon/CMakeLists.txt | 6 +- .../test/test_param_beacon_plugin.cpp | 415 +++++++++++- .../test/test_topic_beacon_plugin.cpp | 10 +- src/ros2_medkit_gateway/CMakeLists.txt | 7 + .../test/test_graph_node_list.cpp | 294 +++++++++ .../CMakeLists.txt | 16 + .../demo_nodes/endpointless_node.cpp | 72 ++ .../demo_nodes/ghost_node_injector.cpp | 577 ++++++++++++++++ src/ros2_medkit_integration_tests/package.xml | 4 + .../ros2_medkit_test_utils/graph_fixtures.py | 187 ++++++ .../ros2_medkit_test_utils/launch_helpers.py | 8 +- .../test_graph_leftover_nodes.test.py | 561 ++++++++++++++++ .../test_graph_leftover_nodes_scale.test.py | 292 ++++++++ .../test_graph_leftover_nodes_startup.test.py | 283 ++++++++ ...t_param_beacon_out_of_range_config.test.py | 624 ++++++++++++++++++ .../test_topic_beacon_config_bounds.test.py | 339 ++++++++++ .../ros2_medkit_graph_watchdog/CMakeLists.txt | 6 + .../test/e2e/test_node_death_e2e.test.py | 90 ++- .../test/test_node_death_integration.cpp | 14 +- 19 files changed, 3780 insertions(+), 25 deletions(-) create mode 100644 src/ros2_medkit_gateway/test/test_graph_node_list.cpp create mode 100644 src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp create mode 100644 src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp create mode 100644 src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_topic_beacon_config_bounds.test.py diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt index 9320d0012..3e60bf7e5 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt @@ -85,11 +85,7 @@ if(BUILD_TESTING) include(ROS2MedkitTestDomain) # Include plugin source in test (plugin is MODULE/dlopen, test instantiates directly) - # TIMEOUT 180: the 12 test cases each spin a node for ~200-400 ms. Under - # plain gcc that's ~4 s total, well under the 30 s ctest default, but TSan - # instrumentation inflates that 5-10x and occasionally trips the 30 s cap - # (CI saw timeouts at test 4/12 around 27 s). 180 s covers the worst-case - # sanitizer slowdown without masking real hangs. + # TIMEOUT 180: the cases spin real nodes, and TSan slows them 5-10x past the 30 s default. medkit_add_gmock(test_param_beacon_plugin test/test_param_beacon_plugin.cpp src/param_beacon_plugin.cpp diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp index cf9503f4f..e826ad5b7 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp @@ -14,14 +14,29 @@ #include #include +#include +#include +#include #include +#include +#include +#include +#include +#include #include +#include #include #include #include #include +#include +#include +#include +#include +#include +#include #include #include "ros2_medkit_gateway/core/plugins/plugin_http_types.hpp" @@ -40,6 +55,7 @@ using ros2_medkit_gateway::PluginResponse; using ros2_medkit_gateway::RosPluginContext; using ros2_medkit_gateway::SovdEntityType; using ros2_medkit_param_beacon::ParameterClientInterface; +using ros2_medkit_param_beacon::RealParameterClient; using ::testing::_; using ::testing::Return; @@ -93,39 +109,41 @@ class MockPluginContext : public RosPluginContext { rclcpp::Node * node() const override { return node_; } - std::optional get_entity(const std::string &) const override { + std::optional get_entity(const std::string & /*entity_id*/) const override { return std::nullopt; } - std::vector get_child_apps(const std::string &) const override { + std::vector get_child_apps(const std::string & /*component_id*/) const override { return {}; } - nlohmann::json list_entity_faults(const std::string &) const override { + nlohmann::json list_entity_faults(const std::string & /*entity_id*/) const override { return nlohmann::json::array(); } - std::optional validate_entity_for_route(const PluginRequest &, PluginResponse &, - const std::string &) const override { + std::optional validate_entity_for_route(const PluginRequest & /*req*/, PluginResponse & /*res*/, + const std::string & /*entity_id*/) const override { return std::nullopt; } void register_capability(SovdEntityType type, const std::string & name) override { registered_capabilities_.push_back({type, name}); } - void register_entity_capability(const std::string &, const std::string &) override { + void register_entity_capability(const std::string & /*entity_id*/, const std::string & /*name*/) override { } - std::vector get_type_capabilities(SovdEntityType) const override { + std::vector get_type_capabilities(SovdEntityType /*type*/) const override { return {}; } - std::vector get_entity_capabilities(const std::string &) const override { + std::vector get_entity_capabilities(const std::string & /*entity_id*/) const override { return {}; } - ros2_medkit_gateway::LockAccessResult check_lock(const std::string &, const std::string &, - const std::string &) const override { + ros2_medkit_gateway::LockAccessResult check_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::string & /*collection*/) const override { return ros2_medkit_gateway::LockAccessResult{true, "", "", ""}; } tl::expected - acquire_lock(const std::string &, const std::string &, const std::vector &, int) override { + acquire_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::vector & /*scopes*/, int /*expiration_seconds*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } - tl::expected release_lock(const std::string &, const std::string &) override { + tl::expected release_lock(const std::string & /*entity_id*/, + const std::string & /*client_id*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } ros2_medkit_gateway::ResourceChangeNotifier * get_resource_change_notifier() override { @@ -144,6 +162,89 @@ class MockPluginContext : public RosPluginContext { rclcpp::Node * node_; }; +/// How a ParameterServiceStub answers get requests. +enum class GetAnswer { kNone, kValue, kNoValues }; + +/// A node with parameter services that answers only as told, on its own executor. Counts list and get requests. +class ParameterServiceStub { + public: + ParameterServiceStub(const std::string & name, bool answer_list, GetAnswer answer_get = GetAnswer::kNone) { + node_ = std::make_shared(name, rclcpp::NodeOptions().start_parameter_services(false)); + using rcl_interfaces::srv::GetParameters; + using rcl_interfaces::srv::ListParameters; + list_ = node_->create_service( + "~/list_parameters", [this, answer_list](const std::shared_ptr & header, + const std::shared_ptr &) { + ++list_requests_; + if (answer_list) { + ListParameters::Response response; + response.result.names = {"ros2_medkit.discovery.entity_id"}; + list_->send_response(*header, response); + } + }); + get_ = node_->create_service( + "~/get_parameters", [this, name, answer_get](const std::shared_ptr & header, + const std::shared_ptr &) { + ++get_requests_; + if (answer_get != GetAnswer::kNone) { + GetParameters::Response response; + if (answer_get == GetAnswer::kValue) { + response.values.push_back(rclcpp::ParameterValue(name).to_value_msg()); + } + get_->send_response(*header, response); + } + }); + types_ = silent("~/get_parameter_types"); + set_ = silent("~/set_parameters"); + atomically_ = silent("~/set_parameters_atomically"); + describe_ = silent("~/describe_parameters"); + executor_.add_node(node_); + spin_thread_ = std::thread([this]() { + executor_.spin(); + }); + } + + ~ParameterServiceStub() { + executor_.cancel(); + spin_thread_.join(); + executor_.remove_node(node_); + } + + ParameterServiceStub(const ParameterServiceStub &) = delete; + ParameterServiceStub & operator=(const ParameterServiceStub &) = delete; + ParameterServiceStub(ParameterServiceStub &&) = delete; + ParameterServiceStub & operator=(ParameterServiceStub &&) = delete; + + std::string fqn() const { + return node_->get_fully_qualified_name(); + } + int list_requests() const { + return list_requests_.load(); + } + int get_requests() const { + return get_requests_.load(); + } + + private: + template + std::shared_ptr silent(const std::string & name) { + return node_->create_service( + name, [](const std::shared_ptr &, const std::shared_ptr &) {}); + } + + rclcpp::Node::SharedPtr node_; + rclcpp::Service::SharedPtr list_; + rclcpp::Service::SharedPtr get_; + std::shared_ptr types_; + std::shared_ptr set_; + std::shared_ptr atomically_; + std::shared_ptr describe_; + rclcpp::executors::SingleThreadedExecutor executor_; + std::thread spin_thread_; + std::atomic list_requests_{0}; + std::atomic get_requests_{0}; +}; + // --- Test Fixture --- class ParamBeaconPluginTest : public ::testing::Test { @@ -467,3 +568,293 @@ TEST_F(ParamBeaconPluginTest, PollCycleAfterShutdownIsNoop) { std::this_thread::sleep_for(std::chrono::milliseconds(500)); EXPECT_EQ(plugin_->store().size(), store_size); } + +// A get request that gets no answer in time throws, like a list request. +TEST_F(ParamBeaconPluginTest, RealClientThrowsWhenGetParametersGetsNoAnswer) { + ParameterServiceStub stub("get_never_answers", true); + RealParameterClient client(node_, stub.fqn(), std::chrono::duration(0.2)); + ASSERT_TRUE(client.wait_for_service(std::chrono::seconds(10))); + const auto names = client.list_parameters({"ros2_medkit.discovery"}, 0).names; + ASSERT_EQ(names, (std::vector{"ros2_medkit.discovery.entity_id"})); + + const auto started = std::chrono::steady_clock::now(); + EXPECT_THROW(client.get_parameters(names), std::runtime_error); + EXPECT_GE(std::chrono::steady_clock::now() - started, std::chrono::milliseconds(200)); + EXPECT_EQ(stub.get_requests(), 1); +} + +// A request that gets no answer in time leaves nothing pending in its client. +TEST_F(ParamBeaconPluginTest, RealClientLeavesNoRequestPendingAfterTimeouts) { + ParameterServiceStub list_silent("pending_list_silent", false); + ParameterServiceStub get_silent("pending_get_silent", true); + RealParameterClient list_client(node_, list_silent.fqn(), std::chrono::duration(0.005)); + RealParameterClient get_client(node_, get_silent.fqn(), std::chrono::duration(0.005)); + ASSERT_TRUE(list_client.wait_for_service(std::chrono::seconds(10))); + ASSERT_TRUE(get_client.wait_for_service(std::chrono::seconds(10))); + + constexpr int kCalls = 50; + int timeouts = 0; + for (int i = 0; i < kCalls; ++i) { + try { + list_client.list_parameters({"ros2_medkit.discovery"}, 0); + } catch (const std::runtime_error &) { + ++timeouts; + } + try { + get_client.get_parameters({"ros2_medkit.discovery.entity_id"}); + } catch (const std::runtime_error &) { + ++timeouts; + } + } + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while ((list_silent.list_requests() < kCalls || get_silent.get_requests() < kCalls) && + std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_EQ(list_silent.list_requests(), kCalls); + ASSERT_EQ(get_silent.get_requests(), kCalls); + EXPECT_EQ(timeouts, 2 * kCalls); + EXPECT_EQ(list_client.prune_pending_requests(), 0U); + EXPECT_EQ(get_client.prune_pending_requests(), 0U); +} + +// A get answer without values is an answer: no exception, no parameters, nothing pending. +TEST_F(ParamBeaconPluginTest, RealClientReturnsNoParametersForAnAnswerWithoutValues) { + ParameterServiceStub stub("get_answers_no_values", true, GetAnswer::kNoValues); + RealParameterClient client(node_, stub.fqn(), std::chrono::duration(10.0)); + ASSERT_TRUE(client.wait_for_service(std::chrono::seconds(10))); + + std::vector parameters{rclcpp::Parameter("placeholder", 1)}; + EXPECT_NO_THROW(parameters = client.get_parameters({"ros2_medkit.discovery.entity_id"})); + EXPECT_TRUE(parameters.empty()); + EXPECT_EQ(stub.get_requests(), 1); + EXPECT_EQ(client.prune_pending_requests(), 0U); +} + +// A wait with no time left returns within 1 s, also when the service does not exist. +TEST_F(ParamBeaconPluginTest, RealClientWaitWithNoTimeLeftReturns) { + ::testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_EXIT( + { + // A wait that never returns ends the child with SIGALRM. + alarm(10); + RealParameterClient client(node_, "/param_beacon_absent_node", std::chrono::duration(1.0)); + const auto start = std::chrono::steady_clock::now(); + const bool found = client.wait_for_service(std::chrono::duration(0.0)); + const auto waited_ms = + std::chrono::duration_cast(std::chrono::steady_clock::now() - start).count(); + // libtsan intercepts _exit, so a race reported in the child sets its exit code. + if (found) { + _exit(1); + } + if (waited_ms >= 1000) { + std::fprintf(stderr, "the wait took %lld ms\n", static_cast(waited_ms)); + _exit(2); + } + _exit(0); + }, + ::testing::ExitedWithCode(0), ""); +} + +// In a graph read, a node whose get or list requests time out is backed off; one that answers is +// polled every cycle. +TEST_F(ParamBeaconPluginTest, RuntimeTargetsWhoseParameterRequestsTimeOutAreBackedOff) { + ParameterServiceStub healthy("beacon_healthy", true, GetAnswer::kValue); + ParameterServiceStub get_silent("beacon_get_silent", true); + ParameterServiceStub list_silent("beacon_list_silent", false); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 0.1; + config["poll_budget_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + // Twelve cycles: a node that times out on every poll is asked on cycles 1, 3, 6 and 11. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 12 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + const int cycles = healthy.list_requests(); + ASSERT_GE(cycles, 12) << "the plugin never polled the node that answers"; + EXPECT_GE(get_silent.get_requests(), 3); + EXPECT_LE(get_silent.get_requests(), 5) << "a node whose get requests time out was asked on " + << get_silent.get_requests() << " of " << cycles << " cycles"; + EXPECT_GE(list_silent.list_requests(), 3); + EXPECT_LE(list_silent.list_requests(), 5) << "a node whose list requests time out was asked on " + << list_silent.list_requests() << " of " << cycles << " cycles"; +} + +// NaN, -inf and values below the minimum clamp to the minimum, +inf to kMaxSeconds. Polling keeps +// its 0.1 s timeout. +TEST_F(ParamBeaconPluginTest, OutOfRangeDurationsAreClampedAndPollingKeepsItsTimeout) { + ParameterServiceStub get_silent("beacon_nonfinite_get_silent", true); + ParameterServiceStub list_silent("beacon_nonfinite_list_silent", false); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = std::numeric_limits::quiet_NaN(); + config["param_timeout_sec"] = -1.0; + config["poll_budget_sec"] = std::numeric_limits::infinity(); + config["beacon_ttl_sec"] = -std::numeric_limits::infinity(); + config["beacon_expiry_sec"] = std::numeric_limits::quiet_NaN(); + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (list_silent.list_requests() < 3 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + EXPECT_GE(list_silent.list_requests(), 3); + EXPECT_GE(get_silent.get_requests(), 1); +} + +// A get answer without values, as rclpy gives for a typed parameter with no value, causes no backoff +// and stores no hint. +TEST_F(ParamBeaconPluginTest, NodeAnsweringWithoutValuesIsPolledEveryCycle) { + ParameterServiceStub healthy("no_values_healthy", true, GetAnswer::kValue); + ParameterServiceStub no_values("no_values_answer", true, GetAnswer::kNoValues); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + config["poll_budget_sec"] = 20.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 12 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + const int cycles = healthy.list_requests(); + ASSERT_GE(cycles, 12) << "the plugin never polled the node that answers"; + EXPECT_GE(no_values.get_requests(), cycles - 1) + << "a node answering without values was asked on " << no_values.get_requests() << " of " << cycles << " cycles"; + EXPECT_TRUE(plugin_->store().get("no_values_healthy").has_value()); + EXPECT_EQ(plugin_->store().size(), 1U); +} + +// Timed-out polls leave nothing pending, and each node keeps one client across cycles. +TEST_F(ParamBeaconPluginTest, TimedOutPollsLeaveNothingPendingAndKeepOneClientPerNode) { + ParameterServiceStub get_silent("pending_poll_get_silent", true); + ParameterServiceStub list_silent("pending_poll_list_silent", false); + auto client_node = std::make_shared("_pending_poll_clients"); + std::mutex created_mutex; + std::map>> created; + + plugin_ = std::make_unique([&](const std::string & target) { + auto client = std::make_shared(client_node, target, std::chrono::duration(0.05)); + std::lock_guard lock(created_mutex); + created[target].push_back(client); + return client; + }); + nlohmann::json config; + config["poll_interval_sec"] = 0.05; + config["param_timeout_sec"] = 1.0; + config["poll_budget_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + // Four timeouts each: cycles 1, 3, 6 and 11 of the backoff. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while ((get_silent.get_requests() < 4 || list_silent.list_requests() < 4) && + std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + ASSERT_GE(get_silent.get_requests(), 4); + ASSERT_GE(list_silent.list_requests(), 4); + + std::lock_guard lock(created_mutex); + std::size_t pending = 0; + for (const auto & [target, clients] : created) { + for (const auto & client : clients) { + pending += client->prune_pending_requests(); + } + } + EXPECT_EQ(pending, 0U); + EXPECT_EQ(created[get_silent.fqn()].size(), 1U); + EXPECT_EQ(created[list_silent.fqn()].size(), 1U); +} + +// A graph read that finds no target drops every client. +TEST_F(ParamBeaconPluginTest, EmptyTargetListDropsEveryClient) { + auto stub = std::make_unique("dropped_when_gone", true, GetAnswer::kValue); + const std::string request_topic = "rq" + stub->fqn() + "/list_parametersRequest"; + auto clients = [&]() { + const auto infos = node_->get_publishers_info_by_topic(request_topic, true); + return std::count_if(infos.begin(), infos.end(), [](const rclcpp::TopicEndpointInfo & info) { + return info.node_name() == "_param_beacon_node"; + }); + }; + auto wait_clients = [&](std::ptrdiff_t expected) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (clients() != expected && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + return clients(); + }; + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + ASSERT_EQ(wait_clients(1), 1) << "the plugin never created a client for " << stub->fqn(); + stub.reset(); + EXPECT_EQ(wait_clients(0), 0) << "the client stayed after the graph listed no target"; +} + +// A graph read skips the gateway's helper nodes. +TEST_F(ParamBeaconPluginTest, GatewayHelperNodesAreNotPolled) { + ParameterServiceStub healthy("helper_skip_healthy", true, GetAnswer::kValue); + ParameterServiceStub fault_clients("test_param_beacon_node_fault_clients", true, GetAnswer::kValue); + ParameterServiceStub state_reader("test_param_beacon_node_lifecycle_state_reader", true, GetAnswer::kValue); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 5 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + ASSERT_GE(healthy.list_requests(), 5); + EXPECT_EQ(fault_clients.list_requests(), 0); + EXPECT_EQ(state_reader.list_requests(), 0); +} + +// The plugin's node joins the graph listener in set_context(). A first join after rclcpp shuts down +// fails half-way, and destroying the node then terminates the process. +TEST_F(ParamBeaconPluginTest, NodeDestroyedAfterShutdownDoesNotTerminate) { + ::testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_EXIT( + { + auto plugin = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 1000.0; + plugin->configure(config); + plugin->set_context(*mock_ctx_); + // Starts the graph listener, so rclcpp::shutdown() shuts it down. + node_->get_node_graph_interface()->get_graph_event(); + rclcpp::shutdown(); + try { + plugin->param_node()->get_node_graph_interface()->get_graph_event(); + } catch (const std::exception &) { + } + plugin.reset(); + std::_Exit(0); + }, + ::testing::ExitedWithCode(0), ""); +} diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp index 0df41cabb..47c735085 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp @@ -120,15 +120,17 @@ class MockPluginContext : public RosPluginContext { ros2_medkit_gateway::ConditionRegistry * get_condition_registry() override { return nullptr; } - ros2_medkit_gateway::LockAccessResult check_lock(const std::string &, const std::string &, - const std::string &) const override { + ros2_medkit_gateway::LockAccessResult check_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::string & /*collection*/) const override { return ros2_medkit_gateway::LockAccessResult{true, "", "", ""}; } tl::expected - acquire_lock(const std::string &, const std::string &, const std::vector &, int) override { + acquire_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::vector & /*scopes*/, int /*expiration_seconds*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } - tl::expected release_lock(const std::string &, const std::string &) override { + tl::expected release_lock(const std::string & /*entity_id*/, + const std::string & /*client_id*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } diff --git a/src/ros2_medkit_gateway/CMakeLists.txt b/src/ros2_medkit_gateway/CMakeLists.txt index 23b20ac4d..6a7c0b481 100644 --- a/src/ros2_medkit_gateway/CMakeLists.txt +++ b/src/ros2_medkit_gateway/CMakeLists.txt @@ -728,6 +728,13 @@ if(BUILD_TESTING) target_link_libraries(test_refresh_debounce gateway_core) endif() + # Node-list reader decisions (header-only), plus one case against a real rcl node + medkit_add_gtest(test_graph_node_list test/test_graph_node_list.cpp) + if(TARGET test_graph_node_list) + medkit_target_dependencies(test_graph_node_list rclcpp) + target_include_directories(test_graph_node_list PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include) + endif() + # SlotStore object pool (pure C++17, no ROS node) medkit_add_gtest(test_slot_store test/test_slot_store.cpp) if(TARGET test_slot_store) diff --git a/src/ros2_medkit_gateway/test/test_graph_node_list.cpp b/src/ros2_medkit_gateway/test/test_graph_node_list.cpp new file mode 100644 index 000000000..b552cc22c --- /dev/null +++ b/src/ros2_medkit_gateway/test/test_graph_node_list.cpp @@ -0,0 +1,294 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// The node-list reader's decisions. The test_graph_leftover_nodes*.test.py suites in +// ros2_medkit_integration_tests cover a real graph. + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" + +using ros2_medkit_gateway::ros2_common::graph_node_fqn; +using ros2_medkit_gateway::ros2_common::graph_node_has_endpoints; +using ros2_medkit_gateway::ros2_common::GraphNodeEntry; +using ros2_medkit_gateway::ros2_common::GraphNodeList; +using ros2_medkit_gateway::ros2_common::GraphNodeListReader; +using ros2_medkit_gateway::ros2_common::LeftoverNodeReporter; + +namespace { + +using NamePair = std::pair; +using Clock = GraphNodeListReader::Clock; +using std::chrono::seconds; + +/// Endpoint lookup over a fixed set of FQNs that resolve endpoints, counting every question asked. +struct EndpointTable { + std::set with_endpoints; + std::map asked; + + auto probe() { + return [this](const std::string & name, const std::string & ns) { + const auto fqn = graph_node_fqn(name, ns); + ++asked[fqn]; + return with_endpoints.count(fqn) > 0; + }; + } +}; + +const Clock::time_point kStart{}; + +GraphNodeEntry running(const std::string & name, const std::string & ns) { + return GraphNodeEntry{name, ns, "/"}; +} + +GraphNodeEntry no_enclave(const std::string & name, const std::string & ns) { + return GraphNodeEntry{name, ns, ""}; +} + +} // namespace + +TEST(GraphNodeList, FqnOfRootAndNestedNamespaces) { + EXPECT_EQ(graph_node_fqn("a", "/"), "/a"); + EXPECT_EQ(graph_node_fqn("a", ""), "/a"); + EXPECT_EQ(graph_node_fqn("a", "/ns/sub"), "/ns/sub/a"); +} + +TEST(GraphNodeListReader, EntryWithEnclaveIsListedWithoutAnEndpointQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({running("quiet", "/ns")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"quiet", "/ns"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); + EXPECT_EQ(reader.remembered(), 0u); +} + +TEST(GraphNodeListReader, EntryWithoutEnclaveOfANodeNeverSeenRunningIsListedWithoutAQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({no_enclave("far", "/router")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"far", "/router"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); + EXPECT_EQ(reader.remembered(), 0u); +} + +TEST(GraphNodeListReader, LeftoverOfANodeSeenRunningIsLeftOutUnlessItHasEndpoints) { + GraphNodeListReader reader; + EndpointTable table; + reader.filter({running("gone", "/ns"), running("bridged", "/ns")}, kStart, table.probe()); + + table.with_endpoints = {"/ns/bridged"}; + auto list = + reader.filter({no_enclave("gone", "/ns"), no_enclave("bridged", "/ns")}, kStart + seconds(1), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"bridged", "/ns"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/ns/gone"})); + EXPECT_EQ(table.asked, (std::map{{"/ns/bridged", 1}, {"/ns/gone", 1}})); +} + +TEST(GraphNodeListReader, EntryWithoutEnclaveNextToOneWithAnEnclaveIsDroppedWithoutAQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({no_enclave("node", "/"), running("node", "/")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"node", "/"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); +} + +TEST(GraphNodeListReader, EndpointsAreAskedOncePerNameAndGraphOrderIsKept) { + GraphNodeListReader reader; + EndpointTable table; + reader.filter({running("x", "/a"), running("x", "/b")}, kStart, table.probe()); + table.with_endpoints = {"/b/x"}; + auto list = reader.filter( + {no_enclave("x", "/a"), running("y", "/"), no_enclave("x", "/b"), no_enclave("x", "/a"), no_enclave("x", "/b")}, + kStart + seconds(1), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"y", "/"}, {"x", "/b"}, {"x", "/b"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/a/x"})); + EXPECT_EQ(table.asked, (std::map{{"/a/x", 1}, {"/b/x", 1}})); +} + +TEST(GraphNodeListReader, AbsenceIsCountedFromTheFirstReadThatFindsTheNameAbsent) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + // The next read comes long after the last one that listed the node; the absence starts here. + reader.filter({}, kStart + seconds(60), table.probe()); + reader.filter({}, kStart + seconds(69), table.probe()); + auto list = + reader.filter({no_enclave("n", "/")}, kStart + seconds(69) + std::chrono::milliseconds(500), table.probe()); + EXPECT_TRUE(list.nodes.empty()); + EXPECT_EQ(list.leftovers, (std::vector{"/n"})); +} + +TEST(GraphNodeListReader, NameAbsentLongerThanTheHoldIsForgottenAndItsLeftoverListed) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + reader.filter({}, kStart + seconds(1), table.probe()); + reader.filter({}, kStart + seconds(11), table.probe()); + EXPECT_EQ(reader.remembered(), 1u); + reader.filter({}, kStart + seconds(12), table.probe()); + EXPECT_EQ(reader.remembered(), 0u); + auto list = reader.filter({no_enclave("n", "/")}, kStart + seconds(13), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"n", "/"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); +} + +TEST(GraphNodeListReader, ListedLeftoverIsRememberedForAsLongAsItIsListed) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + for (int i = 1; i <= 100; ++i) { + auto list = reader.filter({no_enclave("n", "/")}, kStart + seconds(i), table.probe()); + ASSERT_EQ(list.leftovers, (std::vector{"/n"})) << "read " << i; + } +} + +TEST(GraphNodeListReader, NamesRunningNowDoNotCountAgainstTheCapacity) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + std::vector graph; + graph.reserve(9); + for (int i = 0; i < 8; ++i) { + graph.push_back(running("live_" + std::to_string(i), "/")); + } + graph.push_back(running("gone", "/")); + reader.filter(graph, kStart, table.probe()); + reader.filter(graph, kStart + seconds(1), table.probe()); + EXPECT_EQ(reader.remembered(), 0u); + + graph.back() = no_enclave("gone", "/"); + for (int i = 2; i <= 20; ++i) { + auto list = reader.filter(graph, kStart + seconds(i), table.probe()); + ASSERT_EQ(list.leftovers, (std::vector{"/gone"})) << "read " << i; + ASSERT_EQ(list.nodes.size(), 8u) << "read " << i; + } + EXPECT_EQ(reader.remembered(), 1u); +} + +TEST(GraphNodeListReader, ANameThatRunsAgainIsNoLongerDeparted) { + GraphNodeListReader reader(seconds(100), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + reader.filter({no_enclave("n", "/")}, kStart + seconds(1), table.probe()); + EXPECT_EQ(reader.remembered(), 1u); + auto list = reader.filter({no_enclave("n", "/"), running("n", "/")}, kStart + seconds(2), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"n", "/"}})); + EXPECT_EQ(reader.remembered(), 0u); + // It departs again: the leftover of the earlier run is left out once more. + list = reader.filter({no_enclave("n", "/")}, kStart + seconds(3), table.probe()); + EXPECT_TRUE(list.nodes.empty()); + EXPECT_EQ(list.leftovers, (std::vector{"/n"})); +} + +TEST(GraphNodeListReader, PastTheCapacityTheNameUnlistedLongestIsForgottenFirst) { + GraphNodeListReader reader(seconds(100), 3); + EndpointTable table; + reader.filter({running("listed", "/"), running("early", "/"), running("late", "/"), running("new", "/")}, kStart, + table.probe()); + // listed departs first and stays listed through a leftover; early and late depart with no entry left. + reader.filter({no_enclave("listed", "/"), running("early", "/"), running("late", "/"), running("new", "/")}, + kStart + seconds(1), table.probe()); + reader.filter({no_enclave("listed", "/"), running("late", "/"), running("new", "/")}, kStart + seconds(2), + table.probe()); + reader.filter({no_enclave("listed", "/"), running("new", "/")}, kStart + seconds(3), table.probe()); + EXPECT_EQ(reader.remembered(), 3u); + + // new departs past the capacity: early has had no entry for longest, so it is forgotten first, + // although listed departed before it. + auto list = reader.filter({no_enclave("listed", "/"), no_enclave("new", "/")}, kStart + seconds(4), table.probe()); + EXPECT_EQ(reader.remembered(), 3u); + EXPECT_EQ(list.leftovers, (std::vector{"/listed", "/new"})); + list = reader.filter( + {no_enclave("listed", "/"), no_enclave("early", "/"), no_enclave("late", "/"), no_enclave("new", "/")}, + kStart + seconds(5), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"early", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/listed", "/late", "/new"})); +} + +TEST(GraphNodeListReader, PastTheCapacityAmongListedNamesTheOneThatDepartedLongestAgoIsForgottenFirst) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + reader.filter({running("old", "/")}, kStart, table.probe()); + reader.filter({no_enclave("old", "/"), running("mid", "/")}, kStart + seconds(1), table.probe()); + reader.filter({no_enclave("old", "/"), no_enclave("mid", "/"), running("new", "/")}, kStart + seconds(2), + table.probe()); + EXPECT_EQ(reader.remembered(), 2u); + + auto list = reader.filter({no_enclave("old", "/"), no_enclave("mid", "/"), no_enclave("new", "/")}, + kStart + seconds(3), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"old", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/mid", "/new"})); +} + +TEST(GraphNodeListReader, PastTheCapacityANameThatDepartsWithNoEntryLeftIsForgottenBeforeListedLeftovers) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + reader.filter({running("a", "/"), running("b", "/"), running("c", "/")}, kStart, table.probe()); + reader.filter({no_enclave("a", "/"), no_enclave("b", "/"), running("c", "/")}, kStart + seconds(1), table.probe()); + // c departs and the graph lists no entry of it: of three departed names it is the one no entry lists. + auto list = reader.filter({no_enclave("a", "/"), no_enclave("b", "/")}, kStart + seconds(2), table.probe()); + EXPECT_EQ(reader.remembered(), 2u); + EXPECT_EQ(list.leftovers, (std::vector{"/a", "/b"})); + list = reader.filter({no_enclave("a", "/"), no_enclave("b", "/"), no_enclave("c", "/")}, kStart + seconds(3), + table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"c", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/a", "/b"})); +} + +TEST(LeftoverNodeReporter, ReportsOnTheReadWhereANodeStartsBeingLeftOut) { + LeftoverNodeReporter reporter(rclcpp::get_logger("test_graph_node_list")); + const GraphNodeList left_out{{}, {"/ns/gone"}}; + const GraphNodeList listed{{{"gone", "/ns"}}, {}}; + const GraphNodeList absent{}; + + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); + EXPECT_TRUE(reporter.report(left_out).empty()); + EXPECT_TRUE(reporter.report(left_out).empty()); + + EXPECT_TRUE(reporter.report(listed).empty()); + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); + + EXPECT_TRUE(reporter.report(absent).empty()); + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); +} + +// Against a real rcl graph: the one rclcpp error that means "the graph no longer lists this node" +// counts as no endpoints, and the error a shut-down context raises propagates. +TEST(GraphNodeHasEndpoints, NodeTheGraphDoesNotListHasNoneAndAShutDownContextThrows) { + auto context = std::make_shared(); + context->init(0, nullptr); + auto node = std::make_shared("_graph_node_list_probe", rclcpp::NodeOptions().context(context)); + const auto graph = node->get_node_graph_interface(); + + EXPECT_FALSE(graph_node_has_endpoints(*graph, "no_such_node", "/no_such_namespace")); + EXPECT_TRUE(graph_node_has_endpoints(*graph, node->get_name(), node->get_namespace())); + + context->shutdown("test"); + EXPECT_THROW(graph_node_has_endpoints(*graph, "no_such_node", "/no_such_namespace"), rclcpp::exceptions::RCLError); +} diff --git a/src/ros2_medkit_integration_tests/CMakeLists.txt b/src/ros2_medkit_integration_tests/CMakeLists.txt index dab635448..2345cbd00 100644 --- a/src/ros2_medkit_integration_tests/CMakeLists.txt +++ b/src/ros2_medkit_integration_tests/CMakeLists.txt @@ -162,6 +162,20 @@ install(TARGETS topics_test_plugin LIBRARY DESTINATION lib/${PROJECT_NAME}) if(BUILD_TESTING) find_package(launch_testing_ament_cmake REQUIRED) + # Graph fixtures for the test_graph_leftover_nodes suites, installed for tests in any package. + # ghost_node_injector writes rmw_dds_common's discovery message itself. + find_package(rcl REQUIRED) + find_package(rmw REQUIRED) + find_package(rmw_dds_common REQUIRED) + find_package(rosidl_typesupport_cpp REQUIRED) + add_executable(ghost_node_injector demo_nodes/ghost_node_injector.cpp) + target_include_directories(ghost_node_injector PRIVATE ${_demo_include_dir}) + medkit_target_dependencies(ghost_node_injector rclcpp rcl rmw rmw_dds_common rosidl_typesupport_cpp std_msgs) + add_executable(endpointless_node demo_nodes/endpointless_node.cpp) + target_include_directories(endpointless_node PRIVATE ${_demo_include_dir}) + medkit_target_dependencies(endpointless_node rclcpp rcl) + install(TARGETS ghost_node_injector endpointless_node DESTINATION lib/${PROJECT_NAME}) + # Lint the Python that makes up this package. Without this the whole package # sits outside `colcon test -L linter`, so the documented lint command passes # while none of these files are checked, and the pre-commit hooks are the @@ -348,6 +362,8 @@ if(BUILD_TESTING) test_graph_provider_sse 300 test_peer_recovery 300 test_triggers_restore_before_discovery 300 + # Seven cases share one gateway, and their scaled give-up bounds add up past 120s. + test_graph_leftover_nodes 300 # Three discovery budgets in setUpClass before the first case runs, and a # broken merge spends all of them. The glob default cuts that short and # reports a timeout with no test name, which is the one answer that says diff --git a/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp b/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp new file mode 100644 index 000000000..89fb1b599 --- /dev/null +++ b/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp @@ -0,0 +1,72 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +/** + * @file endpointless_node.cpp + * @brief A running node with no endpoints of its own + * + * Test fixture: a bare rcl node with /rosout off, because an rclcpp::Node always + * subscribes to /parameter_events. Set the name with `-r __node:=... -r __ns:=...`. + * Runs until SIGINT or SIGTERM. + */ + +#include +#include + +#include +#include +#include + +#include "ros2_medkit_integration_tests/crash_backtrace.hpp" + +int main(int argc, char ** argv) { + ros2_medkit_integration_tests::install_crash_backtrace(); + + // Blocked before any thread exists, so every thread inherits the mask and the + // signal waits for sigwait() below instead of rclcpp's handler. + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + if (pthread_sigmask(SIG_BLOCK, &mask, nullptr) != 0) { + return 1; + } + rclcpp::init(argc, argv, rclcpp::InitOptions(), rclcpp::SignalHandlerOptions::None); + + rcl_node_t node = rcl_get_zero_initialized_node(); + rcl_node_options_t options = rcl_node_get_default_options(); + options.enable_rosout = false; + auto context = rclcpp::contexts::get_global_default_context()->get_rcl_context(); + if (rcl_node_init(&node, "endpointless_node", "", context.get(), &options) != RCL_RET_OK) { + std::fprintf(stderr, "rcl_node_init failed: %s\n", rcl_get_error_string().str); + rclcpp::shutdown(); + return 1; + } + + int signum = 0; + while (sigwait(&mask, &signum) != 0) { + // Interrupted before a signal was taken: wait again. + } + + int exit_code = 0; + if (rcl_node_fini(&node) != RCL_RET_OK) { + std::fprintf(stderr, "rcl_node_fini failed: %s\n", rcl_get_error_string().str); + exit_code = 1; + } + if (rcl_node_options_fini(&options) != RCL_RET_OK) { + exit_code = 1; + } + rclcpp::shutdown(); + return exit_code; +} diff --git a/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp b/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp new file mode 100644 index 000000000..c230551a0 --- /dev/null +++ b/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp @@ -0,0 +1,577 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +/** + * @file ghost_node_injector.cpp + * @brief Puts nodes into the ROS graph that no running participant stands behind + * + * Test fixture. The DDS RMWs create a graph node entry for any participant GID a + * `ros_discovery_info` message names. + * + * --ghost a message for an unowned GID (starts 00 00) listing the node with + * no endpoints: the shape of a node behind a DDS router. + * --backed the same for a GID starting 00 01 that names a publisher this + * process owns: a node whose participant announces no enclave. + * --leftover leaves a leftover behind a real node: + * 1. creates the node in its own context and captures its participant's last + * discovery message, then prints `ghost_node_injector: leftover_ready + * participant_gid=`; + * 2. with --announce , on an `announce` stdin line, republishes it with + * more nodes `_` and prints `ghost_node_injector: + * leftover_announced count= acked=`. Send it only after the + * target graph lists the node, or the original message can replace it; + * 3. on a `leave` line, destroys the node and its context and waits until this + * process's graph drops it; + * 4. after --delay (default 0), or on a `publish` line after `leave`, + * republishes the message and prints `ghost_node_injector: + * leftover_published matched_subscriptions= acked=`. + * + * The participant GID is the node's publisher GUID prefix plus entity id 00 00 01 c1. + * Step 1 waits for a discovery message with that GID, so a wrong derivation fails there. + * --ghost and --backed print `ghost_node_injector: matched_subscriptions= entries= + * acked=`, and send one message per node in command-line order. Messages go + * through the rmw layer on a reliable, transient-local, keep-all writer, so late graphs + * receive them in that order and they outlive this process. rcl would rename the topic to + * `/ros_discovery_info`, which the RMWs do not read. + * + * Runs until SIGINT or SIGTERM. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_integration_tests/demo_node_main.hpp" + +namespace { + +using ParticipantEntitiesInfo = rmw_dds_common::msg::ParticipantEntitiesInfo; +using NodeEntitiesInfo = rmw_dds_common::msg::NodeEntitiesInfo; +using Gid = rmw_dds_common::msg::Gid; + +constexpr const char * kUsage = + "usage: ghost_node_injector [--ghost ]... [--backed ]... " + "[--leftover [--announce ] [--delay ]]"; + +struct InjectedNode { + std::string name; + std::string ns; + bool backed; +}; + +struct Arguments { + std::vector injected; + std::optional leftover; + size_t announce{0}; + double delay_sec{0.0}; +}; + +InjectedNode parse_fqn(const std::string & fqn, bool backed) { + const auto slash = fqn.rfind('/'); + if (fqn.empty() || fqn.front() != '/' || slash == std::string::npos || slash + 1 == fqn.size()) { + throw std::invalid_argument("not a fully qualified node name: '" + fqn + "'"); + } + std::string ns = slash == 0 ? "/" : fqn.substr(0, slash); + return InjectedNode{fqn.substr(slash + 1), std::move(ns), backed}; +} + +Arguments parse_arguments(const std::vector & args) { + Arguments parsed; + bool announce_given = false; + bool delay_given = false; + // args[0] is the program name. + for (size_t i = 1; i < args.size(); ++i) { + const auto & flag = args[i]; + if (i + 1 >= args.size()) { + throw std::invalid_argument(kUsage); + } + const auto & value = args[++i]; + if (flag == "--ghost" || flag == "--backed") { + parsed.injected.push_back(parse_fqn(value, flag == "--backed")); + } else if (flag == "--leftover" && !parsed.leftover) { + parsed.leftover = parse_fqn(value, false); + } else if (flag == "--announce") { + parsed.announce = std::stoul(value); + announce_given = true; + } else if (flag == "--delay") { + parsed.delay_sec = std::stod(value); + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite. + if (!(parsed.delay_sec >= 0.0 && parsed.delay_sec <= 3600.0)) { // NOLINT(readability-simplify-boolean-expr) + throw std::invalid_argument("--delay must be between 0 and 3600 seconds"); + } + delay_given = true; + } else { + throw std::invalid_argument(kUsage); + } + } + if ((parsed.injected.empty() && !parsed.leftover) || ((announce_given || delay_given) && !parsed.leftover)) { + throw std::invalid_argument(kUsage); + } + return parsed; +} + +std::string to_hex(const Gid & gid) { + static constexpr std::array kDigits{'0', '1', '2', '3', '4', '5', '6', '7', + '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}; + std::string hex; + for (const auto byte : gid.data) { + hex.push_back(kDigits[static_cast(byte >> 4U)]); + hex.push_back(kDigits[static_cast(byte & 0x0FU)]); + } + return hex; +} + +class GhostNodeInjector : public rclcpp::Node { + public: + explicit GhostNodeInjector(Arguments args) : Node("_ghost_node_injector"), args_(std::move(args)) { + backing_publisher_ = create_publisher("~/backing", rclcpp::QoS(1)); + + rmw_node_ = rcl_node_get_rmw_handle(get_node_base_interface()->get_rcl_node_handle()); + rmw_qos_profile_t qos = rmw_qos_profile_default; + qos.history = RMW_QOS_POLICY_HISTORY_KEEP_ALL; + qos.durability = RMW_QOS_POLICY_DURABILITY_TRANSIENT_LOCAL; + qos.reliability = RMW_QOS_POLICY_RELIABILITY_RELIABLE; + qos.avoid_ros_namespace_conventions = true; + const auto * type_support = + rosidl_typesupport_cpp::get_message_type_support_handle(); + rmw_publisher_options_t publisher_options = rmw_get_default_publisher_options(); + discovery_publisher_ = + rmw_create_publisher(rmw_node_, type_support, "ros_discovery_info", &qos, &publisher_options); + if (discovery_publisher_ == nullptr) { + throw std::runtime_error(std::string("rmw_create_publisher(ros_discovery_info) failed: ") + + rmw_get_error_string().str); + } + + if (args_.leftover) { + rmw_subscription_options_t subscription_options = rmw_get_default_subscription_options(); + discovery_subscription_ = + rmw_create_subscription(rmw_node_, type_support, "ros_discovery_info", &qos, &subscription_options); + if (discovery_subscription_ == nullptr) { + throw std::runtime_error(std::string("rmw_create_subscription(ros_discovery_info) failed: ") + + rmw_get_error_string().str); + } + start_leftover_node(); + if (fcntl(STDIN_FILENO, F_SETFL, fcntl(STDIN_FILENO, F_GETFL) | O_NONBLOCK) != 0) { + throw std::runtime_error("could not make stdin non-blocking"); + } + } + + timer_ = create_wall_timer(std::chrono::milliseconds(20), [this]() { + tick_injected(); + tick_leftover(); + }); + } + + ~GhostNodeInjector() override { + timer_.reset(); + stop_leftover_node(); + if (discovery_subscription_ != nullptr) { + if (rmw_destroy_subscription(rmw_node_, discovery_subscription_) != RMW_RET_OK) { + std::fprintf(stderr, "rmw_destroy_subscription(ros_discovery_info) failed: %s\n", rmw_get_error_string().str); + rmw_reset_error(); + } + discovery_subscription_ = nullptr; + } + if (discovery_publisher_ != nullptr) { + if (rmw_destroy_publisher(rmw_node_, discovery_publisher_) != RMW_RET_OK) { + std::fprintf(stderr, "rmw_destroy_publisher(ros_discovery_info) failed: %s\n", rmw_get_error_string().str); + rmw_reset_error(); + } + discovery_publisher_ = nullptr; + } + backing_publisher_.reset(); + } + + GhostNodeInjector(const GhostNodeInjector &) = delete; + GhostNodeInjector & operator=(const GhostNodeInjector &) = delete; + GhostNodeInjector(GhostNodeInjector &&) = delete; + GhostNodeInjector & operator=(GhostNodeInjector &&) = delete; + + private: + enum class LeftoverStep { kCapturing, kReady, kRemoving, kDelaying, kPublished, kFailed }; + + // ---- --ghost / --backed -------------------------------------------------------------- + + /// Publish once something is listening, then report once every matched reader has acknowledged. + void tick_injected() { + if (args_.injected.empty() || injected_reported_) { + return; + } + if (!injected_published_) { + size_t matched = 0; + if (rmw_publisher_count_matched_subscriptions(discovery_publisher_, &matched) != RMW_RET_OK || matched == 0 || + !resolve_backing_gid()) { + if (std::chrono::steady_clock::now() - started_ < kMatchTimeout) { + return; + } + report_injected(matched, false); + return; + } + publish_injected(); + injected_matched_ = matched; + injected_published_ = true; + injected_published_at_ = std::chrono::steady_clock::now(); + } + const bool acked = rmw_publisher_wait_for_all_acked(discovery_publisher_, rmw_time_t{0, 0}) == RMW_RET_OK; + if (acked || std::chrono::steady_clock::now() - injected_published_at_ >= kAckTimeout) { + report_injected(injected_matched_, acked); + } + } + + /// The backing publisher's graph GID. Not PublisherBase::get_gid(): Cyclone returns a local id there. + bool resolve_backing_gid() { + const auto gid = endpoint_gid(*this, backing_publisher_->get_topic_name()); + if (!gid) { + return false; + } + backing_gid_ = *gid; + return true; + } + + void publish_injected() { + std::random_device seed; + std::mt19937 generator(seed()); + std::uniform_int_distribution byte(0, 255); + + for (const auto & node : args_.injected) { + ParticipantEntitiesInfo message; + for (auto & b : message.gid.data) { + b = static_cast>(byte(generator)); + } + message.gid.data[0] = 0x00; + message.gid.data[1] = node.backed ? 0x01 : 0x00; + NodeEntitiesInfo info; + info.node_namespace = node.ns; + info.node_name = node.name; + if (node.backed) { + info.writer_gid_seq.push_back(backing_gid_); + } + message.node_entities_info_seq.push_back(std::move(info)); + publish(message); + } + } + + void report_injected(size_t matched, bool acked) { + std::printf("ghost_node_injector: matched_subscriptions=%zu entries=%zu acked=%s\n", matched, args_.injected.size(), + acked ? "true" : "false"); + std::fflush(stdout); + injected_reported_ = true; + } + + // ---- --leftover ---------------------------------------------------------------------- + + void start_leftover_node() { + rclcpp::InitOptions init_options; + init_options.auto_initialize_logging(false); + leftover_context_ = std::make_shared(); + leftover_context_->init(0, nullptr, init_options); + leftover_node_ = std::make_shared(args_.leftover->name, args_.leftover->ns, + rclcpp::NodeOptions().context(leftover_context_)); + leftover_marker_ = leftover_node_->create_publisher("~/leftover_marker", rclcpp::QoS(1)); + rclcpp::ExecutorOptions executor_options; + executor_options.context = leftover_context_; + leftover_executor_ = std::make_shared(executor_options); + leftover_executor_->add_node(leftover_node_); + leftover_spin_thread_ = std::thread([executor = leftover_executor_]() { + executor->spin(); + }); + } + + /// Destroy the node, which removes its participant, and shut its context down. + void stop_leftover_node() { + if (!leftover_context_) { + return; + } + leftover_executor_->cancel(); + if (leftover_spin_thread_.joinable()) { + leftover_spin_thread_.join(); + } + leftover_executor_->remove_node(leftover_node_); + leftover_executor_.reset(); + leftover_marker_.reset(); + leftover_node_.reset(); + leftover_context_->shutdown("leftover node leaves"); + leftover_context_.reset(); + } + + void tick_leftover() { + if (!args_.leftover) { + return; + } + const auto & target = *args_.leftover; + if (leftover_step_ != LeftoverStep::kCapturing) { + // Keep the keep-all reader empty; only the capture step reads what it takes. + drain_discovery_messages(); + } + switch (leftover_step_) { + case LeftoverStep::kCapturing: + capture_leftover_message(); + if (captured_) { + std::printf("ghost_node_injector: leftover_ready participant_gid=%s\n", to_hex(captured_->gid).c_str()); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kReady; + } else if (std::chrono::steady_clock::now() - started_ >= kMatchTimeout) { + fail("no discovery message for participant GID " + (derived_gid_ ? to_hex(*derived_gid_) : "(none)") + + " lists " + target.ns + "/" + target.name + " with its marker publisher"); + } + break; + case LeftoverStep::kReady: + read_stdin(); + if (!announced_ && args_.announce > 0 && stdin_buffer_.find("announce\n") != std::string::npos) { + announce(); + // No graph in this process sees its own participant's writes; reader acks stand in. + std::printf("ghost_node_injector: leftover_announced count=%zu acked=%s\n", args_.announce, + wait_for_acks() ? "true" : "false"); + std::fflush(stdout); + announced_ = true; + } + if (stdin_buffer_.find("leave\n") != std::string::npos) { + stop_leftover_node(); + left_at_ = std::chrono::steady_clock::now(); + leftover_step_ = LeftoverStep::kRemoving; + } + break; + case LeftoverStep::kRemoving: + if (!own_graph_lists(target)) { + removed_at_ = std::chrono::steady_clock::now(); + std::printf("ghost_node_injector: leftover_removed after_ms=%lld\n", + static_cast( + std::chrono::duration_cast(removed_at_ - left_at_).count())); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kDelaying; + } else if (std::chrono::steady_clock::now() - left_at_ >= kMatchTimeout) { + fail("this process's graph still lists the node after its context was shut down"); + } + break; + case LeftoverStep::kDelaying: + read_stdin(); + if (std::chrono::steady_clock::now() - removed_at_ >= std::chrono::duration(args_.delay_sec) || + stdin_buffer_.find("publish\n", stdin_buffer_.find("leave\n")) != std::string::npos) { + size_t matched = 0; + if (rmw_publisher_count_matched_subscriptions(discovery_publisher_, &matched) != RMW_RET_OK) { + matched = 0; + } + publish(stale_message_); + const bool acked = wait_for_acks(); + std::printf("ghost_node_injector: leftover_published matched_subscriptions=%zu acked=%s\n", matched, + acked ? "true" : "false"); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kPublished; + } + break; + case LeftoverStep::kPublished: + case LeftoverStep::kFailed: + break; + } + } + + /// Take the discovery messages received so far; keep the leftover participant's one that lists + /// the marker publisher, the node's last endpoint. + void capture_leftover_message() { + if (!derived_gid_) { + marker_gid_ = endpoint_gid(*leftover_node_, leftover_marker_->get_topic_name()); + if (!marker_gid_) { + return; + } + Gid participant_gid; + std::fill(participant_gid.data.begin(), participant_gid.data.end(), 0); + std::copy_n(marker_gid_->data.begin(), kGuidPrefixSize, participant_gid.data.begin()); + std::copy(kParticipantEntityId.begin(), kParticipantEntityId.end(), + participant_gid.data.begin() + kGuidPrefixSize); + derived_gid_ = participant_gid; + } + const auto & target = *args_.leftover; + while (true) { + ParticipantEntitiesInfo message; + bool taken = false; + if (rmw_take(discovery_subscription_, &message, &taken, nullptr) != RMW_RET_OK) { + rmw_reset_error(); + return; + } + if (!taken) { + return; + } + if (message.gid != *derived_gid_) { + continue; + } + const auto & nodes = message.node_entities_info_seq; + const bool lists_target = std::any_of(nodes.begin(), nodes.end(), [this, &target](const NodeEntitiesInfo & info) { + const auto & writers = info.writer_gid_seq; + return info.node_name == target.name && info.node_namespace == target.ns && + std::find(writers.begin(), writers.end(), *marker_gid_) != writers.end(); + }); + if (lists_target && !captured_) { + captured_ = message; + stale_message_ = message; + } + } + } + + void drain_discovery_messages() { + ParticipantEntitiesInfo message; + bool taken = true; + while (taken) { + if (rmw_take(discovery_subscription_, &message, &taken, nullptr) != RMW_RET_OK) { + rmw_reset_error(); + return; + } + } + } + + void announce() { + stale_message_ = *captured_; + for (size_t i = 0; i < args_.announce; ++i) { + NodeEntitiesInfo info; + info.node_namespace = args_.leftover->ns; + info.node_name = announced_name(i); + stale_message_.node_entities_info_seq.push_back(std::move(info)); + } + publish(stale_message_); + } + + std::string announced_name(size_t index) const { + auto digits = std::to_string(index); + if (digits.size() < kAnnouncedIndexWidth) { + digits.insert(0, kAnnouncedIndexWidth - digits.size(), '0'); + } + return args_.leftover->name + "_" + digits; + } + + /// Whether this process's graph lists the node with an enclave. + bool own_graph_lists(const InjectedNode & node) { + for (const auto & [name, ns, enclave] : get_node_graph_interface()->get_node_names_with_enclaves()) { + if (name == node.name && ns == node.ns && !enclave.empty()) { + return true; + } + } + return false; + } + + void read_stdin() { + std::array buffer{}; + while (true) { + const auto count = ::read(STDIN_FILENO, buffer.data(), buffer.size()); + if (count <= 0) { + return; + } + stdin_buffer_.append(buffer.data(), static_cast(count)); + } + } + + /// Whether every matched reader acknowledged what was published, within kAckTimeout. + bool wait_for_acks() { + return rmw_publisher_wait_for_all_acked(discovery_publisher_, + rmw_time_t{static_cast(kAckTimeout.count()), 0}) == RMW_RET_OK; + } + + void fail(const std::string & reason) { + std::printf("ghost_node_injector: leftover_failed %s\n", reason.c_str()); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kFailed; + } + + // ---- shared -------------------------------------------------------------------------- + + /// The graph GID of the publisher `node` owns on `topic`, as `node`'s own graph keys it. + static std::optional endpoint_gid(rclcpp::Node & node, const std::string & topic) { + for (const auto & info : node.get_publishers_info_by_topic(topic)) { + if (info.node_name() == node.get_name() && info.node_namespace() == node.get_namespace()) { + Gid gid; + std::fill(gid.data.begin(), gid.data.end(), 0); + const auto & endpoint = info.endpoint_gid(); + std::copy_n(endpoint.begin(), std::min(endpoint.size(), gid.data.size()), gid.data.begin()); + return gid; + } + } + return std::nullopt; + } + + void publish(const ParticipantEntitiesInfo & message) { + if (rmw_publish(discovery_publisher_, &message, nullptr) != RMW_RET_OK) { + RCLCPP_ERROR(get_logger(), "rmw_publish failed: %s", rmw_get_error_string().str); + rmw_reset_error(); + } + } + + static constexpr std::chrono::seconds kMatchTimeout{30}; + static constexpr std::chrono::seconds kAckTimeout{10}; + static constexpr size_t kGuidPrefixSize = 12; + static constexpr size_t kAnnouncedIndexWidth = 5; + static constexpr std::array kParticipantEntityId{0x00, 0x00, 0x01, 0xc1}; + + Arguments args_; + rclcpp::Publisher::SharedPtr backing_publisher_; + Gid backing_gid_; + rmw_node_t * rmw_node_{nullptr}; + rmw_publisher_t * discovery_publisher_{nullptr}; + rmw_subscription_t * discovery_subscription_{nullptr}; + rclcpp::TimerBase::SharedPtr timer_; + const std::chrono::steady_clock::time_point started_{std::chrono::steady_clock::now()}; + + std::chrono::steady_clock::time_point injected_published_at_; + size_t injected_matched_{0}; + bool injected_published_{false}; + bool injected_reported_{false}; + + rclcpp::Context::SharedPtr leftover_context_; + rclcpp::Node::SharedPtr leftover_node_; + rclcpp::Publisher::SharedPtr leftover_marker_; + std::shared_ptr leftover_executor_; + std::thread leftover_spin_thread_; + LeftoverStep leftover_step_{LeftoverStep::kCapturing}; + std::optional marker_gid_; + std::optional derived_gid_; + std::optional captured_; + bool announced_{false}; + ParticipantEntitiesInfo stale_message_; + std::string stdin_buffer_; + std::chrono::steady_clock::time_point left_at_; + std::chrono::steady_clock::time_point removed_at_; +}; + +} // namespace + +int main(int argc, char ** argv) { + Arguments args; + try { + args = parse_arguments(rclcpp::remove_ros_arguments(argc, argv)); + } catch (const std::exception & e) { + std::fprintf(stderr, "%s\n", e.what()); + return 2; + } + return ros2_medkit_integration_tests::run_demo_node(argc, argv, [&args]() -> std::shared_ptr { + return std::make_shared(std::move(args)); + }); +} diff --git a/src/ros2_medkit_integration_tests/package.xml b/src/ros2_medkit_integration_tests/package.xml index 3ae4e8ae0..d2edc11b4 100644 --- a/src/ros2_medkit_integration_tests/package.xml +++ b/src/ros2_medkit_integration_tests/package.xml @@ -33,6 +33,10 @@ launch_testing launch_ros ament_index_python + rcl + rmw + rmw_dds_common + rosidl_typesupport_cpp python3-requests python3-jsonschema ament_cmake_flake8 diff --git a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py new file mode 100644 index 000000000..6a0e0deb7 --- /dev/null +++ b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Drive ``ghost_node_injector`` (demo_nodes/ghost_node_injector.cpp) and read a ROS graph. + +``--leftover`` leaves a real node's leftover in every graph on the domain; ``--ghost`` and +``--backed`` build the same empty-enclave shape for a node no graph saw running. +""" + +import os +import queue +import re +import signal +import subprocess +import threading +import time + +from ament_index_python.packages import get_package_prefix + +PACKAGE = 'ros2_medkit_integration_tests' + + +def fixture_path(name): + """Absolute path of an executable this package installs next to its demo nodes.""" + path = os.path.join(get_package_prefix(PACKAGE), 'lib', PACKAGE, name) + if not os.path.isfile(path): + raise FileNotFoundError(f'test fixture not installed: {path}') + return path + + +def split_fqn(fqn): + """(name, namespace) of a fully qualified node name, as a ROS graph reports them.""" + namespace, _, name = fqn.rpartition('/') + return name, namespace or '/' + + +def observed_enclaves(node, fqn): + """Enclave of every entry `node`'s ROS graph lists for `fqn`, in graph order.""" + name, namespace = split_fqn(fqn) + entries = node.get_node_names_and_namespaces_with_enclaves() + return [ + enclave + for node_name, node_namespace, enclave in entries + if node_name == name and node_namespace == namespace + ] + + +def wait_observed(node, fqn, predicate, timeout): + """Poll `node`'s graph until `predicate(enclaves of fqn)` holds; return whether it did.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(observed_enclaves(node, fqn)): + return True + time.sleep(0.05) + return False + + +def stop_process(proc, timeout): + """SIGTERM `proc` and wait for it, killing it if it outlives `timeout`.""" + if proc is None or proc.poll() is not None: + return + proc.send_signal(signal.SIGTERM) + try: + proc.wait(timeout=timeout) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=10) + + +class _FixtureProcess: + """One ghost_node_injector process, its output collected line by line.""" + + def __init__(self, args, env=None): + self.proc = subprocess.Popen( + [fixture_path('ghost_node_injector')] + args, env=env, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + self._lines = queue.Queue() + self._output = [] + threading.Thread(target=self._read, daemon=True).start() + + def _read(self): + for line in self.proc.stdout: + self._output.append(line) + self._lines.put(line) + + def wait_line(self, pattern, timeout): + """Return the first match of `pattern` on a line printed from now on, or None.""" + regex = re.compile(pattern) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + line = self._lines.get(timeout=0.1) + except queue.Empty: + if self.proc.poll() is not None and self._lines.empty(): + return None + continue + match = regex.search(line) + if match: + return match + return None + + def send(self, command): + self.proc.stdin.write(command + '\n') + self.proc.stdin.flush() + + def output(self): + return ''.join(self._output) + + def stop(self, timeout=30.0): + stop_process(self.proc, timeout) + + +class GhostInjection(_FixtureProcess): + """``--ghost`` and ``--backed`` nodes for GIDs no participant owns. + + The backed nodes go out before the ghosts, so a ROS graph that lists a ghost of this + injection also lists every backed node of it. + """ + + def __init__(self, ghosts=(), backed=(), env=None): + args = [] + for fqn in backed: + args += ['--backed', fqn] + for fqn in ghosts: + args += ['--ghost', fqn] + super().__init__(args, env) + self.entries = len(ghosts) + len(backed) + + def wait_status(self, timeout): + """(matched_subscriptions, entries, acked) once the messages went out, or None.""" + match = self.wait_line( + r'ghost_node_injector: matched_subscriptions=(\d+) entries=(\d+) acked=(true|false)', + timeout) + if match is None: + return None + return int(match.group(1)), int(match.group(2)), match.group(3) == 'true' + + +class LeftoverNode(_FixtureProcess): + """A real node that, on `leave()`, leaves a leftover of itself in every graph. + + `announce_nodes()` adds `announce` nodes ``_`` to its participant; they are + left over with it. + """ + + def __init__(self, fqn, delay_sec, announce=0, env=None): + args = ['--leftover', fqn, '--delay', f'{delay_sec:.3f}'] + if announce: + args += ['--announce', str(announce)] + super().__init__(args, env) + self.fqn = fqn + self.announce = announce + + def announced_fqns(self): + return [f'{self.fqn}_{index:05d}' for index in range(self.announce)] + + def wait_ready(self, timeout): + """Whether the node runs and its participant's discovery message was captured.""" + match = self.wait_line(r'ghost_node_injector: leftover_ready participant_gid=', timeout) + return match is not None + + def announce_nodes(self, timeout): + """Publish the announced nodes; whether every matched reader acknowledged them.""" + self.send('announce') + match = self.wait_line( + r'ghost_node_injector: leftover_announced count=\d+ acked=(true|false)', timeout) + return match is not None and match.group(1) == 'true' + + def leave(self): + """Remove the node and its participant; the late sample follows after the delay.""" + self.send('leave') + + def publish(self): + """Send the late sample now, after `leave()`, rather than when the delay runs out.""" + self.send('publish') diff --git a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py index f36a329de..198028f69 100644 --- a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py +++ b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py @@ -101,7 +101,7 @@ # --------------------------------------------------------------------------- def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', - extra_params=None, coverage=True, extra_env=None, + extra_params=None, parameter_files=(), coverage=True, extra_env=None, respawn=False, respawn_delay=1.0): """Create a ``gateway_node`` launch action with standard config. @@ -114,6 +114,10 @@ def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', so their names do not collide (e.g. ``gateway_with_scripts``). extra_params : dict or None Additional ROS parameters merged into the node config. + parameter_files : sequence of str + Parameter files passed after those parameters, for values a Python dict cannot + spell the way a user's file does (``.nan``, ``.inf``). A key set in both takes the + file's value. coverage : bool If True, set GCOV_PREFIX env vars for code coverage collection. extra_env : dict or None @@ -152,7 +156,7 @@ def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', executable='gateway_node', name=name, output='screen', - parameters=[params], + parameters=[params, *parameter_files], additional_env=env, respawn=respawn, respawn_delay=respawn_delay, diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py new file mode 100644 index 000000000..8bc5104fa --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py @@ -0,0 +1,561 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Discovery hides the leftover of a node it saw running, and lists everything else. + +``ghost_node_injector --leftover`` leaves a real node's leftover (empty enclave, no +endpoints) in the graph. Each case first checks that the node ran and that this process's +graph then lists only the leftover. ``--ghost`` and ``--backed`` build the same shape for +names the gateway never saw, which stay listed. + +All cases share one gateway and use names of their own. The last case rechecks the earlier +leftovers after the hold, so it must run last. +""" + +import os +import signal +import subprocess +import time +import unittest + +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + GhostInjection, + LeftoverNode, + observed_enclaves, + split_fqn, + stop_process, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() + +# Backstop refresh, so the observation windows below count refreshes that ran. +REFRESH_INTERVAL_MS = 500 +REFRESH_DEBOUNCE_MS = 500 +# How long a claim is watched: three refreshes. A window, not a give-up bound, so not scaled. +OBSERVE_SEC = 3 * REFRESH_INTERVAL_MS / 1000.0 +# Give-up bounds for the fixture (30 s of its own), DDS discovery and process start or exit. +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +# From this process's graph listing a node to GET /apps listing it: the debounce, the 100 ms +# graph poll, and up to 4 s by which the gateway's discovery can trail on a loaded host. +LISTED_AFTER_OBSERVED_SEC = ((REFRESH_DEBOUNCE_MS + 100) / 1000.0 + 4.0) * TIME_SCALE +# GraphNodeListReader::kDefaultHold. +LEFTOVER_HOLD_SEC = 10.0 +# Delay of the late sample for leftovers that arrive while the gateway remembers the node. +LEFTOVER_DELAY_SEC = 1.0 +# Inside the hold but six refreshes after the removal: catches forgetting after a few refreshes. +INSIDE_HOLD_DELAY_SEC = 3.0 +# From GET /apps first not listing a departed node until a refresh has run past the hold. +PAST_THE_HOLD_SEC = LEFTOVER_HOLD_SEC + LISTED_AFTER_OBSERVED_SEC +# A --delay longer than any run of this file: the late sample goes out on `publish`. +PUBLISH_ON_COMMAND_DELAY_SEC = 3600.0 +# How many leftovers the many-leftovers case leaves behind next to the live nodes. +MANY_LEFTOVERS = 50 + +# Departs in setUpClass; the last case sends its late sample after the hold. +AFTER_HOLD_FQN = '/leftover_hold_ns/leftover_after' + + +def generate_test_description(): + gateway_node = create_gateway_node( + extra_params={ + 'refresh_interval_ms': REFRESH_INTERVAL_MS, + 'discovery.refresh_debounce_ms': REFRESH_DEBOUNCE_MS, + }, + ) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +def _app_id(fqn): + return split_fqn(fqn)[0] + + +def _app_ids(): + response = requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=10) + response.raise_for_status() + return [item.get('id') for item in response.json().get('items', [])] + + +def _poll_app_listed(app_id, timeout, listed=True): + """Time at which GET /apps first matched `listed` for `app_id`, or None.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if (app_id in _app_ids()) == listed: + return time.monotonic() + except requests.exceptions.RequestException: + pass + time.sleep(0.1) + return None + + +class TestGraphLeftoverNodes(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_observer') + # (fqn, App id, when GET /apps first stopped listing it) per hidden leftover. + cls.hidden_leftovers = [] + + # Departs now with no entry left; its late sample waits for the last case. + cls.after_hold = LeftoverNode( + AFTER_HOLD_FQN, PUBLISH_ON_COMMAND_DELAY_SEC, env=_fixture_env()) + if not cls.after_hold.wait_ready(INJECTION_TIMEOUT_SEC): + raise AssertionError( + f'ghost_node_injector never got {AFTER_HOLD_FQN} running:\n' + f'{cls.after_hold.output()}') + if _poll_app_listed(_app_id(AFTER_HOLD_FQN), APPEAR_TIMEOUT_SEC) is None: + raise AssertionError(f'{AFTER_HOLD_FQN} runs but never appeared in GET /apps') + cls.after_hold.leave() + cls.after_hold_departed = _poll_app_listed( + _app_id(AFTER_HOLD_FQN), LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC, + listed=False) + if cls.after_hold_departed is None: + raise AssertionError(f'{AFTER_HOLD_FQN} left, but GET /apps kept listing it') + + @classmethod + def tearDownClass(cls): + cls.after_hold.stop(PROCESS_EXIT_TIMEOUT_SEC) + cls.observer.destroy_node() + rclpy.shutdown() + + def setUp(self): + self._processes = [] + self._fixtures = [] + + def tearDown(self): + for fixture in self._fixtures: + fixture.stop(PROCESS_EXIT_TIMEOUT_SEC) + for proc in self._processes: + stop_process(proc, PROCESS_EXIT_TIMEOUT_SEC) + + # ---- fixtures ----------------------------------------------------------------------- + + def _start_node(self, fqn, executable='demo_rpm_sensor'): + name, namespace = split_fqn(fqn) + proc = subprocess.Popen( + [fixture_path(executable), '--ros-args', '-r', f'__ns:={namespace}', + '-r', f'__node:={name}'], + env=_fixture_env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + self._processes.append(proc) + return proc + + def _start_leftover(self, fqn, delay_sec=LEFTOVER_DELAY_SEC, announce=0): + """Start a node that can leave a leftover, and wait until GET /apps lists it.""" + leftover = LeftoverNode(fqn, delay_sec, announce=announce, env=_fixture_env()) + self._fixtures.append(leftover) + self.assertTrue( + leftover.wait_ready(INJECTION_TIMEOUT_SEC), + f'ghost_node_injector never got {fqn} running:\n{leftover.output()}') + self.assertTrue( + self._wait_listed(_app_id(fqn), APPEAR_TIMEOUT_SEC), + f'{fqn} runs but never appeared in GET /apps, so the gateway never saw it running') + return leftover + + def _wait_left_over(self, fqn, timeout): + """Assert this process's graph lists `fqn` only as a leftover (empty enclave).""" + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], timeout), + f'after its participant left, the test process graph lists {fqn} with enclaves ' + f'{observed_enclaves(self.observer, fqn)} rather than once with an empty one, so ' + 'no leftover was reproduced') + + def _hidden_since(self, fqn, app_id): + """Wait until GET /apps does not list `app_id`; record the leftover for the last case.""" + hidden_at = _poll_app_listed(app_id, LISTED_AFTER_OBSERVED_SEC, listed=False) + self.assertIsNotNone(hidden_at, f'the leftover of {fqn} is listed as {app_id}') + type(self).hidden_leftovers.append((fqn, app_id, hidden_at)) + + # ---- observations ------------------------------------------------------------------- + + @staticmethod + def _ids(collection): + response = requests.get(f'{DEFAULT_BASE_URL}/{collection}', timeout=10) + response.raise_for_status() + return [item.get('id') for item in response.json().get('items', [])] + + @staticmethod + def _wait_listed(app_id, timeout, listed=True): + return _poll_app_listed(app_id, timeout, listed) is not None + + @staticmethod + def _warnings_for(proc_output, gateway_node, fqn): + text = ''.join( + output.text.decode(errors='replace') for output in proc_output[gateway_node]) + return text.count(f"Node '{fqn}' is not exposed") + + def _wait_warnings(self, proc_output, gateway_node, fqn, expected, + timeout=LISTED_AFTER_OBSERVED_SEC): + deadline = time.monotonic() + timeout + count = self._warnings_for(proc_output, gateway_node, fqn) + while count < expected and time.monotonic() < deadline: + time.sleep(0.2) + count = self._warnings_for(proc_output, gateway_node, fqn) + return count + + @staticmethod + def _observe(check): + """Run `check` every 250 ms for OBSERVE_SEC; it fails the test on the first bad sample.""" + deadline = time.monotonic() + OBSERVE_SEC + while time.monotonic() < deadline: + check() + time.sleep(0.25) + + # ---- cases -------------------------------------------------------------------------- + + def test_01_leftover_of_a_listed_node_inside_the_hold_is_hidden( + self, proc_output, gateway_node): + fqn = '/leftover_solo_ns/leftover_solo' + leftover = self._start_leftover(fqn, delay_sec=INSIDE_HOLD_DELAY_SEC) + self.assertIn( + 'leftover_solo_ns', self._ids('functions'), + f'no Function was derived from the namespace of the running {fqn}, so its absence ' + 'below would show nothing') + + leftover.leave() + self.assertTrue( + self._wait_listed('leftover_solo', LISTED_AFTER_OBSERVED_SEC, listed=False), + f'{fqn} left, but GET /apps kept listing it, so the gateway never read the graph ' + 'without it') + self._wait_left_over(fqn, INSIDE_HOLD_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self._hidden_since(fqn, 'leftover_solo') + + def check(): + app_ids = self._ids('apps') + self.assertNotIn( + 'leftover_solo', app_ids, + f'the late sample for {fqn} arrived {INSIDE_HOLD_DELAY_SEC:.1f} s after its ' + f'participant left, inside the {LEFTOVER_HOLD_SEC:.0f} s hold, and is listed') + detail = requests.get(f'{DEFAULT_BASE_URL}/apps/leftover_solo', timeout=10) + self.assertEqual( + detail.status_code, 404, + f'GET /apps/leftover_solo answered {detail.status_code} for a leftover') + self.assertNotIn( + 'leftover_solo_ns', self._ids('functions'), + f'a Function is still derived from the namespace only the leftover of {fqn} ' + 'occupies') + + self._observe(check) + self.assertEqual( + observed_enclaves(self.observer, fqn), [''], + f'the leftover of {fqn} left the test process graph during the observation, so the ' + 'gateway was not holding it out') + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway must warn exactly once about the leftover of {fqn}') + + def test_02_nodes_the_gateway_never_saw_leave_are_listed(self, proc_output, gateway_node): + far = '/ghost_far_ns/ghost_far' + twin = '/ghost_twin_ns/ghost_twin' + quiet = '/quiet_ns/quiet' + # A never-seen node without an enclave; the same next to a backed twin whose GID sorts + # after the ghost's; and a running node with no endpoints. + injection = GhostInjection(ghosts=[far, twin], backed=[twin], env=_fixture_env()) + self._fixtures.append(injection) + self._start_node(quiet, executable='endpointless_node') + self.assertIsNotNone( + injection.wait_status(INJECTION_TIMEOUT_SEC), + f'ghost_node_injector never published:\n{injection.output()}') + for fqn, enclaves in ((far, ['']), (twin, ['', '']), (quiet, ['/'])): + self.assertTrue( + wait_observed(self.observer, fqn, lambda found, want=enclaves: found == want, + INJECTION_TIMEOUT_SEC), + f'the test process graph lists {fqn} with {observed_enclaves(self.observer, fqn)}' + f', not {enclaves}') + name, namespace = split_fqn(quiet) + endpoints = ( + self.observer.get_publisher_names_and_types_by_node(name, namespace, True) + + self.observer.get_subscriber_names_and_types_by_node(name, namespace, True)) + self.assertEqual( + endpoints, [], + f'the fixture {quiet} is supposed to have no endpoints of its own, so this case ' + f'would not show that endpoints are not the rule: {endpoints}') + ids = {'ghost_far', 'ghost_twin', 'quiet'} + deadline = time.monotonic() + LISTED_AFTER_OBSERVED_SEC + while ids - set(self._ids('apps')) and time.monotonic() < deadline: + time.sleep(0.1) + self.assertEqual( + ids - set(self._ids('apps')), set(), + 'these nodes are not leftovers of nodes the gateway saw running, and never appeared ' + 'in GET /apps') + + def check(): + self.assertEqual(ids - set(self._ids('apps')), set(), + 'a node dropped out of GET /apps') + + self._observe(check) + for fqn in (far, twin, quiet): + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 0) + + def test_03_leftover_then_the_node_runs_again_is_listed_once( + self, proc_output, gateway_node): + fqn = '/leftover_again_ns/leftover_again' + leftover = self._start_leftover(fqn) + leftover.leave() + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertTrue( + self._wait_listed('leftover_again', LISTED_AFTER_OBSERVED_SEC, listed=False), + f'the leftover of {fqn} is listed') + self.assertEqual(self._wait_warnings(proc_output, gateway_node, fqn, 1), 1) + + self._start_node(fqn) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: sorted(enclaves) == ['', '/'], + APPEAR_TIMEOUT_SEC), + f'the test process graph does not list {fqn} running next to its leftover: ' + f'{observed_enclaves(self.observer, fqn)}') + self.assertTrue( + self._wait_listed('leftover_again', LISTED_AFTER_OBSERVED_SEC), + f'{fqn} runs again but never appeared in GET /apps') + + def check(): + app_ids = self._ids('apps') + self.assertEqual( + app_ids.count('leftover_again'), 1, + f'{fqn} runs next to its leftover and must be listed exactly once: {app_ids}') + + self._observe(check) + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 1) + + def test_04_leftover_does_not_rename_a_live_node_sharing_its_bare_name( + self, proc_output, gateway_node): + live = '/leftover_collide_b/leftover_collider' + fqn = '/leftover_collide_a/leftover_collider' + self._start_node(live) + self.assertTrue(self._wait_listed('leftover_collider', APPEAR_TIMEOUT_SEC), + f'the live node {live} never appeared in GET /apps') + leftover = LeftoverNode(fqn, LEFTOVER_DELAY_SEC, env=_fixture_env()) + self._fixtures.append(leftover) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertTrue( + self._wait_listed('leftover_collide_a_leftover_collider', APPEAR_TIMEOUT_SEC), + 'while both nodes run, the gateway never gave them namespace-prefixed ids, so the ' + 'collision rule this case depends on did not engage') + + leftover.leave() + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertTrue( + self._wait_listed('leftover_collider', LISTED_AFTER_OBSERVED_SEC), + f'{live} never got its un-prefixed id back after the other node left') + # Only the gateway's own read logs this, so its graph holds the leftover too. + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway never warned about the leftover of {fqn}, so its graph may never have ' + 'listed it and the ids below would show nothing') + hidden_at = time.monotonic() + + def check(): + app_ids = self._ids('apps') + self.assertIn( + 'leftover_collider', app_ids, + f'{live} lost its un-prefixed App id to the leftover of {fqn}: {app_ids}') + self.assertFalse( + [app_id for app_id in app_ids if app_id.endswith('_leftover_collider')], + f'a namespace-prefixed id was derived for a collision with a leftover: {app_ids}') + + self._observe(check) + # Once the live node is gone, the leftover would take the un-prefixed id. + type(self).hidden_leftovers.append((fqn, 'leftover_collider', hidden_at)) + + def test_05_node_seen_running_that_stays_behind_an_endpoint_is_listed( + self, proc_output, gateway_node): + fqn = '/bridged_ns/bridged' + marker = '/bridged_ns/bridged_marker' + node = self._start_node(fqn) + self.assertTrue(self._wait_listed('bridged', APPEAR_TIMEOUT_SEC), + f'{fqn} never appeared in GET /apps') + # A backed entry of the same name, published while the node runs; it is all that is + # left once the node exits. The marker ghost goes out after it. + injection = GhostInjection(ghosts=[marker], backed=[fqn], env=_fixture_env()) + self._fixtures.append(injection) + self.assertIsNotNone(injection.wait_status(INJECTION_TIMEOUT_SEC), injection.output()) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: sorted(enclaves) == ['', '/'], + INJECTION_TIMEOUT_SEC), + f'the test process graph does not list {fqn} both running and without an enclave: ' + f'{observed_enclaves(self.observer, fqn)}') + # The gateway's discovery of the injector can trail this process's by seconds. + self.assertTrue( + self._wait_listed(_app_id(marker), APPEAR_TIMEOUT_SEC), + f'GET /apps never listed {marker}, so the gateway graph may not hold the backed ' + f'entry of {fqn} when the node exits') + + node.send_signal(signal.SIGTERM) + node.wait(timeout=PROCESS_EXIT_TIMEOUT_SEC) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], + APPEAR_TIMEOUT_SEC), + f'after the node exited, the test process graph lists {fqn} with ' + f'{observed_enclaves(self.observer, fqn)}') + name, namespace = split_fqn(fqn) + self.assertTrue( + self.observer.get_publisher_names_and_types_by_node(name, namespace, True), + f'{fqn} resolves no endpoint in the test process graph, so this case would not show ' + 'that an endpoint keeps it listed') + + def check(): + self.assertIn( + 'bridged', self._ids('apps'), + f'{fqn} was seen running and has an endpoint, but dropped out of GET /apps') + + self._observe(check) + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 0) + + def test_06_many_leftovers_next_to_live_nodes(self, proc_output, gateway_node): + live = [f'/leftover_many_live/live_{index}' for index in range(3)] + for fqn in live: + self._start_node(fqn) + for fqn in live: + self.assertTrue(self._wait_listed(_app_id(fqn), APPEAR_TIMEOUT_SEC), + f'the live node {fqn} never appeared in GET /apps') + + leftover = self._start_leftover( + '/leftover_many_ns/leftover_many', announce=MANY_LEFTOVERS - 1) + # Announce after GET /apps lists the node, or its own discovery message can replace it. + self.assertTrue(leftover.announce_nodes(INJECTION_TIMEOUT_SEC), leftover.output()) + fqns = [leftover.fqn] + leftover.announced_fqns() + ids = {_app_id(fqn) for fqn in fqns} + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while ids - set(self._ids('apps')) and time.monotonic() < deadline: + time.sleep(0.2) + self.assertEqual( + ids - set(self._ids('apps')), set(), + 'the gateway never listed every announced node running, so it did not see them run') + + leftover.leave() + for fqn in fqns: + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + live_ids = {_app_id(fqn) for fqn in live} + + def check(): + app_ids = set(self._ids('apps')) + self.assertEqual(live_ids - app_ids, set(), 'a live node dropped out of GET /apps') + self.assertEqual(app_ids & ids, set(), 'leftovers are listed') + self.assertNotIn('leftover_many_ns', self._ids('functions')) + + self._observe(check) + + late = '/leftover_many_live/late' + self._start_node(late) + self.assertTrue( + wait_observed(self.observer, late, lambda enclaves: '/' in enclaves, + APPEAR_TIMEOUT_SEC), + f'the test process graph never listed {late}') + observed = time.monotonic() + self.assertTrue( + self._wait_listed('late', APPEAR_TIMEOUT_SEC), + f'{late} never appeared in GET /apps next to {MANY_LEFTOVERS} leftovers') + latency = time.monotonic() - observed + self.assertLessEqual( + latency, LISTED_AFTER_OBSERVED_SEC, + f'{late} took {latency:.2f} s to reach GET /apps after the graph listed it, next ' + f'to {MANY_LEFTOVERS} leftovers') + + for fqn in fqns: + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway must warn exactly once about the leftover of {fqn}') + + def test_07_the_hold_runs_out_only_for_a_name_no_entry_lists( + self, proc_output, gateway_node): + # Leftovers hidden by earlier cases, listed for longer than the hold, stay hidden: a + # listed leftover keeps its name remembered. + self.assertTrue(self.hidden_leftovers, 'no earlier case recorded a hidden leftover') + oldest = min(hidden_at for _, _, hidden_at in self.hidden_leftovers) + remaining = PAST_THE_HOLD_SEC - (time.monotonic() - oldest) + if remaining > 0: + time.sleep(remaining) + now = time.monotonic() + past_the_hold = [ + (fqn, app_id) for fqn, app_id, hidden_at in self.hidden_leftovers + if now - hidden_at >= PAST_THE_HOLD_SEC] + app_ids = set(self._ids('apps')) + for fqn, app_id in past_the_hold: + self.assertEqual( + observed_enclaves(self.observer, fqn), [''], + f'the test process graph no longer lists {fqn} as a leftover only') + self.assertNotIn( + app_id, app_ids, + f'the leftover of {fqn} was hidden and is listed again as {app_id} while the ' + 'graph still lists it: the gateway forgot the node after the hold') + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 1) + + # With no entry left, the first refresh past the hold forgets the name (refreshes run + # every 0.5 s here), so a later late sample is listed like a node never seen running. + remaining = PAST_THE_HOLD_SEC - (time.monotonic() - self.after_hold_departed) + if remaining > 0: + time.sleep(remaining) + self.after_hold.publish() + self._wait_left_over(AFTER_HOLD_FQN, INJECTION_TIMEOUT_SEC) + after_id = _app_id(AFTER_HOLD_FQN) + self.assertTrue( + self._wait_listed(after_id, LISTED_AFTER_OBSERVED_SEC), + f'the late sample for {AFTER_HOLD_FQN} arrived more than {PAST_THE_HOLD_SEC:.1f} s ' + f'after it departed, past the {LEFTOVER_HOLD_SEC:.0f} s hold, so the gateway no ' + 'longer remembers the node; it never appeared in GET /apps') + + def after_listed(): + self.assertIn(after_id, self._ids('apps'), + f'the leftover of {AFTER_HOLD_FQN}, past the hold, dropped out of ' + 'GET /apps') + + self._observe(after_listed) + self.assertEqual(self._warnings_for(proc_output, gateway_node, AFTER_HOLD_FQN), 0) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py new file mode 100644 index 000000000..1fb1ea4c1 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Discovery remembers a bounded number of departed nodes, and running nodes cost nothing. + +One participant announces more nodes than discovery remembers. Checks how many leftovers stay +hidden, that a node departing with no entry left is forgotten first past the capacity, and +that new nodes are still listed. Own gateway: a graph this size slows later reads. +""" + +import os +import subprocess +import time +import unittest + +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + LeftoverNode, + observed_enclaves, + split_fqn, + stop_process, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() + +REFRESH_INTERVAL_MS = 500 +REFRESH_DEBOUNCE_MS = 500 +# How long a claim is watched: three refreshes. An observation window, so not scaled. +OBSERVE_SEC = 3 * REFRESH_INTERVAL_MS / 1000.0 +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +# Listing bound of test_graph_leftover_nodes.test.py doubled: each remembered leftover costs +# endpoint queries on every read. +LISTED_AFTER_OBSERVED_SEC = 2 * ((REFRESH_DEBOUNCE_MS + 100) / 1000.0 + 4.0) * TIME_SCALE +LEFTOVER_DELAY_SEC = 1.0 +# A --delay longer than any run of this file: the late sample goes out on `publish`. +PUBLISH_ON_COMMAND_DELAY_SEC = 3600.0 +# GraphNodeListReader::kDefaultCapacity. +CAPACITY = 1024 +# Announced nodes: with the announcing node, more than discovery remembers. +ANNOUNCED = CAPACITY + 76 + +CAP_FQN = '/leftover_cap_ns/leftover_cap' +DEPART_FQN = '/leftover_depart_ns/leftover_depart' +RECENT_FQN = '/leftover_recent_ns/leftover_recent' +LATE_FQN = '/leftover_cap_live/late' + + +def generate_test_description(): + gateway_node = create_gateway_node( + extra_params={ + 'refresh_interval_ms': REFRESH_INTERVAL_MS, + 'discovery.refresh_debounce_ms': REFRESH_DEBOUNCE_MS, + }, + ) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +class TestGraphLeftoverNodesScale(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_scale_observer') + + @classmethod + def tearDownClass(cls): + cls.observer.destroy_node() + rclpy.shutdown() + + def setUp(self): + self._stop = [] + + def tearDown(self): + for stop in reversed(self._stop): + stop() + + @staticmethod + def _app_ids(): + response = requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=30) + response.raise_for_status() + return {item.get('id') for item in response.json().get('items', [])} + + def _wait_ids(self, predicate, timeout, interval=0.5): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + ids = self._app_ids() + if predicate(ids): + return ids + except requests.exceptions.RequestException: + pass + time.sleep(interval) + return None + + def _observe(self, check, interval=0.5): + deadline = time.monotonic() + OBSERVE_SEC + while time.monotonic() < deadline: + check(self._app_ids()) + time.sleep(interval) + + def _leftover(self, fqn, delay_sec, announce=0): + leftover = LeftoverNode(fqn, delay_sec, announce=announce, env=_fixture_env()) + self._stop.append(lambda: leftover.stop(PROCESS_EXIT_TIMEOUT_SEC)) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertIsNotNone( + self._wait_ids(lambda ids: split_fqn(fqn)[0] in ids, APPEAR_TIMEOUT_SEC), + f'{fqn} runs but never appeared in GET /apps') + return leftover + + def _wait_left_over(self, fqn, timeout): + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], timeout), + f'the test process graph lists {fqn} with {observed_enclaves(self.observer, fqn)} ' + 'rather than as a leftover') + + def _warned(self, proc_output, gateway_node, fqns): + text = ''.join( + output.text.decode(errors='replace') for output in proc_output[gateway_node]) + return sum(1 for fqn in fqns if f"Node '{fqn}' is not exposed" in text) + + def _wait_warned(self, proc_output, gateway_node, fqns, expected): + deadline = time.monotonic() + LISTED_AFTER_OBSERVED_SEC + while (self._warned(proc_output, gateway_node, fqns) < expected + and time.monotonic() < deadline): + time.sleep(0.2) + return self._warned(proc_output, gateway_node, fqns) + + def test_more_nodes_than_discovery_remembers(self, proc_output, gateway_node): + cap = self._leftover(CAP_FQN, PUBLISH_ON_COMMAND_DELAY_SEC, announce=ANNOUNCED) + self.assertTrue(cap.announce_nodes(INJECTION_TIMEOUT_SEC), cap.output()) + cap_fqns = [CAP_FQN] + cap.announced_fqns() + cap_ids = {split_fqn(fqn)[0] for fqn in cap_fqns} + self.assertIsNotNone( + self._wait_ids(lambda ids: cap_ids <= ids, 2 * APPEAR_TIMEOUT_SEC), + f'the gateway never listed all {len(cap_ids)} announced nodes running') + + # Running nodes do not count against the capacity: this node's leftover is hidden. + depart_id = split_fqn(DEPART_FQN)[0] + depart = self._leftover(DEPART_FQN, LEFTOVER_DELAY_SEC) + depart.leave() + self._wait_left_over(DEPART_FQN, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertEqual( + self._wait_warned(proc_output, gateway_node, [DEPART_FQN], 1), 1, + f'the gateway never warned about the leftover of {DEPART_FQN} next to ' + f'{len(cap_ids)} running nodes, so it forgot the node') + self.assertIsNotNone( + self._wait_ids(lambda ids: depart_id not in ids and cap_ids <= ids, + LISTED_AFTER_OBSERVED_SEC), + f'the leftover of {DEPART_FQN} is listed next to {len(cap_ids)} running nodes') + + # Every announced node departs with no entry of it left, then leaves a leftover. + cap.leave() + self.assertIsNotNone( + self._wait_ids(lambda ids: not (ids & cap_ids), + LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC), + 'the announced nodes left, but GET /apps kept listing some of them') + cap.publish() + namespace = split_fqn(CAP_FQN)[1] + + def left_over(entries): + enclaves = [enclave for _, node_namespace, enclave in entries + if node_namespace == namespace] + return len(enclaves) == len(cap_fqns) and set(enclaves) == {''} + + deadline = time.monotonic() + INJECTION_TIMEOUT_SEC + while (not left_over(self.observer.get_node_names_and_namespaces_with_enclaves()) + and time.monotonic() < deadline): + time.sleep(0.2) + self.assertTrue( + left_over(self.observer.get_node_names_and_namespaces_with_enclaves()), + f'the test process graph does not list all {len(cap_fqns)} nodes as leftovers') + + # As many leftovers as the capacity allows stay hidden; the rest are listed again. + listed_count = len(cap_fqns) - (CAPACITY - 1) + listed = self._wait_ids( + lambda ids: len(ids & cap_ids) == listed_count, LISTED_AFTER_OBSERVED_SEC) + self.assertIsNotNone( + listed, + f'of {len(cap_ids)} leftovers GET /apps lists {len(self._app_ids() & cap_ids)}; with ' + f'a capacity of {CAPACITY} and one more departed node remembered it must list ' + f'{listed_count}') + + def cap_check(ids): + self.assertEqual(len(ids & cap_ids), listed_count) + self.assertNotIn(depart_id, ids, f'the leftover of {DEPART_FQN} is listed') + + self._observe(cap_check) + self.assertEqual( + self._wait_warned(proc_output, gateway_node, cap_fqns, CAPACITY - 1), CAPACITY - 1) + + # Past the capacity a node with no entry left is forgotten first: its late sample is + # listed, and no older leftover takes its place. + recent_id = split_fqn(RECENT_FQN)[0] + recent = self._leftover(RECENT_FQN, PUBLISH_ON_COMMAND_DELAY_SEC) + recent.leave() + self.assertIsNotNone( + self._wait_ids(lambda ids: recent_id not in ids, + LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC), + f'{RECENT_FQN} left, but GET /apps kept listing it') + recent.publish() + self._wait_left_over(RECENT_FQN, INJECTION_TIMEOUT_SEC) + self.assertIsNotNone( + self._wait_ids(lambda ids: recent_id in ids, LISTED_AFTER_OBSERVED_SEC), + f'the leftover of {RECENT_FQN} is hidden: past the capacity discovery forgot an ' + 'older, listed leftover rather than the name no entry listed') + + def recent_check(ids): + self.assertIn(recent_id, ids, f'the leftover of {RECENT_FQN} dropped out of GET /apps') + self.assertEqual( + len(ids & cap_ids), listed_count, + 'an older leftover was listed in place of the name no entry listed') + self.assertNotIn(depart_id, ids, f'the leftover of {DEPART_FQN} is listed') + + self._observe(recent_check) + self.assertEqual(self._warned(proc_output, gateway_node, [RECENT_FQN]), 0) + + name, namespace = split_fqn(LATE_FQN) + late = subprocess.Popen( + [fixture_path('demo_rpm_sensor'), '--ros-args', '-r', f'__ns:={namespace}', + '-r', f'__node:={name}'], + env=_fixture_env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + self._stop.append(lambda: stop_process(late, PROCESS_EXIT_TIMEOUT_SEC)) + self.assertTrue( + wait_observed(self.observer, LATE_FQN, lambda enclaves: '/' in enclaves, + APPEAR_TIMEOUT_SEC), + f'the test process graph never listed {LATE_FQN}') + observed = time.monotonic() + self.assertIsNotNone( + self._wait_ids(lambda ids: name in ids, APPEAR_TIMEOUT_SEC, interval=0.1), + f'{LATE_FQN} never appeared in GET /apps next to {len(cap_ids)} leftovers') + latency = time.monotonic() - observed + self.assertLessEqual( + latency, LISTED_AFTER_OBSERVED_SEC, + f'{LATE_FQN} took {latency:.2f} s to reach GET /apps after the graph listed it, ' + f'next to {len(cap_ids)} leftovers') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py new file mode 100644 index 000000000..c8bcadff6 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py @@ -0,0 +1,283 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The graph readers outside discovery apply the same leftover rule. + +Covers the startup summary's peer count (discovery's reader) and parameter_beacon (a reader +of its own). Before the gateway starts, the graph lists a ``--ghost`` node, a ``--backed`` +node, a running node, and a running twin whose name is also listed without an enclave. The +``--backed`` node in GET /apps shows the gateway got the injector's transient-local history. +""" + +import os +import re +import time +import unittest + +from ament_index_python.packages import get_package_prefix +from launch import LaunchDescription +from launch.actions import ExecuteProcess, TimerAction +import launch_ros.actions +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + LeftoverNode, + observed_enclaves, + split_fqn, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() +PACKAGE = 'ros2_medkit_integration_tests' + +GHOST = '/ghost_boot_ns/ghost_boot' +BACKED_CONTROL = '/ghost_boot_control/ghost_boot_control' +LIVE = '/ghost_boot_live/ghost_boot_live' +TWIN = '/ghost_boot_twin/ghost_boot_twin' +LEFTOVER = '/leftover_beacon_ns/leftover_beacon' +# Ghost, backed, live and twin (once): the injector's node is hidden, the gateway's are not peers. +EXPECTED_PEERS = 4 + +# The gateway starts after the injected entries are in every graph. +GATEWAY_START_DELAY_SEC = 3.0 +STARTUP_TIMEOUT_SEC = 60.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +LEFTOVER_DELAY_SEC = 1.0 +POLL_INTERVAL_SEC = 1.0 +# Three 1 s poll cycles of watching parameter_beacon's clients. A window, so not scaled. +POLL_HOLD_SEC = 3.0 + + +def _node_action(fqn): + namespace, _, name = fqn.rpartition('/') + return launch_ros.actions.Node( + package=PACKAGE, + executable='demo_rpm_sensor', + name=name, + namespace=namespace, + output='screen', + additional_env=get_coverage_env(PACKAGE), + sigterm_timeout='30', + sigkill_timeout='15', + ) + + +def generate_test_description(): + injector = ExecuteProcess( + cmd=[fixture_path('ghost_node_injector'), '--ghost', GHOST, '--backed', BACKED_CONTROL, + '--ghost', TWIN], + output='screen', + sigterm_timeout='30', + sigkill_timeout='15', + ) + plugin_path = os.path.join( + get_package_prefix('ros2_medkit_param_beacon'), 'lib', 'ros2_medkit_param_beacon', + 'libparam_beacon_plugin.so') + gateway_node = create_gateway_node( + extra_params={ + 'plugins': ['parameter_beacon'], + 'plugins.parameter_beacon.path': plugin_path, + 'plugins.parameter_beacon.poll_interval_sec': POLL_INTERVAL_SEC, + 'plugins.parameter_beacon.param_timeout_sec': 1.0, + }, + ) + return ( + LaunchDescription([ + injector, + _node_action(LIVE), + _node_action(TWIN), + TimerAction(period=GATEWAY_START_DELAY_SEC, actions=[gateway_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'gateway_node': gateway_node, 'injector': injector}, + ) + + +def _output(proc_output, process): + return ''.join(output.text.decode(errors='replace') for output in proc_output[process]) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +class TestGraphLeftoverNodesAtStartup(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + STARTUP_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_startup_observer') + + @classmethod + def tearDownClass(cls): + cls.observer.destroy_node() + rclpy.shutdown() + + @staticmethod + def _app_ids(): + return [item.get('id') for item in + requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=10).json().get('items', [])] + + def _assert_injected(self, proc_output, injector): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + match = None + while match is None and time.monotonic() < deadline: + match = re.search(r'ghost_node_injector: matched_subscriptions=(\d+)', + _output(proc_output, injector)) + time.sleep(0.2) + self.assertIsNotNone(match, 'ghost_node_injector printed no status line') + self.assertGreater( + int(match.group(1)), 0, + 'ghost_node_injector timed out waiting for a matched subscription on ' + 'ros_discovery_info and never published') + for fqn, enclaves in ((GHOST, ['']), (TWIN, ['', '/'])): + self.assertTrue( + wait_observed(self.observer, fqn, + lambda found, want=enclaves: sorted(found) == want, + APPEAR_TIMEOUT_SEC), + f'the test process graph lists {fqn} with ' + f'{observed_enclaves(self.observer, fqn)}, not {enclaves}') + control_id = split_fqn(BACKED_CONTROL)[0] + while control_id not in self._app_ids() and time.monotonic() < deadline: + time.sleep(0.2) + self.assertIn(control_id, self._app_ids(), + f'{BACKED_CONTROL} never appeared in GET /apps, so the gateway never ' + 'received the injection') + + def _beacon_clients(self): + try: + return [name for name, _ in self.observer.get_client_names_and_types_by_node( + '_param_beacon_node', '/')] + except Exception: + # rclpy raises for a node this graph does not list yet. + return [] + + def _wait_polled(self, fqn, timeout, polled=True): + prefix = f'{fqn}/' + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if any(name.startswith(prefix) for name in self._beacon_clients()) == polled: + return True + time.sleep(0.2) + return False + + def test_01_startup_peer_count_counts_what_discovery_exposes( + self, proc_output, gateway_node, injector): + self._assert_injected(proc_output, injector) + deadline = time.monotonic() + STARTUP_TIMEOUT_SEC + summary = None + while summary is None and time.monotonic() < deadline: + summary = re.search(r'Discovery summary: (\d+) peer node\(s\)', + _output(proc_output, gateway_node)) + time.sleep(0.2) + self.assertIsNotNone(summary, 'the gateway never logged its startup discovery summary') + self.assertEqual( + int(summary.group(1)), EXPECTED_PEERS, + f'the startup summary must count {LIVE}, {BACKED_CONTROL}, {GHOST} and {TWIN} once') + + def test_02_parameter_beacon_polls_a_node_it_never_saw_running(self, proc_output, injector): + self._assert_injected(proc_output, injector) + self.assertTrue( + self._wait_polled(GHOST, APPEAR_TIMEOUT_SEC), + f'parameter_beacon never created a parameter client for {GHOST}: it has no enclave ' + 'and no endpoints, but the plugin never saw it running, so it is not a leftover') + + def _participant_prefix(self, node_name): + """GUID prefix of the participant `node_name` publishes /rosout from, or None.""" + for info in self.observer.get_publishers_info_by_topic('/rosout'): + if info.node_name == node_name and info.node_namespace == '/': + return bytes(info.endpoint_gid[:12]) + return None + + def test_03_parameter_beacon_does_not_poll_the_leftover_of_a_node_it_polled( + self, proc_output, gateway_node): + leftover = LeftoverNode(LEFTOVER, LEFTOVER_DELAY_SEC, env=_fixture_env()) + self.addCleanup(leftover.stop, PROCESS_EXIT_TIMEOUT_SEC) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertTrue( + self._wait_polled(LEFTOVER, APPEAR_TIMEOUT_SEC), + f'parameter_beacon never polled {LEFTOVER} while it ran, so it never saw it running') + + leftover.leave() + self.assertTrue( + wait_observed(self.observer, LEFTOVER, lambda enclaves: enclaves == [''], + LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC), + f'the test process graph lists {LEFTOVER} with ' + f'{observed_enclaves(self.observer, LEFTOVER)} rather than as a leftover') + # The plugin's node shares the gateway's participant, so discovery's warning shows the + # plugin's graph lists the leftover too. + gateway_prefix = self._participant_prefix('ros2_medkit_gateway') + self.assertIsNotNone(gateway_prefix, 'the gateway node publishes no /rosout') + self.assertEqual( + self._participant_prefix('_param_beacon_node'), gateway_prefix, + "parameter_beacon's node does not share the gateway node's participant, so the " + 'discovery warning would say nothing about the graph the plugin reads') + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + warning = f"Node '{LEFTOVER}' is not exposed" + while warning not in _output(proc_output, gateway_node) and time.monotonic() < deadline: + time.sleep(0.2) + self.assertTrue( + warning in _output(proc_output, gateway_node), + f'discovery never warned about the leftover of {LEFTOVER}, so the graph the plugin ' + 'reads may never have listed it') + + self.assertTrue( + self._wait_polled(LEFTOVER, APPEAR_TIMEOUT_SEC, polled=False), + f'parameter_beacon still has a parameter client for the leftover of {LEFTOVER}') + deadline = time.monotonic() + POLL_HOLD_SEC + while time.monotonic() < deadline: + clients = [name for name in self._beacon_clients() + if name.startswith(f'{LEFTOVER}/')] + self.assertFalse( + clients, + f'parameter_beacon polls the leftover of {LEFTOVER}, a node it polled while it ' + f'ran: {clients}') + time.sleep(0.2) + self.assertEqual(observed_enclaves(self.observer, LEFTOVER), ['']) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py b/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py new file mode 100644 index 000000000..9edd4d43f --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py @@ -0,0 +1,624 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""parameter_beacon duration and max_hints bounds, and polling of nodes that answer badly or never. + +One gateway loads the plugin once per instance, each with its own prefix. The sweep sets +durations to NaN, -inf, below the minimum, the minimum, the maximum, just above it, +inf and +3e9; the steady instance has a working configuration. More instances take one max_hints point +each: 0, -5, 1, 2147483647, 2147483648, 2^32 + 1, 1e12 and NaN. The instance loaded first +answers the beacon endpoint and has NaN TTL and expiry. Polled nodes: answering, unserved +(never answers), get_silent (answers list only), typed_unset (rclpy, a typed parameter without +a value, so its get answers carry no values), two hint sources that give each max_hints +instance a hint of its own, and lifetime_beacon, which gives the first instance a hint for +itself until the test stops it. +""" + +import collections +import os +import tempfile +import threading +import time +import unittest + +from ament_index_python.packages import get_package_prefix +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +from rcl_interfaces.msg import ParameterType, ParameterValue +from rcl_interfaces.srv import GetParameters, ListParameters +import rclpy +from rclpy.executors import SingleThreadedExecutor +from rclpy.parameter import Parameter +import requests + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, DEFAULT_BASE_URL, get_time_scale +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +TIME_SCALE = get_time_scale() +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +# Give-up bound for anything the instances do; every wait ends as soon as its condition holds. +POLL_TIMEOUT_SEC = 30.0 * TIME_SCALE +# A thread busier than this over the CPU window is spinning. +SPIN_CPU_FRACTION = 0.5 +CPU_WINDOW_SEC = 3.0 + +MAX_SECONDS = '2147483647' +DURATIONS = ('poll_interval_sec', 'poll_budget_sec', 'param_timeout_sec', 'beacon_ttl_sec', + 'beacon_expiry_sec') +MINIMUM = dict.fromkeys(DURATIONS, '0.1') +MINIMUM['beacon_expiry_sec'] = '1' +BELOW = dict.fromkeys(DURATIONS, '0.05') +BELOW['beacon_expiry_sec'] = '0.5' + + +def _point(name, key): + """(YAML spelling, how the plugin prints it, what it clamps to or None) of a sweep point.""" + return { + 'nan': ('.nan', 'nan', MINIMUM[key]), + 'neg_inf': ('-.inf', '-inf', MINIMUM[key]), + 'below': (BELOW[key], BELOW[key], MINIMUM[key]), + 'min': (MINIMUM[key] if key != 'beacon_expiry_sec' else '1.0', MINIMUM[key], None), + 'max': (MAX_SECONDS + '.0', MAX_SECONDS, None), + 'above': ('2147483648.0', '2147483648', MAX_SECONDS), + 'inf': ('.inf', 'inf', MAX_SECONDS), + 'huge': ('3000000000.0', '3000000000', MAX_SECONDS), + }[name] + + +LOW_POINTS = ('nan', 'neg_inf', 'below', 'min') +HIGH_POINTS = ('max', 'above', 'inf', 'huge') +# Every duration at a low point: the instance keeps cycling with 0.1 s timeouts. +CYCLING = {f'low_{p}': {key: p for key in DURATIONS} for p in LOW_POINTS} +# Timeout, budget, TTL and expiry at a high point, the interval at its minimum: the instance +# reaches a node that never answers and waits there. +WAITING = {f'high_{p}': dict({key: p for key in DURATIONS[1:]}, poll_interval_sec='min') + for p in HIGH_POINTS} +# The interval at a high point: the instance runs one cycle. +ONE_CYCLE = {f'interval_{p}': {'poll_interval_sec': p} for p in HIGH_POINTS} +SWEEP = dict(**CYCLING, **WAITING, **ONE_CYCLE) +STEADY = 'steady' +STEADY_CONFIG = {'poll_interval_sec': '0.1', 'poll_budget_sec': '30.0', 'param_timeout_sec': '0.3'} +# Bound for the steady counts to settle: a list answer of get_silent is counted before its get. +SETTLE_TIMEOUT_SEC = 1.0 * TIME_SCALE + +# max_hints points: instance -> (YAML spelling, warning or None, hints kept of the two sources) +HINT_INSTANCES = { + 'hints_zero': ('0', 'max_hints clamped from 0 to 1', 1), + 'hints_negative': ('-5', 'max_hints clamped from -5 to 1', 1), + 'hints_min': ('1', None, 1), + 'hints_max': ('2147483647', None, 2), + 'hints_above': ('2147483648', 'max_hints clamped from 2147483648 to 2147483647', 2), + 'hints_wrapping': ('4294967297', 'max_hints clamped from 4294967297 to 2147483647', 2), + 'hints_double': ('1.0e12', 'max_hints 1e+12 is not an integer, using 10000', 2), + 'hints_nan': ('.nan', 'max_hints nan is not an integer, using 10000', 2), +} +HINT_CONFIG = {'poll_interval_sec': '1.0', 'param_timeout_sec': '0.1'} +HINT_SOURCES = ('a', 'b') +CAPACITY_WARNING = 'BeaconHintStore capacity reached (max_hints=1)' + +# Loaded first, so its route answers the beacon endpoint. TTL becomes 3 * 0.1 s, expiry 1 s. +LIFETIME = 'lifetime' +LIFETIME_CONFIG = {'poll_interval_sec': '0.1', 'param_timeout_sec': '0.3', + 'beacon_ttl_sec': '.nan', 'beacon_expiry_sec': '.nan'} +LIFETIME_TTL_SEC = 0.3 +LIFETIME_EXPIRY_SEC = 1.0 +# How late the test may see the beacon go stale or be removed. +LIFETIME_LATENESS_SEC = 0.3 * TIME_SCALE + + +def _prefix(instance): + return f'oor_{instance}' + + +def _parameter_file(): + plugin_path = os.path.join( + get_package_prefix('ros2_medkit_param_beacon'), 'lib', 'ros2_medkit_param_beacon', + 'libparam_beacon_plugin.so') + settings = {LIFETIME: LIFETIME_CONFIG} + for instance, points in SWEEP.items(): + settings[instance] = {key: _point(point, key)[0] for key, point in points.items()} + settings[STEADY] = STEADY_CONFIG + for instance, (value, _, _) in HINT_INSTANCES.items(): + settings[instance] = dict(HINT_CONFIG, max_hints=value) + lines = [ + 'ros2_medkit_gateway:', + ' ros__parameters:', + ' plugins: [' + ', '.join(settings) + ']', + ] + for instance, values in settings.items(): + lines.append(f' plugins.{instance}.path: "{plugin_path}"') + lines.append(f' plugins.{instance}.parameter_prefix: "{_prefix(instance)}"') + for key, value in values.items(): + lines.append(f' plugins.{instance}.{key}: {value}') + handle = tempfile.NamedTemporaryFile( + 'w', prefix='param_beacon_bounds_', suffix='.yaml', delete=False) + with handle: + handle.write('\n'.join(lines) + '\n') + return handle.name + + +PARAMETER_FILE = _parameter_file() + + +def generate_test_description(): + gateway_node = create_gateway_node(parameter_files=[PARAMETER_FILE]) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _prefix_of(name): + return name.split('.', 1)[0] + + +class _Counter: + """Thread-safe request counts per (node, service, prefix).""" + + def __init__(self): + self._counts = collections.Counter() + self._lock = threading.Lock() + + def add(self, node, service, prefixes): + with self._lock: + for prefix in prefixes: + self._counts[(node, service, prefix)] += 1 + + def get(self, node, service, instance): + with self._lock: + return self._counts[(node, service, _prefix(instance))] + + def snapshot(self): + with self._lock: + return collections.Counter(self._counts) + + +class _UnansweredServices: + """Takes requests from services no executor serves, counts them and never answers.""" + + def __init__(self, counter): + self._counter = counter + self._services = [] + self._stop = threading.Event() + self._thread = threading.Thread(target=self._run, daemon=True) + + def add(self, node_label, service): + self._services.append((node_label, service)) + + def start(self): + self._thread.start() + + def stop(self): + self._stop.set() + self._thread.join(timeout=10) + + def _run(self): + while not self._stop.is_set(): + for node_label, service in self._services: + while True: + with service.handle: + request, _header = service.handle.service_take_request( + service.srv_type.Request) + if request is None: + break + if service.srv_type is ListParameters: + self._counter.add(node_label, 'list', request.prefixes) + else: + self._counter.add(node_label, 'get', + {_prefix_of(name) for name in request.names}) + self._stop.wait(0.02) + + +def _thread_cpu_ticks(pid): + ticks = {} + for tid in os.listdir(f'/proc/{pid}/task'): + try: + with open(f'/proc/{pid}/task/{tid}/stat') as stat: + fields = stat.read().rsplit(')', 1)[1].split() + except OSError: + continue + ticks[tid] = int(fields[11]) + int(fields[12]) + return ticks + + +class TestParamBeaconBounds(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.counter = _Counter() + cls.nodes = [] + + # The max_hints instances get their hints from the hint sources only. + hint_prefixes = {_prefix(instance) for instance in HINT_INSTANCES} + answering = cls._node('answering_beacon') + answering.create_service( + ListParameters, '~/list_parameters', cls._answer_list('answering', hint_prefixes)) + answering.create_service(GetParameters, '~/get_parameters', cls._answer_get) + + get_silent = cls._node('get_silent') + get_silent.create_service( + ListParameters, '~/list_parameters', cls._answer_list('get_silent')) + # Its get service lives on a hidden node no executor spins. + get_backend = cls._node('_get_silent_backend') + get_silent_get = get_backend.create_service( + GetParameters, '/get_silent/get_parameters', cls._answer_get) + + cls.unserved = cls._node('unserved_parameters') + unserved_list = cls.unserved.create_service( + ListParameters, '~/list_parameters', cls._answer_list('unserved')) + unserved_get = cls.unserved.create_service( + GetParameters, '~/get_parameters', cls._answer_get) + + typed_unset = rclpy.create_node('typed_unset') + cls.nodes.append(typed_unset) + typed_unset.declare_parameter(f'{_prefix(STEADY)}.entity_id', 'typed_unset') + typed_unset.declare_parameter(f'{_prefix(STEADY)}.display_name', 'Typed Unset') + typed_unset.declare_parameter(f'{_prefix(STEADY)}.stable_id', Parameter.Type.STRING) + get_service = next( + service for service in typed_unset.services + if service.srv_name.endswith('/get_parameters')) + rclpy_get = get_service.callback + + def counting_get(request, response): + cls.counter.add('typed_unset', 'get', {_prefix_of(name) for name in request.names}) + return rclpy_get(request, response) + + get_service.callback = counting_get + + sources = [] + for label in HINT_SOURCES: + source = cls._node(f'hint_source_{label}') + source.create_service(ListParameters, '~/list_parameters', + cls._answer_list_for(hint_prefixes)) + source.create_service(GetParameters, '~/get_parameters', cls._answer_hint(label)) + sources.append(source) + + cls.lifetime_serving = threading.Event() + cls.lifetime_serving.set() + cls.lifetime_last_get = None + lifetime = cls._node('lifetime_beacon') + lifetime.create_service(ListParameters, '~/list_parameters', cls._answer_lifetime_list) + lifetime.create_service(GetParameters, '~/get_parameters', cls._answer_lifetime_get) + + cls.unanswered = _UnansweredServices(cls.counter) + cls.unanswered.add('unserved', unserved_list) + cls.unanswered.add('unserved', unserved_get) + cls.unanswered.add('get_silent', get_silent_get) + cls.unanswered.start() + + cls.executor = SingleThreadedExecutor() + for node in (answering, get_silent, typed_unset, *sources, lifetime): + cls.executor.add_node(node) + cls.spin_thread = threading.Thread(target=cls.executor.spin, daemon=True) + cls.spin_thread.start() + + @classmethod + def tearDownClass(cls): + cls.unanswered.stop() + cls.executor.shutdown() + cls.spin_thread.join(timeout=10) + for node in cls.nodes: + node.destroy_node() + rclpy.shutdown() + os.remove(PARAMETER_FILE) + + @classmethod + def _node(cls, name): + node = rclpy.create_node(name, start_parameter_services=False) + cls.nodes.append(node) + return node + + @classmethod + def _answer_list(cls, label, skipped=frozenset()): + def answer(request, response): + cls.counter.add(label, 'list', request.prefixes) + response.result.names = [ + f'{prefix}.entity_id' for prefix in request.prefixes if prefix not in skipped] + return response + return answer + + @staticmethod + def _answer_get(request, response): + response.values = [ + ParameterValue(type=ParameterType.PARAMETER_STRING, string_value='answering_beacon') + for _ in request.names] + return response + + @staticmethod + def _answer_list_for(prefixes): + def answer(request, response): + response.result.names = [ + f'{prefix}.entity_id' for prefix in request.prefixes if prefix in prefixes] + return response + return answer + + @staticmethod + def _answer_hint(label): + """Entity id '_