diff --git a/docs/api/rest.rst b/docs/api/rest.rst index 5f55fb862..00aceb0aa 100644 --- a/docs/api/rest.rst +++ b/docs/api/rest.rst @@ -1517,13 +1517,19 @@ Query and manage faults. - ``freeze_frame``: Data captured at fault confirmation. Entity frames for faults that were already confirmed when the gateway started are captured at gateway start instead and carry ``"capture_origin": "startup"`` in - their ``x-medkit`` block. For a plugin-backed entity that reports its - link down, the values are the plugin's last known ones and may predate - the confirmation by the length of the outage; such entries carry - ``connected`` (the payload's link flag, ``false`` for the loss-of-comms - case) and ``source_timestamp`` (the payload's own timestamp, verbatim) - in ``x-medkit``, both only when the plugin's payload reports them. - ``captured_at`` always dates the capture, not the values. + their ``x-medkit`` block. An entity frame also carries ``source`` in + ``x-medkit``, naming the path that read the values + (``plugin_data_provider`` for the owning plugin's DataProvider, + ``plugin_x_plc_data_route`` for its ``x-plc-data`` route). These values + are not a ROS message, so ``topic`` and ``message_type`` are empty and + ``source`` is the only field saying where the numbers came from. For a + plugin-backed entity that reports its link down, the values are the + plugin's last known ones and may predate the confirmation by the length of + the outage. Such entries carry ``connected`` (the payload's link flag, + ``false`` for the loss-of-comms case) and ``source_timestamp`` (the + payload's own timestamp, verbatim) in ``x-medkit``, both only when the + plugin's payload reports them. ``captured_at`` always dates the capture, + not the values. - ``rosbag``: Recording file available via bulk-data endpoint **Response codes:** diff --git a/docs/config/discovery-options.rst b/docs/config/discovery-options.rst index c47613dc0..59374a167 100644 --- a/docs/config/discovery-options.rst +++ b/docs/config/discovery-options.rst @@ -85,8 +85,76 @@ filters out ROS 2 internal infrastructure nodes such as ``_ros2cli_*``, SOVD entities. The filter applies to both locally discovered Apps and peer-discovered Apps (after stripping the peer prefix). +The same switch also excludes the helper nodes the gateway runs inside its own +process. Three of them exist in every deployment, named after the gateway node: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Node + - What it is + * - ``_sub`` + - The subscription executor that serves ``/data`` reads and cyclic + subscriptions. + * - ``_fault_clients`` + - The service clients that talk to the fault manager. + * - ``_lifecycle_state_reader`` + - The client that reads managed nodes' lifecycle state. + +None of these names begins with an underscore, so the convention above does not +cover them, and without this rule the gateway would list its own plumbing as +diagnosable Apps. They carry no parameters, services or actions of their own, so +there is nothing on them to diagnose. + +While this setting is ``true`` they are also left out of the linking report on +``GET /health`` - ``discovery.linking.orphan_count``, and the +``ros_node_fqns`` of the ``unmanifested_nodes`` warning raised under +``unmanifested_nodes: error``: a node that can never be an App is not a node to +declare in a manifest. That report exists only in ``hybrid`` mode, where +manifest apps are linked to runtime nodes; ``runtime_only`` has no linking block +at all. + +**The gateway's own node stays an App.** Its ROS parameters are what the +gateway serves as that App's configurations, so +``/apps//configurations`` is where a client reads and writes them - +there is no other entity carrying, for instance, +``aggregation.peer_auth_header``. Two gateways watching one graph also have to +agree about what is on it, which they cannot do if each hides a different node. + +Namespaces: the subscription node is created with the gateway's own namespace, +while the other two are created from the gateway's node name alone and take the +process default namespace. A process-wide remap (``-r __ns:=/line_a``) moves all +four together; a remap naming the gateway alone +(``-r ros2_medkit_gateway:__ns:=/line_a``) moves the gateway and the +subscription node and leaves the other two in the default namespace. Both +spellings are recognised, so the rule holds either way. + +.. note:: + + A node named ``_fault_clients`` or + ``_lifecycle_state_reader`` in the **root** namespace is treated as + plumbing whichever gateway created it. Two gateways that keep the default + node name and differ only in namespace produce the same fully qualified name + for those two nodes, so the name cannot say whose they are, and each gateway + will filter the other's. They carry nothing to diagnose in either process, + and the alternative is that a namespaced gateway lists and counts its own + plumbing. Give each gateway its own node name (``-r __node:=gateway_a``, as + :doc:`../tutorials/multi-instance` does) when several run on one ROS graph + and you want each one's nodes distinguishable. The subscription node is not + affected: it always follows its gateway's namespace. + +A manifest- or plugin-declared App bound to one of these nodes is not served, +and the gateway logs a warning naming it: bind the App to the node you meant, +or drop the declaration. + Set to ``false`` if you need to expose all ROS 2 nodes regardless of naming -convention. +convention. That re-exposes the three helper nodes as well as the underscore +ones, and in ``hybrid`` mode the linking report follows: a helper node the +gateway serves as an App is also an undeclared one, so it is counted in +``discovery.linking.orphan_count`` and, under ``unmanifested_nodes: error``, +named in the ``ros_node_fqns`` of the ``unmanifested_nodes`` warning. With +``true`` it appears in neither. Function Entities from Namespaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -446,6 +514,17 @@ Configuration # Default: 100 plugins.topic_beacon.max_messages_per_second: 100 +``beacon_ttl_sec`` takes 0.1 to 2147483647 s, ``beacon_expiry_sec`` 1.0 to 2147483647 s, and +``max_messages_per_second`` 1 to 10000. A value above its maximum, including ``.inf`` in a +parameter file, becomes the maximum. NaN, ``-.inf`` and a value below the minimum become the +minimum. The gateway logs a warning naming the key, the value and what replaced it. Messages +over the rate limit are dropped without a log. + +``max_hints`` takes an integer from 1 to 2147483647. A 64-bit integer outside that range +becomes the nearer bound, with the same warning. The parameter parser reads an integer that +does not fit in 64 bits as a double. A double, such as ``1.0e12`` or ``.nan``, is refused with +a warning and the default 10000 is used. + Beacon Lifecycle ^^^^^^^^^^^^^^^^ @@ -601,6 +680,36 @@ Configuration # Default: 10000 plugins.parameter_beacon.max_hints: 10000 +Every duration takes its minimum - 0.1 s, and 1.0 s for ``beacon_expiry_sec`` - up to +2147483647 s. Fast DDS keeps the seconds of a wait in a 32-bit signed integer, and a longer +``param_timeout_sec`` would make the poll thread spin. A value above the maximum, including +``.inf`` in a parameter file, becomes the maximum. NaN, ``-.inf`` and a value below the +minimum become the minimum. The gateway logs a warning naming the key, the value and what +replaced it. + +``max_hints`` takes an integer from 1 to 2147483647. A 64-bit integer outside that range +becomes the nearer bound, with the same warning. The parameter parser reads an integer that +does not fit in 64 bits as a double. A double, such as ``1.0e12`` or ``.nan``, is refused with +a warning and the default 10000 is used. + +In ``runtime_only`` and ``manifest_only`` mode the plugin reads its poll targets from the +ROS graph. In ``hybrid`` mode the merge pipeline passes it the discovered Apps, but the +gateway's refresh then calls it again with no Apps, which clears them. So a poll cycle mostly +reads the graph there too; only a cycle that starts between the two calls polls the nodes of +the online Apps that discovery bound to a node. A graph read skips hidden nodes (a name +starting with ``_``), the gateway's own node and its helper nodes (``_sub``, +``_fault_clients`` and ``_lifecycle_state_reader``): they carry no beacon. + +A parameter request that gets no answer within ``param_timeout_sec`` - waiting for the +service, listing parameters or getting their values - is given up and removed from its +client, and the node is skipped for the next 1, 2, 4 and then 8 poll cycles while it keeps +timing out. A node that answers is polled on every cycle, also when its answer carries no +values: rclpy answers so when one of the parameters asked for is declared with a type and no +value, and the plugin then stores no hint for the node. None of this is logged. + +Each node keeps one parameter client across poll cycles. A node that is no longer a poll +target, also when no node is, loses its client and its skip count. + Parameter Naming ^^^^^^^^^^^^^^^^ diff --git a/docs/config/server.rst b/docs/config/server.rst index 0171b268a..8f5410c7d 100644 --- a/docs/config/server.rst +++ b/docs/config/server.rst @@ -480,10 +480,12 @@ How long a departed node keeps being listed Once a node has actually left the ROS graph, ``GET /apps`` stops listing it within roughly one refresh: at most ``discovery.refresh_debounce_ms`` plus the 100 ms graph -poll, or ``refresh_interval_ms``, whichever comes first. Nothing is retained behind -that - every refresh rebuilds the entity set from a live read of the graph - so this -is the whole of the gateway's share, and it is the only part of a departure the -gateway can be held to. +poll, or ``refresh_interval_ms``, whichever comes first. Every refresh rebuilds the +entity set from a live read of the graph, so this is the whole of the gateway's share, +and it is the only part of a departure the gateway can be held to. What the gateway +remembers between reads - the names it saw running, described below - decides only +whether an entry the graph still lists after its participant left is exposed; it never +keeps listing a node the graph no longer lists. Before that point nothing here is promised, and the difference is not small: @@ -504,6 +506,83 @@ rather than assume a deadline. Anything that measures a departure should establi that the process has actually exited first, and only then hold the gateway to the figure above. +The graph itself can also keep a node after its process has exited, for as long as the +gateway runs. The DDS RMWs take a participant's node list from discovery messages on an +internal topic, read on a listener thread of their own. When that thread takes a message +only after the participant was removed, the message puts the node back, with an empty +enclave and none of its endpoints, and nothing removes it again. This page calls such an +entry a leftover. + +An empty enclave on its own does not make a leftover. A node on the far side of a DDS +router, or a node whose participant was created outside rcl (a micro-ROS agent's, for +example), also reads an empty enclave, and the router's far side resolves no endpoints +either. So the gateway leaves an entry out only for a node it has itself seen running. +Per node name, on every read of the graph: + +* an entry with an enclave is listed, and the name counts as seen running; +* an entry without an enclave is dropped when another entry of the same name has one, so a + restarted node is listed once; +* an entry without an enclave for a name the gateway has seen running is left out when the + graph resolves no publisher or subscriber for the node (a service or client is a request + and a reply endpoint, so it counts too); +* an entry without an enclave for a name the gateway has never seen running is listed. + +A leftover is left out of ``GET /apps`` and ``GET /apps/{id}``, out of the Functions +derived from namespaces, and out of the bare-name collision check that would otherwise +rename a live node sharing its name. The read that starts leaving a node out logs: + +.. code-block:: text + + [WARN] [ros2_medkit_gateway]: Node '/ns/name' is not exposed: this gateway saw it running, and the ROS graph still lists it after its participant left, with no endpoints + +Later reads that keep leaving it out stay quiet. The warning comes back only after a read +that did not leave the node out. + +Which names the gateway saw running comes from the reads themselves. The names running on +the previous read are that read's own list, so they cost nothing however many nodes run. A +name that ran on the previous read and does not run on this one - the graph lists no entry +of it, or only entries without an enclave - has departed, and the gateway remembers it until +it runs again or until it is forgotten on a read: the first read that finds no entry of it +more than 10 seconds after the first read that found none. While an entry of it without an +enclave is listed, those 10 seconds do not start. They cover the gap between the +participant's removal and the late message: the time the listener thread takes to process a +message it already holds, measured under a millisecond on an idle host and up to 0.64 s with +its process and eight busy threads sharing two cores. + +The name is forgotten only on a read. Reads come from graph changes, the refresh backstop +(``refresh_interval_ms``), the gateway's start and, in ``runtime_only`` mode, a request for an +App or a Function that a refresh removed from the entity cache while the request ran. A late +message that arrives before the read that forgets the name stays hidden, however long after +the 10 seconds it comes: the late message itself changes the graph, and the read that follows +finds its entry. Only a late message that arrives after that read is listed, like a node the +gateway never saw running. + +The gateway remembers at most 1024 departed names. Past that it forgets first the names the +graph has listed no entry of for longest, and then the names that departed longest ago; a +forgotten name whose only entries are leftovers is listed again, without a warning. So once +1024 leftovers are hidden, a node that departs with no entry of it left is the first name +forgotten, and a late message for it is listed. Every remembered leftover costs two endpoint +queries on each read, so the cap also bounds that work. + +A node that ran on this host and then appears only behind a DDS router, as an entry without +an enclave, is hidden like a leftover if it appears before the gateway forgets the name or +while a leftover of it is still listed. + +The startup discovery summary counts peer nodes through the same memory, so in +``runtime_only`` mode, and in ``hybrid`` mode with the runtime layer enabled, it counts what +discovery would list at that moment. When discovery does not read the graph itself - in +``manifest_only`` mode, or in ``hybrid`` mode with ``discovery.runtime.enabled: false`` - the +summary's read is the first, so no node has been seen running: it counts every entry the +graph lists, except an entry without an enclave next to an entry of the same name with one. + +The ``param_beacon`` plugin reads the graph itself only when discovery gives it no poll +targets, and then through a memory of its own: a node its reads saw running is not polled +once only a leftover of it is listed. It logs nothing about it. + +A node restarted while a leftover of its previous instance is still listed can show no +services, and so no operations, because the graph answers per-node queries from whichever +of the two participants sorts first. + Thread Pools ------------ diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index 1070aa373..98f54b757 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -239,6 +239,26 @@ with: ros2 run ros2_medkit_gateway gateway_node --ros-args \ -p entity_freeze_frame.enabled:=false +A plugin entity's values are not a ROS message, so ``topic`` and +``message_type`` are empty on these frames. ``x-medkit.source`` names the +capture path instead, so a consumer can still tell where the values came +from: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - ``x-medkit.source`` + - Meaning + * - ``plugin_data_provider`` + - Read through the owning plugin's ``DataProvider::list_data``. + * - ``plugin_x_plc_data_route`` + - Read by dispatching the owning plugin's own ``x-plc-data`` route + in-process (plugins that export no ``DataProvider``). + +The field is absent on freeze-frames captured by the fault manager from a ROS +topic. Those carry a real ``topic`` and ``message_type`` instead. + Example plugin-entity freeze-frame in the fault response: .. code-block:: json @@ -250,6 +270,7 @@ Example plugin-entity freeze-frame in the fault response: "x-medkit": { "topic": "", "message_type": "", + "source": "plugin_x_plc_data_route", "full_data": {"tank_level": 87.5, "pump_running": true}, "captured_at": "2026-07-14T12:00:00.000Z" } diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt index 9320d0012..3e60bf7e5 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/CMakeLists.txt @@ -85,11 +85,7 @@ if(BUILD_TESTING) include(ROS2MedkitTestDomain) # Include plugin source in test (plugin is MODULE/dlopen, test instantiates directly) - # TIMEOUT 180: the 12 test cases each spin a node for ~200-400 ms. Under - # plain gcc that's ~4 s total, well under the 30 s ctest default, but TSan - # instrumentation inflates that 5-10x and occasionally trips the 30 s cap - # (CI saw timeouts at test 4/12 around 27 s). 180 s covers the worst-case - # sanitizer slowdown without masking real hangs. + # TIMEOUT 180: the cases spin real nodes, and TSan slows them 5-10x past the 30 s default. medkit_add_gmock(test_param_beacon_plugin test/test_param_beacon_plugin.cpp src/param_beacon_plugin.cpp diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md index a7fe897cb..1fcd41e2d 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/README.md @@ -11,8 +11,35 @@ through standard ROS 2 parameters. 3. Entity metadata is mapped into the SOVD hierarchy via `BeaconEntityMapper` 4. Results are exposed at the `x-medkit-param-beacon` vendor extension endpoint -In non-hybrid discovery mode, the plugin discovers poll targets automatically from the -ROS 2 graph. In hybrid mode, targets come from the manifest. +In `runtime_only` and `manifest_only` discovery the plugin reads its poll targets from the +ROS 2 graph. In `hybrid` mode the merge pipeline passes it the discovered Apps, but the +gateway's refresh then calls it again with no Apps, which clears them, so a poll cycle mostly +reads the graph there too. A graph read skips hidden nodes (a name starting with `_`), the +gateway's own node and its helper nodes (`_sub`, `_fault_clients`, +`_lifecycle_state_reader`): they carry no beacon. The plugin remembers what its own +reads saw: a node that ran on one of its reads is not polled once the graph lists only a +leftover of it, an entry with an empty enclave and no endpoints, which a node whose +participant has left can leave behind (see "How long a departed node keeps being listed" in +the gateway's `docs/config/server.rst`). + +Each node keeps one parameter client across poll cycles. A node that is no longer a target, +also when no node is, loses its client. A parameter request that gets no answer within +`param_timeout_sec` - waiting for the service, listing parameters or getting their values - +is given up and removed from its client, and the node is skipped for the next 1, 2, 4 and +then 8 poll cycles while it keeps timing out, so a node whose parameter services never answer +does not stall polling. A node that answers is polled on every cycle, also when its answer +carries no values: rclpy answers so when one of the parameters asked for is declared with a +type and no value, and the plugin then stores no hint. None of this is logged. + +Every duration below takes its minimum (0.1 s, `beacon_expiry_sec` 1.0 s) up to 2147483647 s, +the longest wait Fast DDS keeps without the poll thread spinning. A value above the maximum, +including `.inf` in a parameter file, becomes the maximum; NaN, `-.inf` and a value below the +minimum become the minimum. Each replacement is logged as a warning. + +`max_hints` takes an integer from 1 to 2147483647. A 64-bit integer outside that range becomes +the nearer bound. The parameter parser reads an integer that does not fit in 64 bits as a +double. A double, such as `1.0e12` or `.nan`, is refused and the default 10000 is used. Both +are logged as warnings. ## Configuration diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst index 3dd579cb3..e02376f5f 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/design/index.rst @@ -21,9 +21,9 @@ components. How It Works ------------ -1. During each poll cycle, the plugin retrieves the current list of ROS 2 nodes - from the gateway's entity cache -2. For each node, it creates (or reuses) an ``AsyncParametersClient`` and fetches +1. During each poll cycle, the plugin takes its targets from the last ``introspect()`` + input, or reads the ROS 2 graph when that input had none +2. For each node, it creates (or reuses) a ``RealParameterClient`` and fetches all parameters matching the configured prefix 3. Parameters are parsed into a ``BeaconHint``: ``entity_id``, ``stable_id``, ``function_ids``, ``metadata.*`` keys, etc. @@ -46,18 +46,25 @@ their parameters once at startup. Client Management ~~~~~~~~~~~~~~~~~ -The plugin maintains a cache of ``AsyncParametersClient`` instances keyed by node -FQN. Clients for nodes that disappear from the graph are evicted after a -configurable timeout. A lock ordering protocol (``nodes_mutex_`` then -``clients_mutex_`` then ``param_ops_mutex_``) prevents deadlocks between the -poll thread and the introspection callback. +The plugin keeps one ``RealParameterClient`` per node FQN across cycles. It holds an +``rclcpp::Client`` for ``list_parameters`` and one for ``get_parameters`` on the plugin's +own node, and spins that node on its own executor for the length of a call. A request that +gets no answer within ``param_timeout_sec`` is removed from its client with +``remove_pending_request()``, so a node that never answers leaves nothing pending. + +At the start of each cycle the poll thread drops the clients of nodes that are not among the +cycle's targets, also when there are none. Only the poll thread spins the plugin's node, and +only inside a call, so no executor holds a client when the poll thread, or ``shutdown()`` +after joining it, destroys one. A lock ordering protocol (``nodes_mutex_`` then +``clients_mutex_`` then ``param_ops_mutex_``) prevents deadlocks between the poll thread and +the introspection callback. Backoff and Budget ~~~~~~~~~~~~~~~~~~ Nodes that fail to respond (timeout, unavailable) accumulate a backoff counter. Subsequent poll cycles skip backed-off nodes with exponentially increasing skip -counts. A per-cycle time budget (default 10 seconds) prevents a few slow nodes +counts. Any answer clears the counter, including a get answer without values. A per-cycle time budget (default 10 seconds) prevents a few slow nodes from starving the rest of the poll targets. The start offset rotates each cycle so that all nodes eventually get polled even under budget pressure. diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp index 97e1bd5cf..020dd712f 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/param_beacon_plugin.hpp @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include #include #include @@ -39,6 +41,7 @@ #include "ros2_medkit_gateway/core/plugins/plugin_types.hpp" #include "ros2_medkit_gateway/core/providers/introspection_provider.hpp" #include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_param_beacon/parameter_client_interface.hpp" class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, @@ -46,6 +49,10 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public: ParameterBeaconPlugin() = default; ~ParameterBeaconPlugin() noexcept override; + ParameterBeaconPlugin(const ParameterBeaconPlugin &) = delete; + ParameterBeaconPlugin & operator=(const ParameterBeaconPlugin &) = delete; + ParameterBeaconPlugin(ParameterBeaconPlugin &&) = delete; + ParameterBeaconPlugin & operator=(ParameterBeaconPlugin &&) = delete; /// Constructor with injectable client factory (for testing). explicit ParameterBeaconPlugin(ros2_medkit_param_beacon::ParameterClientFactory factory) @@ -66,6 +73,9 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, ros2_medkit_beacon::BeaconHintStore & store() { return *store_; } + const rclcpp::Node::SharedPtr & param_node() const { + return param_node_; + } private: // Polling @@ -77,9 +87,17 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, // Client management std::shared_ptr get_or_create_client(const std::string & fqn); - void evict_stale_clients(); + /// Drop the client, and the backoff, of every node that is not among this cycle's targets. + void evict_stale_clients(const std::vector & targets); + /// Count one more timeout for the node and set how many cycles skip it. + void back_off(const std::string & fqn); // Config + /// Longest duration in seconds. Fast DDS keeps a wait's seconds in an int32; a longer wait spins. + static constexpr double kMaxSeconds = 2147483647.0; + /// max_hints takes 1 to kMaxHints. + static constexpr std::int64_t kMaxHints = 2147483647; + static constexpr std::size_t kDefaultMaxHints = 10000; std::string parameter_prefix_{"ros2_medkit.discovery"}; std::chrono::duration poll_interval_{5.0}; double poll_budget_sec_{10.0}; @@ -87,6 +105,8 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, // State ros2_medkit_gateway::RosPluginContext * ctx_{nullptr}; + /// The gateway node's FQN. Graph reads skip it and its helper nodes. + std::string gateway_fqn_; rclcpp::Node::SharedPtr param_node_; std::thread poll_thread_; std::atomic shutdown_requested_{false}; @@ -102,12 +122,14 @@ class ParameterBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, std::map> clients_; ros2_medkit_param_beacon::ParameterClientFactory client_factory_; - // Serialization for SyncParametersClient operations (never hold while acquiring clients_mutex_) + // Serializes parameter client calls (never hold while acquiring clients_mutex_) std::mutex param_ops_mutex_; // Node list (shared between introspect and poll threads) mutable std::shared_mutex nodes_mutex_; std::vector poll_targets_; + // Reads the graph when poll_targets_ is empty. Hides leftovers of nodes its own reads saw running. + ros2_medkit_gateway::ros2_common::GraphNodeListReader graph_node_reader_; // Backoff tracking std::unordered_map backoff_counts_; diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp index 1524e6990..624f7cac8 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/include/ros2_medkit_param_beacon/parameter_client_interface.hpp @@ -14,15 +14,21 @@ #pragma once +#include #include +#include #include #include +#include #include #include #include +#include +#include #include #include +#include namespace ros2_medkit_param_beacon { @@ -34,34 +40,103 @@ class ParameterClientInterface { virtual bool wait_for_service(std::chrono::duration timeout) = 0; + /// Throws when no answer arrives in time. virtual rcl_interfaces::msg::ListParametersResult list_parameters(const std::vector & prefixes, uint64_t depth) = 0; + /// Throws when no answer arrives in time. An answer without values gives an empty vector. virtual std::vector get_parameters(const std::vector & names) = 0; }; -/// Production implementation wrapping rclcpp::SyncParametersClient. +/// Production implementation: a client per parameter service, spun on its own executor. +/// +/// Every call waits at most `timeout`. A request that gets no answer in time is removed from its client. class RealParameterClient : public ParameterClientInterface { public: - RealParameterClient(rclcpp::Node::SharedPtr node, const std::string & target_node) - : client_(std::make_shared(node, target_node)) { + RealParameterClient(const rclcpp::Node::SharedPtr & node, const std::string & target_node, + std::chrono::duration timeout) + : node_(node->get_node_base_interface()) + , list_client_(create_client(*node, target_node + "/list_parameters")) + , get_client_(create_client(*node, target_node + "/get_parameters")) + , timeout_(std::chrono::duration_cast(timeout)) { } bool wait_for_service(std::chrono::duration timeout) override { - return client_->wait_for_service(std::chrono::duration_cast(timeout)); + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::duration_cast(timeout); + if (!list_client_->wait_for_service( + std::max(std::chrono::steady_clock::duration::zero(), deadline - std::chrono::steady_clock::now()))) { + return false; + } + return get_client_->wait_for_service( + std::max(std::chrono::steady_clock::duration::zero(), deadline - std::chrono::steady_clock::now())); } rcl_interfaces::msg::ListParametersResult list_parameters(const std::vector & prefixes, uint64_t depth) override { - return client_->list_parameters(prefixes, depth); + auto request = std::make_shared(); + request->prefixes = prefixes; + request->depth = depth; + return call(*list_client_, request)->result; } std::vector get_parameters(const std::vector & names) override { - return client_->get_parameters(names); + auto request = std::make_shared(); + request->names = names; + const auto response = call(*get_client_, request); + std::vector parameters; + // A service that cannot get one of the names answers with no values. + if (response->values.size() != names.size()) { + return parameters; + } + parameters.reserve(names.size()); + for (std::size_t i = 0; i < names.size(); ++i) { + parameters.emplace_back(names[i], rclcpp::ParameterValue(response->values[i])); + } + return parameters; + } + + /// Drops the requests still waiting for an answer and returns how many there were. + std::size_t prune_pending_requests() { + return list_client_->prune_pending_requests() + get_client_->prune_pending_requests(); } private: - std::shared_ptr client_; + template + static typename rclcpp::Client::SharedPtr create_client(rclcpp::Node & node, const std::string & name) { +#if RCLCPP_VERSION_MAJOR >= 28 + return node.create_client(name, rclcpp::ParametersQoS()); +#else + return node.create_client(name, rmw_qos_profile_parameters); +#endif + } + + template + typename ServiceT::Response::SharedPtr call(rclcpp::Client & client, + const typename ServiceT::Request::SharedPtr & request) { + auto future = client.async_send_request(request); + executor_.add_node(node_); + rclcpp::FutureReturnCode code = rclcpp::FutureReturnCode::INTERRUPTED; + try { + code = executor_.spin_until_future_complete(future, timeout_); + } catch (...) { + executor_.remove_node(node_); + client.remove_pending_request(future); + throw; + } + executor_.remove_node(node_); + if (code != rclcpp::FutureReturnCode::SUCCESS) { + client.remove_pending_request(future); + throw std::runtime_error(std::string("No answer from ") + client.get_service_name() + " in time"); + } + return future.get(); + } + + rclcpp::node_interfaces::NodeBaseInterface::SharedPtr node_; + rclcpp::Client::SharedPtr list_client_; + rclcpp::Client::SharedPtr get_client_; + std::chrono::nanoseconds timeout_; + rclcpp::executors::SingleThreadedExecutor executor_; }; /// Factory function type for creating parameter clients. diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp index a00dc5ff5..c5a47243e 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/src/param_beacon_plugin.cpp @@ -14,8 +14,16 @@ #include "ros2_medkit_param_beacon/param_beacon_plugin.hpp" +#include #include +#include +#include +#include +#include +#include +#include #include +#include #include #include @@ -35,10 +43,8 @@ using ros2_medkit_gateway::PluginContext; using ros2_medkit_gateway::SovdEntityType; ParameterBeaconPlugin::~ParameterBeaconPlugin() noexcept { - // On Lyrical (originally observed on Rolling), ~rclcpp::Node can throw - // graph_listener::NodeNotFoundError once rclcpp::shutdown() has invalidated - // the context. An exception escaping a destructor calls std::terminate(), - // so swallow it here. + // shutdown() can throw std::system_error when it locks clients_mutex_. An exception leaving this + // destructor terminates the process. try { shutdown(); } catch (...) { @@ -57,30 +63,58 @@ void ParameterBeaconPlugin::configure(const nlohmann::json & config) { auto beacon_ttl = config.value("beacon_ttl_sec", 15.0); auto beacon_expiry = config.value("beacon_expiry_sec", 300.0); - auto max_hints = static_cast(std::max(config.value("max_hints", 10000), 1)); - // Clamp to safe minimums - if (poll_interval_.count() < 0.1) { - log_warn("poll_interval_sec clamped from " + std::to_string(poll_interval_.count()) + " to 0.1"); - poll_interval_ = std::chrono::duration(0.1); - } - if (poll_budget_sec_ < 0.1) { - log_warn("poll_budget_sec clamped from " + std::to_string(poll_budget_sec_) + " to 0.1"); - poll_budget_sec_ = 0.1; - } - if (param_timeout_sec_ < 0.1) { - log_warn("param_timeout_sec clamped from " + std::to_string(param_timeout_sec_) + " to 0.1"); - param_timeout_sec_ = 0.1; - } - if (beacon_ttl < 0.1) { - log_warn("beacon_ttl_sec clamped from " + std::to_string(beacon_ttl) + " to 0.1"); - beacon_ttl = 0.1; - } - if (beacon_expiry < 1.0) { - log_warn("beacon_expiry_sec clamped from " + std::to_string(beacon_expiry) + " to 1.0"); - beacon_expiry = 1.0; - } - // max_hints already clamped to >= 1 via std::max above + // max_hints is checked as int64 before it narrows. An integer outside 1 to kMaxHints becomes the nearer + // bound. Any other value, a double included, is rejected and the default stays. + auto read_max_hints = [this, &config]() -> std::size_t { + const auto it = config.find("max_hints"); + if (it == config.end()) { + return kDefaultMaxHints; + } + if (!it->is_number_integer()) { + std::ostringstream message; + message << std::setprecision(12) << "max_hints "; + if (it->is_number_float()) { + message << it->get(); + } else { + message << it->dump(); + } + message << " is not an integer, using " << kDefaultMaxHints; + log_warn(message.str()); + return kDefaultMaxHints; + } + // Only an unsigned JSON integer can exceed int64; it is above kMaxHints either way. + constexpr auto kInt64Max = std::numeric_limits::max(); + const bool beyond_int64 = + it->is_number_unsigned() && it->get() > static_cast(kInt64Max); + const std::int64_t value = beyond_int64 ? kInt64Max : it->get(); + const std::int64_t clamped = std::clamp(value, 1, kMaxHints); + if (clamped != value) { + log_warn("max_hints clamped from " + it->dump() + " to " + std::to_string(clamped)); + } + return static_cast(clamped); + }; + const std::size_t max_hints = read_max_hints(); + + // A duration above kMaxSeconds, +inf included, becomes kMaxSeconds. NaN and a duration below the + // minimum become the minimum. + auto clamp_seconds = [this](const char * key, double value, double minimum) { + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite: it lets NaN through. + // NOLINTNEXTLINE(readability-simplify-boolean-expr) + if (!(std::isfinite(value) && value >= minimum && value <= kMaxSeconds)) { + const double clamped = value > kMaxSeconds ? kMaxSeconds : minimum; + std::ostringstream message; + message << std::setprecision(12) << key << " clamped from " << value << " to " << clamped; + log_warn(message.str()); + return clamped; + } + return value; + }; + poll_interval_ = std::chrono::duration(clamp_seconds("poll_interval_sec", poll_interval_.count(), 0.1)); + poll_budget_sec_ = clamp_seconds("poll_budget_sec", poll_budget_sec_, 0.1); + param_timeout_sec_ = clamp_seconds("param_timeout_sec", param_timeout_sec_, 0.1); + beacon_ttl = clamp_seconds("beacon_ttl_sec", beacon_ttl, 0.1); + beacon_expiry = clamp_seconds("beacon_expiry_sec", beacon_expiry, 1.0); // Config validation if (beacon_ttl <= poll_interval_.count()) { @@ -110,6 +144,7 @@ void ParameterBeaconPlugin::configure(const nlohmann::json & config) { void ParameterBeaconPlugin::set_context(PluginContext & context) { ctx_ = as_ros_plugin_context(context); + gateway_fqn_ = ctx_->node()->get_fully_qualified_name(); if (!store_) { store_ = std::make_unique(); @@ -121,12 +156,16 @@ void ParameterBeaconPlugin::set_context(PluginContext & context) { options.start_parameter_event_publisher(false); options.use_global_arguments(false); param_node_ = std::make_shared("_param_beacon_node", options); + // Join rclcpp's graph listener now. wait_for_service() would join it on first use, and a join + // after rclcpp shuts down fails half-way, so ~NodeGraph later throws and terminates the process. + param_node_->get_node_graph_interface()->get_graph_event(); // Set default client factory if not injected (tests inject mock factory) if (!client_factory_) { auto node = param_node_; - client_factory_ = [node](const std::string & target) { - return std::make_shared(node, target); + const std::chrono::duration timeout(param_timeout_sec_); + client_factory_ = [node, timeout](const std::string & target) { + return std::make_shared(node, target, timeout); }; } @@ -158,14 +197,8 @@ void ParameterBeaconPlugin::shutdown() { backoff_counts_.clear(); skip_remaining_.clear(); } - // ~rclcpp::Node can throw graph_listener::NodeNotFoundError on Lyrical - // (and Rolling) when the context was already torn down by rclcpp::shutdown(). Swallow - // it so the plugin_manager shutdown sequence (and the plugin destructor - // that calls back into us) does not abort the process. - try { - param_node_.reset(); - } catch (...) { - } + // With the default client factory, which holds a copy of the node, ~Node runs when client_factory_ is destroyed. + param_node_.reset(); } std::vector ParameterBeaconPlugin::get_routes() { @@ -256,33 +289,34 @@ void ParameterBeaconPlugin::poll_cycle() { // "rcl node's context is invalid" if the poll timer fires between // SIGINT handling and the executor stopping; swallow it so the // shutdown path isn't aborted by std::terminate. + // The reader leaves out the leftovers of nodes it saw running (see GraphNodeListReader). std::vector> names_and_ns; try { - names_and_ns = param_node_->get_node_graph_interface()->get_node_names_and_namespaces(); + names_and_ns = graph_node_reader_.read(*param_node_->get_node_graph_interface()).nodes; } catch (const std::runtime_error & ex) { - RCLCPP_DEBUG(param_node_->get_logger(), "get_node_names_and_namespaces threw during shutdown: %s", ex.what()); + RCLCPP_DEBUG(param_node_->get_logger(), "Reading the node list threw during shutdown: %s", ex.what()); return; } for (const auto & [name, ns] : names_and_ns) { - // Skip internal nodes (leading underscore) and the gateway - if (name.empty() || name[0] == '_' || name == "ros2_medkit_gateway") { + // Skip hidden nodes (leading underscore), the gateway and its helper nodes: they carry no beacon. + if (name.empty() || name[0] == '_') { + continue; + } + auto fqn = ros2_medkit_gateway::ros2_common::graph_node_fqn(name, ns); + if (fqn == gateway_fqn_ || ros2_medkit_gateway::ros2_common::is_own_gateway_helper_node(fqn, gateway_fqn_)) { continue; } - auto fqn = (ns == "/" ? "/" : ns + "/") + name; - targets.push_back(fqn); + targets.push_back(std::move(fqn)); } } + evict_stale_clients(targets); if (targets.empty()) { return; } - // Evict stale clients - evict_stale_clients(); - auto cycle_start = std::chrono::steady_clock::now(); auto n = targets.size(); - size_t polled = 0; for (size_t i = 0; i < n; ++i) { // Budget check @@ -302,7 +336,6 @@ void ParameterBeaconPlugin::poll_cycle() { } poll_node(fqn); - ++polled; } start_offset_ = (start_offset_ + 1) % n; @@ -318,39 +351,30 @@ void ParameterBeaconPlugin::poll_node(const std::string & fqn) { std::lock_guard ops_lock(param_ops_mutex_); if (!client->wait_for_service(std::chrono::duration(param_timeout_sec_))) { - // Timeout - apply backoff - auto & count = backoff_counts_[fqn]; - if (count < 100) { - ++count; // cap to prevent overflow - } - int skip = std::min(1 << std::min(count - 1, 3), 8); - skip_remaining_[fqn] = skip; + back_off(fqn); return; } - - // List parameters under prefix auto list_result = client->list_parameters({parameter_prefix_}, 0); - if (list_result.names.empty()) { - return; // No beacon parameters declared + std::vector params; + if (!list_result.names.empty()) { + params = client->get_parameters(list_result.names); } - // Fetch parameter values - auto params = client->get_parameters(list_result.names); + // The node answered, so it is not backed off. An answer without values stores no hint. + backoff_counts_.erase(fqn); + skip_remaining_.erase(fqn); - // Convert to BeaconHint auto hint = parse_parameters(fqn, params); if (hint.entity_id.empty()) { - return; // No entity_id parameter - skip + return; } - // Validate auto result = validate_beacon_hint(hint, limits_); if (!result.valid) { log_warn("Beacon hint rejected for '" + hint.entity_id + "': " + result.reason); return; } - // Store if (!store_->update(hint)) { if (!capacity_warned_) { log_warn("BeaconHintStore capacity reached (max_hints=" + std::to_string(store_->size()) + @@ -358,20 +382,18 @@ void ParameterBeaconPlugin::poll_node(const std::string & fqn) { capacity_warned_ = true; } } + } catch (const std::exception &) { + // No answer in time, or the node disappeared. + back_off(fqn); + } +} - // Reset backoff on success - backoff_counts_.erase(fqn); - skip_remaining_.erase(fqn); - - } catch (const std::exception & e) { - // Node disappeared or service error - apply backoff - auto & count = backoff_counts_[fqn]; - if (count < 100) { - ++count; // cap to prevent overflow - } - int skip = std::min(1 << std::min(count - 1, 3), 8); - skip_remaining_[fqn] = skip; +void ParameterBeaconPlugin::back_off(const std::string & fqn) { + auto & count = backoff_counts_[fqn]; + if (count < 100) { + ++count; // cap to prevent overflow } + skip_remaining_[fqn] = std::min(1 << std::min(count - 1, 3), 8); } BeaconHint ParameterBeaconPlugin::parse_parameters(const std::string & /*fqn*/, @@ -382,7 +404,7 @@ BeaconHint ParameterBeaconPlugin::parse_parameters(const std::string & /*fqn*/, std::string metadata_prefix = parameter_prefix_ + ".metadata."; for (const auto & param : params) { - auto param_name = param.get_name(); + const auto & param_name = param.get_name(); // Strip prefix to get the field name if (param_name.rfind(parameter_prefix_ + ".", 0) != 0) { @@ -444,19 +466,12 @@ ParameterBeaconPlugin::get_or_create_client(const std::string & fqn) { return client; } -void ParameterBeaconPlugin::evict_stale_clients() { - std::shared_lock nodes_lock(nodes_mutex_); +void ParameterBeaconPlugin::evict_stale_clients(const std::vector & targets) { + const std::unordered_set current(targets.begin(), targets.end()); std::lock_guard clients_lock(clients_mutex_); for (auto it = clients_.begin(); it != clients_.end();) { - bool found = false; - for (const auto & target : poll_targets_) { - if (target == it->first) { - found = true; - break; - } - } - if (!found) { + if (current.count(it->first) == 0) { backoff_counts_.erase(it->first); skip_remaining_.erase(it->first); it = clients_.erase(it); diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp index cf9503f4f..e826ad5b7 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_param_beacon/test/test_param_beacon_plugin.cpp @@ -14,14 +14,29 @@ #include #include +#include +#include +#include #include +#include +#include +#include +#include +#include #include +#include #include #include #include #include +#include +#include +#include +#include +#include +#include #include #include "ros2_medkit_gateway/core/plugins/plugin_http_types.hpp" @@ -40,6 +55,7 @@ using ros2_medkit_gateway::PluginResponse; using ros2_medkit_gateway::RosPluginContext; using ros2_medkit_gateway::SovdEntityType; using ros2_medkit_param_beacon::ParameterClientInterface; +using ros2_medkit_param_beacon::RealParameterClient; using ::testing::_; using ::testing::Return; @@ -93,39 +109,41 @@ class MockPluginContext : public RosPluginContext { rclcpp::Node * node() const override { return node_; } - std::optional get_entity(const std::string &) const override { + std::optional get_entity(const std::string & /*entity_id*/) const override { return std::nullopt; } - std::vector get_child_apps(const std::string &) const override { + std::vector get_child_apps(const std::string & /*component_id*/) const override { return {}; } - nlohmann::json list_entity_faults(const std::string &) const override { + nlohmann::json list_entity_faults(const std::string & /*entity_id*/) const override { return nlohmann::json::array(); } - std::optional validate_entity_for_route(const PluginRequest &, PluginResponse &, - const std::string &) const override { + std::optional validate_entity_for_route(const PluginRequest & /*req*/, PluginResponse & /*res*/, + const std::string & /*entity_id*/) const override { return std::nullopt; } void register_capability(SovdEntityType type, const std::string & name) override { registered_capabilities_.push_back({type, name}); } - void register_entity_capability(const std::string &, const std::string &) override { + void register_entity_capability(const std::string & /*entity_id*/, const std::string & /*name*/) override { } - std::vector get_type_capabilities(SovdEntityType) const override { + std::vector get_type_capabilities(SovdEntityType /*type*/) const override { return {}; } - std::vector get_entity_capabilities(const std::string &) const override { + std::vector get_entity_capabilities(const std::string & /*entity_id*/) const override { return {}; } - ros2_medkit_gateway::LockAccessResult check_lock(const std::string &, const std::string &, - const std::string &) const override { + ros2_medkit_gateway::LockAccessResult check_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::string & /*collection*/) const override { return ros2_medkit_gateway::LockAccessResult{true, "", "", ""}; } tl::expected - acquire_lock(const std::string &, const std::string &, const std::vector &, int) override { + acquire_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::vector & /*scopes*/, int /*expiration_seconds*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } - tl::expected release_lock(const std::string &, const std::string &) override { + tl::expected release_lock(const std::string & /*entity_id*/, + const std::string & /*client_id*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } ros2_medkit_gateway::ResourceChangeNotifier * get_resource_change_notifier() override { @@ -144,6 +162,89 @@ class MockPluginContext : public RosPluginContext { rclcpp::Node * node_; }; +/// How a ParameterServiceStub answers get requests. +enum class GetAnswer { kNone, kValue, kNoValues }; + +/// A node with parameter services that answers only as told, on its own executor. Counts list and get requests. +class ParameterServiceStub { + public: + ParameterServiceStub(const std::string & name, bool answer_list, GetAnswer answer_get = GetAnswer::kNone) { + node_ = std::make_shared(name, rclcpp::NodeOptions().start_parameter_services(false)); + using rcl_interfaces::srv::GetParameters; + using rcl_interfaces::srv::ListParameters; + list_ = node_->create_service( + "~/list_parameters", [this, answer_list](const std::shared_ptr & header, + const std::shared_ptr &) { + ++list_requests_; + if (answer_list) { + ListParameters::Response response; + response.result.names = {"ros2_medkit.discovery.entity_id"}; + list_->send_response(*header, response); + } + }); + get_ = node_->create_service( + "~/get_parameters", [this, name, answer_get](const std::shared_ptr & header, + const std::shared_ptr &) { + ++get_requests_; + if (answer_get != GetAnswer::kNone) { + GetParameters::Response response; + if (answer_get == GetAnswer::kValue) { + response.values.push_back(rclcpp::ParameterValue(name).to_value_msg()); + } + get_->send_response(*header, response); + } + }); + types_ = silent("~/get_parameter_types"); + set_ = silent("~/set_parameters"); + atomically_ = silent("~/set_parameters_atomically"); + describe_ = silent("~/describe_parameters"); + executor_.add_node(node_); + spin_thread_ = std::thread([this]() { + executor_.spin(); + }); + } + + ~ParameterServiceStub() { + executor_.cancel(); + spin_thread_.join(); + executor_.remove_node(node_); + } + + ParameterServiceStub(const ParameterServiceStub &) = delete; + ParameterServiceStub & operator=(const ParameterServiceStub &) = delete; + ParameterServiceStub(ParameterServiceStub &&) = delete; + ParameterServiceStub & operator=(ParameterServiceStub &&) = delete; + + std::string fqn() const { + return node_->get_fully_qualified_name(); + } + int list_requests() const { + return list_requests_.load(); + } + int get_requests() const { + return get_requests_.load(); + } + + private: + template + std::shared_ptr silent(const std::string & name) { + return node_->create_service( + name, [](const std::shared_ptr &, const std::shared_ptr &) {}); + } + + rclcpp::Node::SharedPtr node_; + rclcpp::Service::SharedPtr list_; + rclcpp::Service::SharedPtr get_; + std::shared_ptr types_; + std::shared_ptr set_; + std::shared_ptr atomically_; + std::shared_ptr describe_; + rclcpp::executors::SingleThreadedExecutor executor_; + std::thread spin_thread_; + std::atomic list_requests_{0}; + std::atomic get_requests_{0}; +}; + // --- Test Fixture --- class ParamBeaconPluginTest : public ::testing::Test { @@ -467,3 +568,293 @@ TEST_F(ParamBeaconPluginTest, PollCycleAfterShutdownIsNoop) { std::this_thread::sleep_for(std::chrono::milliseconds(500)); EXPECT_EQ(plugin_->store().size(), store_size); } + +// A get request that gets no answer in time throws, like a list request. +TEST_F(ParamBeaconPluginTest, RealClientThrowsWhenGetParametersGetsNoAnswer) { + ParameterServiceStub stub("get_never_answers", true); + RealParameterClient client(node_, stub.fqn(), std::chrono::duration(0.2)); + ASSERT_TRUE(client.wait_for_service(std::chrono::seconds(10))); + const auto names = client.list_parameters({"ros2_medkit.discovery"}, 0).names; + ASSERT_EQ(names, (std::vector{"ros2_medkit.discovery.entity_id"})); + + const auto started = std::chrono::steady_clock::now(); + EXPECT_THROW(client.get_parameters(names), std::runtime_error); + EXPECT_GE(std::chrono::steady_clock::now() - started, std::chrono::milliseconds(200)); + EXPECT_EQ(stub.get_requests(), 1); +} + +// A request that gets no answer in time leaves nothing pending in its client. +TEST_F(ParamBeaconPluginTest, RealClientLeavesNoRequestPendingAfterTimeouts) { + ParameterServiceStub list_silent("pending_list_silent", false); + ParameterServiceStub get_silent("pending_get_silent", true); + RealParameterClient list_client(node_, list_silent.fqn(), std::chrono::duration(0.005)); + RealParameterClient get_client(node_, get_silent.fqn(), std::chrono::duration(0.005)); + ASSERT_TRUE(list_client.wait_for_service(std::chrono::seconds(10))); + ASSERT_TRUE(get_client.wait_for_service(std::chrono::seconds(10))); + + constexpr int kCalls = 50; + int timeouts = 0; + for (int i = 0; i < kCalls; ++i) { + try { + list_client.list_parameters({"ros2_medkit.discovery"}, 0); + } catch (const std::runtime_error &) { + ++timeouts; + } + try { + get_client.get_parameters({"ros2_medkit.discovery.entity_id"}); + } catch (const std::runtime_error &) { + ++timeouts; + } + } + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while ((list_silent.list_requests() < kCalls || get_silent.get_requests() < kCalls) && + std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + ASSERT_EQ(list_silent.list_requests(), kCalls); + ASSERT_EQ(get_silent.get_requests(), kCalls); + EXPECT_EQ(timeouts, 2 * kCalls); + EXPECT_EQ(list_client.prune_pending_requests(), 0U); + EXPECT_EQ(get_client.prune_pending_requests(), 0U); +} + +// A get answer without values is an answer: no exception, no parameters, nothing pending. +TEST_F(ParamBeaconPluginTest, RealClientReturnsNoParametersForAnAnswerWithoutValues) { + ParameterServiceStub stub("get_answers_no_values", true, GetAnswer::kNoValues); + RealParameterClient client(node_, stub.fqn(), std::chrono::duration(10.0)); + ASSERT_TRUE(client.wait_for_service(std::chrono::seconds(10))); + + std::vector parameters{rclcpp::Parameter("placeholder", 1)}; + EXPECT_NO_THROW(parameters = client.get_parameters({"ros2_medkit.discovery.entity_id"})); + EXPECT_TRUE(parameters.empty()); + EXPECT_EQ(stub.get_requests(), 1); + EXPECT_EQ(client.prune_pending_requests(), 0U); +} + +// A wait with no time left returns within 1 s, also when the service does not exist. +TEST_F(ParamBeaconPluginTest, RealClientWaitWithNoTimeLeftReturns) { + ::testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_EXIT( + { + // A wait that never returns ends the child with SIGALRM. + alarm(10); + RealParameterClient client(node_, "/param_beacon_absent_node", std::chrono::duration(1.0)); + const auto start = std::chrono::steady_clock::now(); + const bool found = client.wait_for_service(std::chrono::duration(0.0)); + const auto waited_ms = + std::chrono::duration_cast(std::chrono::steady_clock::now() - start).count(); + // libtsan intercepts _exit, so a race reported in the child sets its exit code. + if (found) { + _exit(1); + } + if (waited_ms >= 1000) { + std::fprintf(stderr, "the wait took %lld ms\n", static_cast(waited_ms)); + _exit(2); + } + _exit(0); + }, + ::testing::ExitedWithCode(0), ""); +} + +// In a graph read, a node whose get or list requests time out is backed off; one that answers is +// polled every cycle. +TEST_F(ParamBeaconPluginTest, RuntimeTargetsWhoseParameterRequestsTimeOutAreBackedOff) { + ParameterServiceStub healthy("beacon_healthy", true, GetAnswer::kValue); + ParameterServiceStub get_silent("beacon_get_silent", true); + ParameterServiceStub list_silent("beacon_list_silent", false); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 0.1; + config["poll_budget_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + // Twelve cycles: a node that times out on every poll is asked on cycles 1, 3, 6 and 11. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 12 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + const int cycles = healthy.list_requests(); + ASSERT_GE(cycles, 12) << "the plugin never polled the node that answers"; + EXPECT_GE(get_silent.get_requests(), 3); + EXPECT_LE(get_silent.get_requests(), 5) << "a node whose get requests time out was asked on " + << get_silent.get_requests() << " of " << cycles << " cycles"; + EXPECT_GE(list_silent.list_requests(), 3); + EXPECT_LE(list_silent.list_requests(), 5) << "a node whose list requests time out was asked on " + << list_silent.list_requests() << " of " << cycles << " cycles"; +} + +// NaN, -inf and values below the minimum clamp to the minimum, +inf to kMaxSeconds. Polling keeps +// its 0.1 s timeout. +TEST_F(ParamBeaconPluginTest, OutOfRangeDurationsAreClampedAndPollingKeepsItsTimeout) { + ParameterServiceStub get_silent("beacon_nonfinite_get_silent", true); + ParameterServiceStub list_silent("beacon_nonfinite_list_silent", false); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = std::numeric_limits::quiet_NaN(); + config["param_timeout_sec"] = -1.0; + config["poll_budget_sec"] = std::numeric_limits::infinity(); + config["beacon_ttl_sec"] = -std::numeric_limits::infinity(); + config["beacon_expiry_sec"] = std::numeric_limits::quiet_NaN(); + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (list_silent.list_requests() < 3 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + EXPECT_GE(list_silent.list_requests(), 3); + EXPECT_GE(get_silent.get_requests(), 1); +} + +// A get answer without values, as rclpy gives for a typed parameter with no value, causes no backoff +// and stores no hint. +TEST_F(ParamBeaconPluginTest, NodeAnsweringWithoutValuesIsPolledEveryCycle) { + ParameterServiceStub healthy("no_values_healthy", true, GetAnswer::kValue); + ParameterServiceStub no_values("no_values_answer", true, GetAnswer::kNoValues); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + config["poll_budget_sec"] = 20.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 12 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + const int cycles = healthy.list_requests(); + ASSERT_GE(cycles, 12) << "the plugin never polled the node that answers"; + EXPECT_GE(no_values.get_requests(), cycles - 1) + << "a node answering without values was asked on " << no_values.get_requests() << " of " << cycles << " cycles"; + EXPECT_TRUE(plugin_->store().get("no_values_healthy").has_value()); + EXPECT_EQ(plugin_->store().size(), 1U); +} + +// Timed-out polls leave nothing pending, and each node keeps one client across cycles. +TEST_F(ParamBeaconPluginTest, TimedOutPollsLeaveNothingPendingAndKeepOneClientPerNode) { + ParameterServiceStub get_silent("pending_poll_get_silent", true); + ParameterServiceStub list_silent("pending_poll_list_silent", false); + auto client_node = std::make_shared("_pending_poll_clients"); + std::mutex created_mutex; + std::map>> created; + + plugin_ = std::make_unique([&](const std::string & target) { + auto client = std::make_shared(client_node, target, std::chrono::duration(0.05)); + std::lock_guard lock(created_mutex); + created[target].push_back(client); + return client; + }); + nlohmann::json config; + config["poll_interval_sec"] = 0.05; + config["param_timeout_sec"] = 1.0; + config["poll_budget_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + // Four timeouts each: cycles 1, 3, 6 and 11 of the backoff. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while ((get_silent.get_requests() < 4 || list_silent.list_requests() < 4) && + std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + ASSERT_GE(get_silent.get_requests(), 4); + ASSERT_GE(list_silent.list_requests(), 4); + + std::lock_guard lock(created_mutex); + std::size_t pending = 0; + for (const auto & [target, clients] : created) { + for (const auto & client : clients) { + pending += client->prune_pending_requests(); + } + } + EXPECT_EQ(pending, 0U); + EXPECT_EQ(created[get_silent.fqn()].size(), 1U); + EXPECT_EQ(created[list_silent.fqn()].size(), 1U); +} + +// A graph read that finds no target drops every client. +TEST_F(ParamBeaconPluginTest, EmptyTargetListDropsEveryClient) { + auto stub = std::make_unique("dropped_when_gone", true, GetAnswer::kValue); + const std::string request_topic = "rq" + stub->fqn() + "/list_parametersRequest"; + auto clients = [&]() { + const auto infos = node_->get_publishers_info_by_topic(request_topic, true); + return std::count_if(infos.begin(), infos.end(), [](const rclcpp::TopicEndpointInfo & info) { + return info.node_name() == "_param_beacon_node"; + }); + }; + auto wait_clients = [&](std::ptrdiff_t expected) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(30); + while (clients() != expected && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + return clients(); + }; + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + ASSERT_EQ(wait_clients(1), 1) << "the plugin never created a client for " << stub->fqn(); + stub.reset(); + EXPECT_EQ(wait_clients(0), 0) << "the client stayed after the graph listed no target"; +} + +// A graph read skips the gateway's helper nodes. +TEST_F(ParamBeaconPluginTest, GatewayHelperNodesAreNotPolled) { + ParameterServiceStub healthy("helper_skip_healthy", true, GetAnswer::kValue); + ParameterServiceStub fault_clients("test_param_beacon_node_fault_clients", true, GetAnswer::kValue); + ParameterServiceStub state_reader("test_param_beacon_node_lifecycle_state_reader", true, GetAnswer::kValue); + + plugin_ = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 0.1; + config["param_timeout_sec"] = 5.0; + plugin_->configure(config); + plugin_->set_context(*mock_ctx_); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(60); + while (healthy.list_requests() < 5 && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + plugin_->shutdown(); + ASSERT_GE(healthy.list_requests(), 5); + EXPECT_EQ(fault_clients.list_requests(), 0); + EXPECT_EQ(state_reader.list_requests(), 0); +} + +// The plugin's node joins the graph listener in set_context(). A first join after rclcpp shuts down +// fails half-way, and destroying the node then terminates the process. +TEST_F(ParamBeaconPluginTest, NodeDestroyedAfterShutdownDoesNotTerminate) { + ::testing::FLAGS_gtest_death_test_style = "threadsafe"; + EXPECT_EXIT( + { + auto plugin = std::make_unique(); + nlohmann::json config; + config["poll_interval_sec"] = 1000.0; + plugin->configure(config); + plugin->set_context(*mock_ctx_); + // Starts the graph listener, so rclcpp::shutdown() shuts it down. + node_->get_node_graph_interface()->get_graph_event(); + rclcpp::shutdown(); + try { + plugin->param_node()->get_node_graph_interface()->get_graph_event(); + } catch (const std::exception &) { + } + plugin.reset(); + std::_Exit(0); + }, + ::testing::ExitedWithCode(0), ""); +} diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md index c3fcb99e8..3ca8b10d9 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/README.md @@ -14,6 +14,17 @@ to enrich the SOVD entity tree. Hints transition through states: **active** (within TTL) -> **stale** (TTL expired, data still served with stale marker) -> **expired** (removed from store). +`beacon_ttl_sec` takes 0.1 to 2147483647 s, `beacon_expiry_sec` 1.0 to 2147483647 s and +`max_messages_per_second` 1 to 10000. A value above its maximum, including `.inf` in a +parameter file, becomes the maximum; NaN, `-.inf` and a value below the minimum become the +minimum. Each replacement is logged as a warning. Messages over the rate limit are dropped +without a log. + +`max_hints` takes an integer from 1 to 2147483647. A 64-bit integer outside that range becomes +the nearer bound. The parameter parser reads an integer that does not fit in 64 bits as a +double. A double, such as `1.0e12` or `.nan`, is refused and the default 10000 is used. Both +are logged as warnings. + ## Configuration ```yaml diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst index ccc797ef2..11bd54129 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/design/index.rst @@ -42,10 +42,17 @@ publish beacons. Rate Limiting ~~~~~~~~~~~~~ -A ``TokenBucket`` rate limiter (default 100 messages/second) protects the gateway -from beacon floods. The bucket refills continuously and drops excess messages with -a single log warning. The rate limiter is thread-safe, as the DDS callback may fire -from any executor thread. +A ``TokenBucket`` rate limiter (default 100 messages/second, at most 10000) protects the +gateway from beacon floods. The bucket refills continuously and drops excess messages +without a log. The rate limiter is thread-safe, as the DDS callback may fire from any +executor thread. + +``configure()`` replaces a value above its maximum, ``+inf`` included, by the maximum, and +NaN or a value below the minimum by the minimum, with a warning. The check is the in-range +test negated, so NaN fails it. ``max_hints`` is read as an int64 before it is narrowed; a +64-bit integer outside 1 to 2147483647 becomes the nearer bound, and a value that is not an +integer keeps the default, each with a warning. The parameter parser reads an integer that +does not fit in 64 bits as a double, so it keeps the default. Timestamp Back-Projection ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp index e5753100d..de63d6f87 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/include/ros2_medkit_topic_beacon/topic_beacon_plugin.hpp @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include #include #include @@ -77,11 +79,17 @@ class TokenBucket { class TopicBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public ros2_medkit_gateway::IntrospectionProvider { public: + TopicBeaconPlugin() = default; + ~TopicBeaconPlugin() noexcept override; + TopicBeaconPlugin(const TopicBeaconPlugin &) = delete; + TopicBeaconPlugin & operator=(const TopicBeaconPlugin &) = delete; + TopicBeaconPlugin(TopicBeaconPlugin &&) = delete; + TopicBeaconPlugin & operator=(TopicBeaconPlugin &&) = delete; + std::string name() const override; void configure(const nlohmann::json & config) override; void set_context(ros2_medkit_gateway::PluginContext & context) override; void shutdown() override; - ~TopicBeaconPlugin() noexcept override; std::vector get_routes() override; ros2_medkit_gateway::IntrospectionResult introspect(const ros2_medkit_gateway::IntrospectionInput & input) override; @@ -94,7 +102,15 @@ class TopicBeaconPlugin : public ros2_medkit_gateway::GatewayPlugin, public ros2 } private: - void on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr & msg); + void on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg); + + /// Longest TTL and expiry in seconds, the same bound as parameter_beacon's durations. + static constexpr double kMaxSeconds = 2147483647.0; + /// Highest rate limit: one beacon per second for each of the store's default 10000 hints. + static constexpr double kMaxMessagesPerSecond = 10000.0; + /// max_hints takes 1 to kMaxHints. + static constexpr std::int64_t kMaxHints = 2147483647; + static constexpr std::size_t kDefaultMaxHints = 10000; std::string topic_{"/ros2_medkit/discovery"}; ros2_medkit_gateway::RosPluginContext * ctx_{nullptr}; diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp index c133cb9c4..75014c444 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/src/topic_beacon_plugin.cpp @@ -16,6 +16,12 @@ #include #include +#include +#include +#include +#include +#include +#include #include "ros2_medkit_beacon_common/beacon_response_builder.hpp" @@ -33,14 +39,7 @@ using ros2_medkit_gateway::PluginContext; using ros2_medkit_gateway::SovdEntityType; TopicBeaconPlugin::~TopicBeaconPlugin() noexcept { - // On Lyrical (originally observed on Rolling), ~rclcpp::Subscription can throw - // graph_listener::NodeNotFoundError once rclcpp::shutdown() has invalidated - // the context. An exception escaping a destructor calls std::terminate(), - // so swallow it here. - try { - shutdown(); - } catch (...) { - } + shutdown(); } std::string TopicBeaconPlugin::name() const { @@ -53,23 +52,57 @@ void TopicBeaconPlugin::configure(const nlohmann::json & config) { BeaconHintStore::Config store_config; auto beacon_ttl = config.value("beacon_ttl_sec", 10.0); auto beacon_expiry = config.value("beacon_expiry_sec", 300.0); - auto max_hints = static_cast(std::max(config.value("max_hints", 10000), 1)); auto max_mps = config.value("max_messages_per_second", 100.0); - // Clamp to safe minimums - if (beacon_ttl < 0.1) { - log_warn("beacon_ttl_sec clamped from " + std::to_string(beacon_ttl) + " to 0.1"); - beacon_ttl = 0.1; - } - if (beacon_expiry < 1.0) { - log_warn("beacon_expiry_sec clamped from " + std::to_string(beacon_expiry) + " to 1.0"); - beacon_expiry = 1.0; - } - // max_hints already clamped to >= 1 via std::max above - if (max_mps < 1.0) { - log_warn("max_messages_per_second clamped from " + std::to_string(max_mps) + " to 1.0"); - max_mps = 1.0; - } + // max_hints is checked as int64 before it narrows. An integer outside 1 to kMaxHints becomes the nearer + // bound. Any other value, a double included, is rejected and the default stays. + auto read_max_hints = [this, &config]() -> std::size_t { + const auto it = config.find("max_hints"); + if (it == config.end()) { + return kDefaultMaxHints; + } + if (!it->is_number_integer()) { + std::ostringstream message; + message << std::setprecision(12) << "max_hints "; + if (it->is_number_float()) { + message << it->get(); + } else { + message << it->dump(); + } + message << " is not an integer, using " << kDefaultMaxHints; + log_warn(message.str()); + return kDefaultMaxHints; + } + // Only an unsigned JSON integer can exceed int64; it is above kMaxHints either way. + constexpr auto kInt64Max = std::numeric_limits::max(); + const bool beyond_int64 = + it->is_number_unsigned() && it->get() > static_cast(kInt64Max); + const std::int64_t value = beyond_int64 ? kInt64Max : it->get(); + const std::int64_t clamped = std::clamp(value, 1, kMaxHints); + if (clamped != value) { + log_warn("max_hints clamped from " + it->dump() + " to " + std::to_string(clamped)); + } + return static_cast(clamped); + }; + const std::size_t max_hints = read_max_hints(); + + // A value above its maximum, +inf included, becomes the maximum. NaN and a value below the minimum + // become the minimum. + auto clamp = [this](const char * key, double value, double minimum, double maximum) { + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite: it lets NaN through. + // NOLINTNEXTLINE(readability-simplify-boolean-expr) + if (!(std::isfinite(value) && value >= minimum && value <= maximum)) { + const double clamped = value > maximum ? maximum : minimum; + std::ostringstream message; + message << std::setprecision(12) << key << " clamped from " << value << " to " << clamped; + log_warn(message.str()); + return clamped; + } + return value; + }; + beacon_ttl = clamp("beacon_ttl_sec", beacon_ttl, 0.1, kMaxSeconds); + beacon_expiry = clamp("beacon_expiry_sec", beacon_expiry, 1.0, kMaxSeconds); + max_mps = clamp("max_messages_per_second", max_mps, 1.0, kMaxMessagesPerSecond); store_config.beacon_ttl_sec = beacon_ttl; store_config.beacon_expiry_sec = beacon_expiry; @@ -99,7 +132,8 @@ void TopicBeaconPlugin::set_context(PluginContext & context) { // Create subscription on configured topic subscription_ = node->create_subscription( - topic_, rclcpp::QoS(100).reliable(), [this](const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr msg) { + topic_, rclcpp::QoS(100).reliable(), + [this](const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg) { on_beacon(msg); }); @@ -114,13 +148,8 @@ void TopicBeaconPlugin::shutdown() { if (shutdown_requested_.exchange(true)) { return; } - // ~rclcpp::Subscription can throw on Lyrical (and Rolling) when the rclcpp - // context was torn down before us; swallow so plugin_manager shutdown and - // the plugin destructor calling back into us do not abort the process. - try { - subscription_.reset(); - } catch (...) { - } + // The callback captures this. A callback the executor already took returns early in on_beacon(). + subscription_.reset(); } std::vector TopicBeaconPlugin::get_routes() { @@ -172,7 +201,7 @@ IntrospectionResult TopicBeaconPlugin::introspect(const IntrospectionInput & inp return result; } -void TopicBeaconPlugin::on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::SharedPtr & msg) { +void TopicBeaconPlugin::on_beacon(const ros2_medkit_msgs::msg::MedkitDiscoveryHint::ConstSharedPtr & msg) { if (shutdown_requested_.load()) { return; } diff --git a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp index 0df41cabb..47c735085 100644 --- a/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp +++ b/src/ros2_medkit_discovery_plugins/ros2_medkit_topic_beacon/test/test_topic_beacon_plugin.cpp @@ -120,15 +120,17 @@ class MockPluginContext : public RosPluginContext { ros2_medkit_gateway::ConditionRegistry * get_condition_registry() override { return nullptr; } - ros2_medkit_gateway::LockAccessResult check_lock(const std::string &, const std::string &, - const std::string &) const override { + ros2_medkit_gateway::LockAccessResult check_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::string & /*collection*/) const override { return ros2_medkit_gateway::LockAccessResult{true, "", "", ""}; } tl::expected - acquire_lock(const std::string &, const std::string &, const std::vector &, int) override { + acquire_lock(const std::string & /*entity_id*/, const std::string & /*client_id*/, + const std::vector & /*scopes*/, int /*expiration_seconds*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } - tl::expected release_lock(const std::string &, const std::string &) override { + tl::expected release_lock(const std::string & /*entity_id*/, + const std::string & /*client_id*/) override { return tl::make_unexpected(ros2_medkit_gateway::LockError{"lock-disabled", "Not available", 503, std::nullopt}); } diff --git a/src/ros2_medkit_gateway/CMakeLists.txt b/src/ros2_medkit_gateway/CMakeLists.txt index 23b20ac4d..6a7c0b481 100644 --- a/src/ros2_medkit_gateway/CMakeLists.txt +++ b/src/ros2_medkit_gateway/CMakeLists.txt @@ -728,6 +728,13 @@ if(BUILD_TESTING) target_link_libraries(test_refresh_debounce gateway_core) endif() + # Node-list reader decisions (header-only), plus one case against a real rcl node + medkit_add_gtest(test_graph_node_list test/test_graph_node_list.cpp) + if(TARGET test_graph_node_list) + medkit_target_dependencies(test_graph_node_list rclcpp) + target_include_directories(test_graph_node_list PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include) + endif() + # SlotStore object pool (pure C++17, no ROS node) medkit_add_gtest(test_slot_store test/test_slot_store.cpp) if(TARGET test_slot_store) diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 5c0f55990..c475170ff 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -7,7 +7,7 @@ HTTP gateway node for the ros2_medkit diagnostics system. The ROS 2 Medkit Gateway exposes ROS 2 system information and data through a RESTful HTTP API. It automatically discovers nodes in the ROS 2 system, organizes them into a SOVD-aligned entity hierarchy (Areas, Components, Apps, Functions), and provides endpoints to query and interact with them. **Key Features:** -- **Auto-discovery**: Automatically detects ROS 2 nodes and topics +- **Auto-discovery**: Automatically detects ROS 2 nodes and topics. A node the gateway saw running that the ROS graph still lists after its participant left, with no endpoints, is not exposed, and a warning is logged when that starts (see "How long a departed node keeps being listed" in `docs/config/server.rst`) - **SOVD entity model**: Areas, Components (host-level), Apps (ROS 2 nodes), and Functions (namespace-based logical grouping) - **REST API**: Standard HTTP/JSON interface - **Incremental entity cache**: Discovery refresh diffs add/remove/change and performs zero structural allocations in the cache layer at steady state (object-pool backed, fixed capacity reserved at init via `entity_cache.capacity`) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp index c650d0212..6ff15b3a0 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/discovery_manager.hpp @@ -27,6 +27,7 @@ #include "ros2_medkit_gateway/discovery/manifest/manifest_manager.hpp" #include "ros2_medkit_gateway/discovery/merge_pipeline.hpp" #include "ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include #include @@ -183,6 +184,9 @@ class DiscoveryManager : public ServiceActionResolver { */ std::vector discover_apps(); + /// The ROS graph's node list through runtime discovery's reader, without its leftovers. Logs nothing. + ros2_common::GraphNodeList read_graph_nodes(); + /** * @brief Discover all functions * @return Vector of discovered Function entities (empty in runtime-only mode) diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp index 56769b7a4..6a14fa4d9 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp @@ -95,8 +95,16 @@ class RuntimeLinker { /** * @brief Constructor * @param node ROS node for logging (can be nullptr for testing) + * @param filter_internal_nodes The effective + * `discovery.runtime.filter_internal_nodes` setting. It decides + * whether this gateway's own in-process helper nodes are left out of + * `orphan_nodes`, because it is the same setting that decides whether + * they are filtered out of the served apps. With it off the helpers + * ARE served as apps, so reporting them as unmanifested is the truth; + * omitting them there would hide from `/health` exactly the nodes the + * operator turned the filter off to see. */ - explicit RuntimeLinker(rclcpp::Node * node = nullptr); + explicit RuntimeLinker(rclcpp::Node * node = nullptr, bool filter_internal_nodes = true); /** * @brief Link manifest apps to runtime apps (nodes) @@ -190,6 +198,8 @@ class RuntimeLinker { void log_error(const std::string & msg) const; rclcpp::Node * node_; + /// See the constructor: the helper-node skip below follows the app filter. + bool filter_internal_nodes_{true}; LinkingResult last_result_; }; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp index 7dc081dc0..d3c041ee4 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/entity_freeze_frame_capture.hpp @@ -60,6 +60,13 @@ namespace ros2_medkit_gateway { */ class EntityFreezeFrameCapture { public: + /// Capture-path identifiers stored in Frame::source and served as + /// ``x-medkit.source``. The plugin's own DataProvider, and the in-process + /// dispatch of the plugin's `x-plc-data` route for plugins that export no + /// DataProvider. + static constexpr const char * kSourceDataProvider = "plugin_data_provider"; + static constexpr const char * kSourceXPlcDataRoute = "plugin_x_plc_data_route"; + /// One captured frame: the entity's data values at fault-confirm time. /// captured_at_ns dates the capture, not the values - a disconnected entity /// serves its last known values, whose age is bounded only by the outage. @@ -72,6 +79,13 @@ class EntityFreezeFrameCapture { bool startup_catchup{false}; std::optional connected; ///< payload's top-level link flag, when reported nlohmann::json source_timestamp; ///< payload's own "timestamp" field verbatim (null when absent) + /// Which capture path read the values (kSourceDataProvider / + /// kSourceXPlcDataRoute), served as ``x-medkit.source``. These values are + /// entity data, not a ROS message, so ``topic`` and ``message_type`` are + /// empty on the wire and would otherwise leave a consumer with nothing at + /// all saying where the numbers came from. Empty when the caller named no + /// path. + std::string source; }; /// Resolves an entity id to its owning plugin's DataProvider (nullptr when @@ -187,9 +201,10 @@ class EntityFreezeFrameCapture { bool capture_for_event(const ros2_medkit_msgs::msg::FaultEvent & event, bool startup_catchup = false); /// Build a frame from list-data-shaped content, enforcing the shared - /// no-row-of-nulls invariant on both capture paths. + /// no-row-of-nulls invariant on both capture paths. @p source names the path + /// that read the content and is stored verbatim in Frame::source. std::optional frame_from_content(const std::string & entity_id, const std::string & fault_code, - const nlohmann::json & content); + const nlohmann::json & content, const std::string & source); /// Capture via the plugin's own x-plc-data route (no DataProvider exported). /// Returns nullopt when the route yields nothing usable. diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index a46a11bf5..4836d63b7 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -63,6 +64,7 @@ #include "ros2_medkit_gateway/ros2/transports/ros2_topic_transport.hpp" #include "ros2_medkit_gateway/ros2/trigger_topic_subscriber.hpp" #include "ros2_medkit_gateway/ros2_common/callback_groups.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_gateway/trigger_fault_subscriber.hpp" namespace ros2_medkit_gateway { @@ -440,6 +442,9 @@ class GatewayNode : public rclcpp::Node { // One-shot WARN when entity_cache capacity is exceeded (grew on first refresh after reserve). // Cleared only at construction time; never reset so the WARN fires at most once per run. bool warned_cache_grow_{false}; + /// Declared apps the helper-node rule last warned about, so a static + /// misconfiguration is reported when it appears or changes, not every refresh. + std::set warned_helper_bound_apps_; // Graph-change-driven discovery refresh. // @@ -497,6 +502,8 @@ class GatewayNode : public rclcpp::Node { std::unique_ptr server_thread_; }; +using ros2_common::is_own_gateway_helper_node; + /** * @brief Filter ROS 2 internal nodes from an app list * @@ -505,11 +512,40 @@ class GatewayNode : public rclcpp::Node { * before checking for the underscore prefix, using the routing table for precise * prefix detection. * + * Also removes local apps bound to one of the gateway's in-process helper nodes + * (is_own_gateway_helper_node), which the underscore rule cannot see. The + * gateway's own node is NOT removed - it is a diagnosable App whose ROS + * parameters are served as its configurations. The test is on the bound node + * FQN, and only for apps with no routing-table entry: a peer's helper nodes are + * the peer's business and are left to the peer's own filter. + * * @param apps App vector to filter in place * @param peer_routing_table Maps entity_id -> peer_name for remote entities + * @param self_fqn The gateway node's own FQN. Empty disables the helper check + * @param dropped_declared_apps Optional sink for " -> " of + * every app removed by the helper rule whose source is not runtime + * discovery. Removing a declared entity silently would override the + * manifest without saying so, and this function has no logger * @return Number of apps removed */ size_t filter_internal_node_apps(std::vector & apps, - const std::unordered_map & peer_routing_table); + const std::unordered_map & peer_routing_table, + const std::string & self_fqn, + std::vector * dropped_declared_apps = nullptr); + +/** + * @brief Remember which declared apps were dropped, and say whether that changed + * + * The condition this gates is a static misconfiguration, while the caller runs + * on every graph event and on the refresh cadence, so warning per call would + * repeat the same line for the life of the process. Returns true only when the + * set differs from the remembered one and is not empty; the remembered set is + * updated either way, so a condition that clears and returns is reported again. + * + * @param dropped App ids (with their bound FQNs) dropped by the helper rule + * @param remembered In/out: the set the caller last warned about + * @return true when the caller should warn + */ +bool remember_dropped_declared_apps(const std::vector & dropped, std::set & remembered); } // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp index 9a2a4eaff..cac62795e 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2/providers/ros2_runtime_introspection.hpp @@ -21,6 +21,7 @@ #include "ros2_medkit_gateway/core/discovery/models/component.hpp" #include "ros2_medkit_gateway/core/discovery/models/function.hpp" #include "ros2_medkit_gateway/core/providers/introspection_provider.hpp" +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" #include "ros2_medkit_serialization/type_introspection.hpp" #include @@ -93,9 +94,13 @@ class Ros2RuntimeIntrospection : public IntrospectionProvider { // --------------------------------------------------------------------------- /// Discover the live nodes as Apps. Always queries the ROS 2 graph; do not - /// call from hot paths. + /// call from hot paths. A leftover (see ros2_common::GraphNodeListReader) is + /// not an App; the read that starts leaving it out logs it. std::vector discover_apps(); + /// The node list through discover_apps()'s reader, without its leftovers. Logs nothing. + ros2_common::GraphNodeList read_graph_nodes(); + /// Group nodes by namespace into Function entities (no graph query). std::vector discover_functions(const std::vector & apps); @@ -153,6 +158,11 @@ class Ros2RuntimeIntrospection : public IntrospectionProvider { std::map cached_topic_map_; bool topic_map_ready_{false}; + + /// Nodes this discovery saw running, shared by discover_apps() and read_graph_nodes(). + ros2_common::GraphNodeListReader graph_node_reader_; + /// Logs the leftovers discover_apps() leaves out. + ros2_common::LeftoverNodeReporter leftover_node_reporter_; }; } // namespace ros2_medkit_gateway::ros2 diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp new file mode 100644 index 000000000..2d78be2e3 --- /dev/null +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/ros2_common/graph_node_list.hpp @@ -0,0 +1,320 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +// Header-only: the discovery plugins and their tests use it without the gateway's libraries. + +namespace ros2_medkit_gateway::ros2_common { + +/// One entry of `get_node_names_with_enclaves()`: name, namespace, enclave. +using GraphNodeEntry = std::tuple; + +/// The ROS graph's node list as a GraphNodeListReader reads it. +struct GraphNodeList { + /// (name, namespace) of every entry kept, in graph order. + std::vector> nodes; + /// FQN of every node left out as a leftover, once each, in graph order. + std::vector leftovers; +}; + +/// Fully qualified name from the name and namespace the graph reports. +inline std::string graph_node_fqn(const std::string & name, const std::string & ns) { + if (ns.empty() || ns == "/") { + return "/" + name; + } + return ns + "/" + name; +} + +/** + * @brief Is this FQN one of the gateway's in-process helper nodes? + * + * True for `_sub`, `_fault_clients` and `_lifecycle_state_reader`, exact matches + * only. False for the gateway node itself, which is a diagnosable App. The last two take only the + * gateway's node name, so they also match as `/`; two gateways with one name in + * different namespaces both claim those root-namespace nodes. + * + * @param node_fqn Fully qualified node name to test ("/ns/node") + * @param self_fqn The gateway node's own FQN. An empty value matches nothing + */ +inline bool is_own_gateway_helper_node(const std::string & node_fqn, const std::string & self_fqn) { + if (self_fqn.empty() || node_fqn.empty()) { + return false; + } + struct HelperNode { + const char * suffix; + bool follows_gateway_namespace; + }; + static constexpr std::array kHelperNodes{{ + {"_sub", true}, + {"_fault_clients", false}, + {"_lifecycle_state_reader", false}, + }}; + const auto last_slash = self_fqn.rfind('/'); + const std::string bare_name = last_slash == std::string::npos ? self_fqn : self_fqn.substr(last_slash + 1); + return std::any_of(kHelperNodes.begin(), kHelperNodes.end(), [&](const HelperNode & helper) { + if (node_fqn == self_fqn + helper.suffix) { + return true; + } + return !helper.follows_gateway_namespace && !bare_name.empty() && node_fqn == "/" + bare_name + helper.suffix; + }); +} + +/** + * @brief Whether the ROS graph resolves any publisher or subscriber for a node. + * + * On the DDS RMWs (rmw_dds_common), without demangling these also cover services and clients (`rq/`, `rr/` topics). + * A node the graph no longer lists has none. Other errors, such as a shut-down context, propagate. + */ +inline bool graph_node_has_endpoints(const rclcpp::node_interfaces::NodeGraphInterface & graph, + const std::string & name, const std::string & ns) { + try { + return !graph.get_publisher_names_and_types_by_node(name, ns, true).empty() || + !graph.get_subscriber_names_and_types_by_node(name, ns, true).empty(); + } catch (const rclcpp::exceptions::RCLError & error) { + if (error.ret == RCL_RET_NODE_NAME_NON_EXISTENT) { + return false; + } + throw; + } +} + +/** + * @brief Reads the ROS graph's node list, leaving out the leftovers of nodes it saw running. + * + * A leftover is an entry with an empty enclave and no endpoints that a late `ros_discovery_info` + * sample puts back after its participant left. Nothing removes it again. An empty enclave alone is + * not enough: micro-ROS and DDS-router nodes read the same, so only names this reader saw running + * are hidden. Per FQN, on every read: + * - an entry with an enclave is listed; + * - an entry without an enclave is dropped when another entry of the name has one; + * - an entry without an enclave for a departed name is left out when it has no endpoints; + * - any other entry without an enclave is listed. + * + * A name that ran on the previous read and does not run now has departed. It is forgotten when it + * runs again, or on the first read that finds no entry of it more than the hold after the first + * read that found none. A listed entry without an enclave stops that clock. At most `capacity` + * departed names are kept; the ones unlisted longest go first, then the ones departed longest ago. + * + * Thread-safe. A read holds the lock from the list query to the last endpoint query. + */ +class GraphNodeListReader { + public: + using Clock = std::chrono::steady_clock; + + /// How long after the first read without an entry a departed name is kept. It covers the listener + /// thread's delay in taking a late sample, which exceeds 0.5 s on a loaded host. + static constexpr std::chrono::seconds kDefaultHold{10}; + /// Most departed names kept. Each hidden leftover costs two endpoint queries per read. + static constexpr std::size_t kDefaultCapacity = 1024; + + GraphNodeListReader() = default; + GraphNodeListReader(Clock::duration hold, std::size_t capacity) : hold_(hold), capacity_(capacity) { + } + + /** + * @brief Read the node list, leaving out leftovers of nodes this reader saw running. + * + * Throws what the graph queries throw, for example once the context is shut down. + */ + GraphNodeList read(const rclcpp::node_interfaces::NodeGraphInterface & graph) { + std::lock_guard lock(mutex_); + auto entries = graph.get_node_names_with_enclaves(); + return filter_locked(entries, Clock::now(), [&graph](const std::string & name, const std::string & ns) { + return graph_node_has_endpoints(graph, name, ns); + }); + } + + /// The decision read() makes, for `entries` read at `now`. + /// @param has_endpoints `bool(const std::string & name, const std::string & ns)` + template + GraphNodeList filter(const std::vector & entries, Clock::time_point now, + HasEndpoints && has_endpoints) { + std::lock_guard lock(mutex_); + return filter_locked(entries, now, std::forward(has_endpoints)); + } + + /// How many departed names the reader keeps. + std::size_t remembered() const { + std::lock_guard lock(mutex_); + return departed_.size(); + } + + private: + /// A name the reader saw running that no entry with an enclave lists. + struct Departed { + /// The read that first found the name not running. + Clock::time_point departed_at; + /// First read of the current run that listed no entry of the name; empty while one is listed. + std::optional absent_since; + }; + + template + GraphNodeList filter_locked(const std::vector & entries, Clock::time_point now, + HasEndpoints && has_endpoints) { + std::unordered_set running; + std::unordered_set listed; + for (const auto & [name, ns, enclave] : entries) { + auto fqn = graph_node_fqn(name, ns); + if (!enclave.empty()) { + running.insert(fqn); + } + listed.insert(std::move(fqn)); + } + remember(std::move(running), listed, now); + + GraphNodeList result; + result.nodes.reserve(entries.size()); + std::unordered_map resolved; + for (const auto & [name, ns, enclave] : entries) { + if (!enclave.empty()) { + result.nodes.emplace_back(name, ns); + continue; + } + auto fqn = graph_node_fqn(name, ns); + if (running_.count(fqn) > 0) { + continue; + } + if (departed_.count(fqn) == 0) { + result.nodes.emplace_back(name, ns); + continue; + } + auto found = resolved.find(fqn); + if (found == resolved.end()) { + const bool has = has_endpoints(name, ns); + found = resolved.emplace(fqn, has).first; + if (!has) { + result.leftovers.push_back(fqn); + } + } + if (found->second) { + result.nodes.emplace_back(name, ns); + } + } + return result; + } + + void remember(std::unordered_set running, const std::unordered_set & listed, + Clock::time_point now) { + for (const auto & fqn : running) { + departed_.erase(fqn); + } + for (const auto & fqn : running_) { + if (running.count(fqn) == 0) { + departed_.try_emplace(fqn, Departed{now, std::nullopt}); + } + } + running_ = std::move(running); + for (auto it = departed_.begin(); it != departed_.end();) { + auto & departed = it->second; + if (listed.count(it->first) > 0) { + departed.absent_since.reset(); + } else if (!departed.absent_since) { + departed.absent_since = now; + } else if (now - *departed.absent_since > hold_) { + it = departed_.erase(it); + continue; + } + ++it; + } + if (departed_.size() <= capacity_) { + return; + } + // Drop the names unlisted longest first, then the ones that departed longest ago. + std::vector::iterator> order; + order.reserve(departed_.size()); + for (auto it = departed_.begin(); it != departed_.end(); ++it) { + order.push_back(it); + } + const auto excess = static_cast(departed_.size() - capacity_); + std::nth_element(order.begin(), order.begin() + excess, order.end(), [](const auto & lhs, const auto & rhs) { + const auto & a = lhs->second; + const auto & b = rhs->second; + if (a.absent_since.has_value() != b.absent_since.has_value()) { + return a.absent_since.has_value(); + } + if (a.absent_since) { + return *a.absent_since < *b.absent_since; + } + return a.departed_at < b.departed_at; + }); + for (auto it = order.begin(); it != order.begin() + excess; ++it) { + departed_.erase(*it); + } + } + + Clock::duration hold_{kDefaultHold}; + std::size_t capacity_{kDefaultCapacity}; + mutable std::mutex mutex_; + /// Names an entry with an enclave listed on the latest read. + std::unordered_set running_; + /// Names seen running that no entry with an enclave lists since, at most `capacity_`. + std::unordered_map departed_; +}; + +/** + * @brief Logs a WARN for each leftover on the first read of a run of reads that leave it out. + * + * Thread-safe: the refresh and HTTP handlers both read the node list. + */ +class LeftoverNodeReporter { + public: + explicit LeftoverNodeReporter(rclcpp::Logger logger) : logger_(std::move(logger)) { + } + + /// Log the leftovers `list` leaves out that the previous read did not. Returns their FQNs. + std::vector report(const GraphNodeList & list) { + std::vector newly_left_out; + std::set current(list.leftovers.begin(), list.leftovers.end()); + std::lock_guard lock(mutex_); + for (const auto & fqn : list.leftovers) { + if (reported_.count(fqn) == 0) { + newly_left_out.push_back(fqn); + RCLCPP_WARN(logger_, + "Node '%s' is not exposed: this gateway saw it running, and the ROS graph still lists it " + "after its participant left, with no endpoints", + fqn.c_str()); + } + } + reported_ = std::move(current); + return newly_left_out; + } + + private: + rclcpp::Logger logger_; + std::mutex mutex_; + std::set reported_; +}; + +} // namespace ros2_medkit_gateway::ros2_common diff --git a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp index e8c1bf49f..ec7304474 100644 --- a/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp +++ b/src/ros2_medkit_gateway/src/discovery/discovery_manager.cpp @@ -213,7 +213,8 @@ void DiscoveryManager::build_pipeline() { // RuntimeLinker only makes sense when runtime is enabled. if (config_.runtime_enabled) { - pipeline->set_linker(std::make_unique(node_), get_manifest_config()); + pipeline->set_linker(std::make_unique(node_, config_.runtime.filter_internal_nodes), + get_manifest_config()); } pipeline_ = std::move(pipeline); @@ -289,6 +290,10 @@ std::vector DiscoveryManager::discover_apps() { return runtime_introspection_->discover_apps(); } +ros2_common::GraphNodeList DiscoveryManager::read_graph_nodes() { + return runtime_introspection_->read_graph_nodes(); +} + std::vector DiscoveryManager::discover_functions() { if (config_.mode == DiscoveryMode::MANIFEST_ONLY && manifest_manager_ && manifest_manager_->is_manifest_active()) { return manifest_manager_->get_functions(); diff --git a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp index 565a23736..d2ed63566 100644 --- a/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp +++ b/src/ros2_medkit_gateway/src/discovery/manifest/runtime_linker.cpp @@ -16,6 +16,7 @@ #include "ros2_medkit_gateway/core/discovery/merge_types.hpp" #include "ros2_medkit_gateway/core/http/warning_codes.hpp" +#include "ros2_medkit_gateway/gateway_node.hpp" #include @@ -68,7 +69,8 @@ bool topic_path_matches(const std::string & topic, const std::string & topic_nam } // namespace -RuntimeLinker::RuntimeLinker(rclcpp::Node * node) : node_(node) { +RuntimeLinker::RuntimeLinker(rclcpp::Node * node, bool filter_internal_nodes) + : node_(node), filter_internal_nodes_(filter_internal_nodes) { } LinkingResult RuntimeLinker::link(const std::vector & manifest_apps, const std::vector & runtime_apps, @@ -182,9 +184,23 @@ LinkingResult RuntimeLinker::link(const std::vector & manifest_apps, const result.linked_apps.push_back(linked_app); } - // Find orphan nodes (runtime apps not matching any manifest app) + // Find orphan nodes (runtime apps not matching any manifest app). + // + // The gateway's own in-process helper nodes are skipped WHILE the app filter + // removes them - the two are one setting, discovery.runtime.filter_internal_nodes. + // With it on they are not entities, so reporting them here would tell the + // operator to declare, in the manifest, nodes that can never become apps, and + // the 'error' policy words that report as an instruction ("Declare them in + // the manifest"). With it off they ARE served as apps, and an unmanifested + // served app is precisely what orphan_nodes and /health's orphan_count are + // for. An empty self FQN, which is what a linker constructed without a node + // has, matches nothing either way. + const std::string self_fqn = node_ != nullptr ? node_->get_fully_qualified_name() : std::string(); for (const auto & rt_app : runtime_apps) { if (rt_app.bound_fqn.has_value() && matched_nodes.find(rt_app.bound_fqn.value()) == matched_nodes.end()) { + if (filter_internal_nodes_ && is_own_gateway_helper_node(rt_app.bound_fqn.value(), self_fqn)) { + continue; + } result.orphan_nodes.push_back(rt_app.bound_fqn.value()); } } diff --git a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp index fe060826c..01bde9674 100644 --- a/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/src/entity_freeze_frame_capture.cpp @@ -188,13 +188,14 @@ EntityFreezeFrameCapture::standing_faults_from_list_reply(const nlohmann::json & std::optional EntityFreezeFrameCapture::frame_from_content(const std::string & entity_id, const std::string & fault_code, - const nlohmann::json & content) { + const nlohmann::json & content, const std::string & source) { if (!content_has_live_data(content)) { log_fallback_failure_once(fault_code, "entity '" + entity_id + "' returned no data items"); return std::nullopt; } Frame frame; frame.entity_id = entity_id; + frame.source = source; frame.values = values_from_list_content(content); if (!values_have_data(frame.values)) { // Items present but nothing usable in them (all-null values, or no usable @@ -228,7 +229,7 @@ EntityFreezeFrameCapture::capture_via_route(const std::string & entity_id, const if (!content) { return std::nullopt; // not plugin-owned, no x-plc-data route, or handler error } - return frame_from_content(entity_id, fault_code, *content); + return frame_from_content(entity_id, fault_code, *content, kSourceXPlcDataRoute); } void EntityFreezeFrameCapture::log_fallback_failure_once(const std::string & fault_code, const std::string & message) { @@ -410,7 +411,7 @@ bool EntityFreezeFrameCapture::capture_for_event(const ros2_medkit_msgs::msg::Fa log_fallback_failure_once(fault_code, "list_data('" + source + "') failed: " + result.error().message); continue; } - if (auto frame = frame_from_content(source, fault_code, result->content)) { + if (auto frame = frame_from_content(source, fault_code, result->content, kSourceDataProvider)) { frames.push_back(std::move(*frame)); } } catch (const std::exception & e) { diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index a8f32986e..b9fec6ca1 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -31,6 +31,7 @@ #include "ros2_medkit_gateway/core/aggregation/network_utils.hpp" #include "ros2_medkit_gateway/core/data/topic_data_provider.hpp" +#include "ros2_medkit_gateway/core/discovery/merge_types.hpp" #include "ros2_medkit_gateway/core/discovery/refresh_debounce.hpp" #include "ros2_medkit_gateway/core/entity_validation.hpp" #include "ros2_medkit_gateway/core/faults/fault_scope.hpp" @@ -1608,10 +1609,10 @@ size_t GatewayNode::count_peer_nodes(const std::vector_monitor" or "2"). - if (fqn == self_fqn || fqn == self_fqn + "_sub" || fqn == self_fqn + "_fault_clients") { + // The gateway is not a peer of itself, and neither are the helper nodes it + // runs in its own process: a gateway alone on the graph has zero peers, + // which is what raises the empty-graph warning. + if (fqn == self_fqn || is_own_gateway_helper_node(fqn, self_fqn)) { continue; } ++count; @@ -1645,8 +1646,8 @@ void GatewayNode::log_startup_summary() { ++topic_count; } } - peer_node_count = - count_peer_nodes(get_node_graph_interface()->get_node_names_and_namespaces(), get_fully_qualified_name()); + // Read through discovery's own reader, so a leftover discovery leaves out is not a peer either. + peer_node_count = count_peer_nodes(discovery_mgr_->read_graph_nodes().nodes, get_fully_qualified_name()); } catch (const std::exception & e) { RCLCPP_DEBUG(get_logger(), "Startup summary: graph query failed: %s", e.what()); } @@ -2457,14 +2458,32 @@ void GatewayNode::refresh_cache() { } } - // Filter ROS 2 internal nodes (underscore prefix convention). + // Filter ROS 2 internal nodes (underscore prefix convention) and this + // gateway's own helper nodes. // Controlled by discovery.runtime.filter_internal_nodes parameter (default: true). // Covers local heuristic apps (which bypass the merge pipeline orphan filter // in runtime_only mode) and any peer apps that slipped through fetch_entities. if (filter_internal_nodes_) { - auto removed = filter_internal_node_apps(apps, peer_routing_table); + std::vector dropped_declared_apps; + auto removed = + filter_internal_node_apps(apps, peer_routing_table, get_fully_qualified_name(), &dropped_declared_apps); if (removed > 0) { - RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix)", removed); + RCLCPP_DEBUG(get_logger(), "Filtered %zu internal node apps (_ prefix or own helper node)", removed); + } + // Warn on the set, not on the refresh. The condition is a static + // misconfiguration - an app declared against a helper node stays declared + // - while this function runs on every graph event and again on the + // backstop cadence, which the integration fixtures set to one second. The + // same one-shot discipline as the entity-cache WARN below, widened to + // re-fire when the set of offending apps actually changes. + if (remember_dropped_declared_apps(dropped_declared_apps, warned_helper_bound_apps_)) { + for (const auto & dropped : dropped_declared_apps) { + RCLCPP_WARN(get_logger(), + "Declared app '%s' is bound to one of this gateway's own in-process helper nodes and is not " + "served. Those nodes carry no parameters, services or actions to diagnose. Bind the app to the " + "node you meant, or drop the declaration.", + dropped.c_str()); + } } } @@ -2560,10 +2579,22 @@ void GatewayNode::stop_rest_server() { } } +bool remember_dropped_declared_apps(const std::vector & dropped, std::set & remembered) { + std::set current(dropped.begin(), dropped.end()); + if (current == remembered) { + return false; + } + remembered = std::move(current); + // An empty set is remembered so the condition clearing and coming back warns + // again, but there is nothing to say about no apps at all. + return !remembered.empty(); +} + size_t filter_internal_node_apps(std::vector & apps, - const std::unordered_map & peer_routing_table) { + const std::unordered_map & peer_routing_table, + const std::string & self_fqn, std::vector * dropped_declared_apps) { auto before = apps.size(); - auto end = std::remove_if(apps.begin(), apps.end(), [&peer_routing_table](const App & app) { + auto end = std::remove_if(apps.begin(), apps.end(), [&](const App & app) { std::string original_id = app.id; auto rt_it = peer_routing_table.find(app.id); if (rt_it != peer_routing_table.end()) { @@ -2573,6 +2604,28 @@ size_t filter_internal_node_apps(std::vector & apps, if (original_id.size() > prefix.size() && original_id.compare(0, prefix.size(), prefix) == 0) { original_id = original_id.substr(prefix.size()); } + } else if (is_own_gateway_helper_node(app.effective_fqn(), self_fqn)) { + // A local app bound to one of this gateway's own helper nodes. Those + // names do not start with '_' ("_sub", + // "_fault_clients", ...), so only the FQN test catches them, and + // without it the gateway advertises its own plumbing as diagnosable apps. + // The gateway's own node is deliberately not covered: its ROS parameters + // are served as that App's configurations, so callers reach them at + // /apps//configurations. Remote entities are skipped too - a + // peer's helper nodes carry the same FQNs and are the peer's own filter's + // business. + // + // Dropping a runtime-discovered app is the whole point and stays quiet. + // Dropping one somebody DECLARED is different: in manifest and hybrid + // mode the manifest is the source of truth, so removing an entry from it + // without a word is a silent override. The declared sources are the ones + // the merge pipeline already protects from orphan suppression, so the + // same predicate decides it here; those are reported to the caller, which + // owns the logger. + if (dropped_declared_apps != nullptr && discovery::is_protected_source(app.source)) { + dropped_declared_apps->push_back(app.id + " -> " + app.effective_fqn()); + } + return true; } // ROS 2 internal nodes use _ prefix convention return !original_id.empty() && original_id[0] == '_'; diff --git a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp index b235acc53..0df7419b9 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp @@ -263,6 +263,13 @@ json FaultHandlers::merge_entity_freeze_frames(json env_data, snap["topic"] = ""; // entity data values, not a ROS topic snap["message_type"] = ""; snap["captured_at_ns"] = frame.captured_at_ns; + // Capture provenance. topic/message_type stay empty because these values + // are not a ROS message, which leaves "source" as the only field naming + // where the numbers came from - so carry it whenever the capture named a + // path. + if (!frame.source.empty()) { + snap["source"] = frame.source; + } if (frame.startup_catchup) { // Values were read at gateway start, not when the fault confirmed; // absent marker = captured on the confirm edge. @@ -344,6 +351,13 @@ dto::FaultDetail FaultHandlers::build_sovd_fault_response(const json & fault_jso snap["x-medkit"]["capture_origin"] = s["capture_origin"]; } // Entity-frame provenance (merge_entity_freeze_frames), only when known. + // "source" names the capture path (a plugin DataProvider or the + // plugin's x-plc-data route). It is where a consumer reads the + // provenance from: an entity frame necessarily carries an empty + // topic/message_type. + if (s.contains("source") && s["source"].is_string()) { + snap["x-medkit"]["source"] = s["source"]; + } if (s.contains("connected") && s["connected"].is_boolean()) { snap["x-medkit"]["connected"] = s["connected"]; } diff --git a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp index fcc0948d0..cbe232f20 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/health_handlers.cpp @@ -104,13 +104,19 @@ http::Result HealthHandlers::get_health(const http::TypedRequest & "policy to 'warn' or 'ignore'."; // The subjects of this warning are ROS nodes, so they go in // ros_node_fqns and entity_ids stays empty. Not a formality: the - // orphan list is taken from the UNFILTERED runtime app list - // (runtime_layer.cpp), before gap-fill, the namespace filters and - // the policy filter, so some of these nodes have no SOVD entity at - // all - and for those that do, the App id is derived from the bare - // node name and only becomes namespace-qualified when some other - // node collides with it (ros2_runtime_introspection.cpp), so it can - // change between two polls while the FQN cannot. + // orphan list is taken from the runtime app list (runtime_layer.cpp) + // before gap-fill, the namespace filters and the policy filter, so + // some of these nodes have no SOVD entity at all - and for those that + // do, the App id is derived from the bare node name and only becomes + // namespace-qualified when some other node collides with it + // (ros2_runtime_introspection.cpp), so it can change between two + // polls while the FQN cannot. One thing is taken out of it while + // discovery.runtime.filter_internal_nodes is on (runtime_linker.cpp): + // the gateway's own in-process helper nodes, which the app filter + // then removes, so naming them here would be an instruction to + // declare something that can never become an App. With the filter off + // they are served as Apps and appear here like any other undeclared + // node. warning.ros_node_fqns = linking->orphan_nodes; warnings.push_back(std::move(warning)); } diff --git a/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp b/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp index 66381e166..f2eab8339 100644 --- a/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp +++ b/src/ros2_medkit_gateway/src/ros2/providers/ros2_runtime_introspection.cpp @@ -49,7 +49,12 @@ bool Ros2RuntimeIntrospection::is_internal_service(const std::string & service_p service_path.find("/_action/") != std::string::npos; } -Ros2RuntimeIntrospection::Ros2RuntimeIntrospection(rclcpp::Node * node) : node_(node) { +Ros2RuntimeIntrospection::Ros2RuntimeIntrospection(rclcpp::Node * node) + : node_(node), leftover_node_reporter_(node->get_logger()) { +} + +ros2_common::GraphNodeList Ros2RuntimeIntrospection::read_graph_nodes() { + return graph_node_reader_.read(*node_->get_node_graph_interface()); } void Ros2RuntimeIntrospection::set_config(const RuntimeConfig & config) { @@ -86,14 +91,14 @@ std::vector Ros2RuntimeIntrospection::discover_apps() { auto node_graph = node_->get_node_graph_interface(); std::vector> names_and_namespaces; try { - names_and_namespaces = node_graph->get_node_names_and_namespaces(); + // Leftovers go before the bare-name collision count, so they never rename a live node. + auto node_list = graph_node_reader_.read(*node_graph); + leftover_node_reporter_.report(node_list); + names_and_namespaces = std::move(node_list.nodes); } catch (const std::runtime_error & ex) { - // rclcpp throws "rcl node's context is invalid" when get_node_names_* - // is called after rclcpp::shutdown (e.g. refresh timer fires once - // between SIGINT handling and the executor stopping). Swallow and - // return empty so ~GatewayNode's shutdown path isn't aborted mid-run - // by std::terminate; callers handle empty gracefully. - RCLCPP_DEBUG(node_->get_logger(), "get_node_names_and_namespaces threw during shutdown: %s", ex.what()); + // A graph query throws once rclcpp is shut down, e.g. a refresh between SIGINT and the + // executor stopping. Return empty so the shutdown path does not end in std::terminate. + RCLCPP_DEBUG(node_->get_logger(), "Reading the node list threw during shutdown: %s", ex.what()); return {}; } diff --git a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp index b9a44b2e0..da14202d2 100644 --- a/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp +++ b/src/ros2_medkit_gateway/test/test_entity_freeze_frame_capture.cpp @@ -573,6 +573,15 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedEntityWithLastKnownValuesIsCapt ASSERT_TRUE(frames[0].connected.has_value()); EXPECT_FALSE(*frames[0].connected); EXPECT_EQ(frames[0].source_timestamp, 1234567890); + // Which path read the values. This capture had no DataProvider and went + // through the route fallback, so the frame must name that path. The + // DataProvider flavour of the same case asserts the other constant, which is + // what stops the two from being swapped at their call sites unnoticed. + EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceXPlcDataRoute); + // The wire value itself, because comparing symbol against symbol cannot see + // the two constants' contents being swapped, and that swap is an API break + // for every consumer of x-medkit.source. + EXPECT_EQ(frames[0].source, "plugin_x_plc_data_route"); } /// @verifies REQ_INTEROP_088 @@ -642,6 +651,11 @@ TEST_F(EntityFreezeFrameCaptureTest, DisconnectedDataProviderWithLastKnownValues ASSERT_TRUE(frames[0].connected.has_value()); EXPECT_FALSE(*frames[0].connected); EXPECT_TRUE(frames[0].source_timestamp.is_null()); // provider content has no timestamp field + // The provider path names itself, and the route path (same case, above) names + // the other constant: the pair is what makes a swap of the two call sites + // visible. The literal pins the wire value the API reference documents. + EXPECT_EQ(frames[0].source, EntityFreezeFrameCapture::kSourceDataProvider); + EXPECT_EQ(frames[0].source, "plugin_data_provider"); } /// @verifies REQ_INTEROP_088 @@ -787,6 +801,40 @@ TEST(MergeEntityFreezeFrames, AppendsWhenNoConfiguredFreezeFrame) { EXPECT_FALSE(snap.contains("capture_origin")); // confirm-edge frames carry no marker } +TEST(MergeEntityFreezeFrames, CarriesCapturePathAsSource) { + // An entity frame has no ROS topic, so topic/message_type are necessarily + // empty, so "source" is the only field left saying where they came from. + json env_data = {{"snapshots", json::array()}}; + EntityFreezeFrameCapture::Frame frame; + frame.entity_id = "plc_app"; + frame.values = {{"temperature", 42.5}}; + frame.captured_at_ns = 1234; + frame.source = EntityFreezeFrameCapture::kSourceXPlcDataRoute; + + auto merged = FaultHandlers::merge_entity_freeze_frames(env_data, {frame}); + ASSERT_EQ(merged["snapshots"].size(), 1u); + const auto & snap = merged["snapshots"][0]; + EXPECT_EQ(snap["source"], EntityFreezeFrameCapture::kSourceXPlcDataRoute); + EXPECT_EQ(snap["topic"], ""); + EXPECT_EQ(snap["message_type"], ""); +} + +TEST(MergeEntityFreezeFrames, OmitsSourceForAFrameThatNamesNoPath) { + // A merge-helper contract, not a control for the capture tests: both capture + // paths always name themselves (asserted from real captures in + // Disconnected{Entity,DataProvider}WithLastKnownValuesIsCaptured), so this + // frame is one only a caller can build. The helper leaves the key out, so + // the wire carries no provenance a consumer would trust. + json env_data = {{"snapshots", json::array()}}; + EntityFreezeFrameCapture::Frame frame; + frame.entity_id = "plc_app"; + frame.values = {{"temperature", 42.5}}; + + auto merged = FaultHandlers::merge_entity_freeze_frames(env_data, {frame}); + ASSERT_EQ(merged["snapshots"].size(), 1u); + EXPECT_FALSE(merged["snapshots"][0].contains("source")); +} + TEST(MergeEntityFreezeFrames, StartupCatchUpFrameCarriesCaptureOrigin) { json env_data = {{"snapshots", json::array()}}; EntityFreezeFrameCapture::Frame frame; diff --git a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp index 9c7f08a91..29b53ef22 100644 --- a/src/ros2_medkit_gateway/test/test_fault_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_fault_handlers.cpp @@ -142,6 +142,54 @@ TEST_F(FaultHandlersTest, BuildSovdFaultResponsePropagatesCaptureOrigin) { EXPECT_EQ(snap["x-medkit"]["capture_origin"], "startup"); } +TEST_F(FaultHandlersTest, BuildSovdFaultResponseServesEntityFrameSource) { + // A plugin-captured entity frame reaches the wire with an empty topic and + // message_type (the values are not a ROS message) plus x-medkit.source + // naming the capture path that read them. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "PLC_ALARM"; + + json env_data = {{"snapshots", json::array({{{"type", "freeze_frame"}, + {"snapshot_type", "freeze_frame"}, + {"name", "plc_app"}, + {"data", R"({"tank_level": 87.5})"}, + {"topic", ""}, + {"message_type", ""}, + {"captured_at_ns", 1234}, + {"source", "plugin_x_plc_data_route"}}})}}; + + auto response = to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_data, "/apps/plc_app")); + + auto & snap = response["environment_data"]["snapshots"][0]; + EXPECT_EQ(snap["x-medkit"]["source"], "plugin_x_plc_data_route"); + EXPECT_EQ(snap["x-medkit"]["topic"], ""); + EXPECT_EQ(snap["x-medkit"]["message_type"], ""); +} + +TEST_F(FaultHandlersTest, BuildSovdFaultResponseOmitsSourceWhenTheSnapshotHasNone) { + // Absence control for the test above, on the same harness: a topic-captured + // freeze frame (the fault_manager's own) carries no source, and none is + // invented for it. + ros2_medkit_msgs::msg::Fault fault; + fault.fault_code = "TEMP_FAULT"; + + ros2_medkit_msgs::msg::EnvironmentData env_data; + ros2_medkit_msgs::msg::Snapshot freeze_frame; + freeze_frame.type = "freeze_frame"; + freeze_frame.name = "temperature"; + freeze_frame.data = R"({"temperature": 85.5})"; + freeze_frame.topic = "/motor/temperature"; + freeze_frame.message_type = "sensor_msgs/msg/Temperature"; + env_data.snapshots.push_back(freeze_frame); + + auto response = + to_json(FaultHandlers::build_sovd_fault_response(fault_json(fault), env_json(env_data), "/apps/motor")); + + auto & snap = response["environment_data"]["snapshots"][0]; + EXPECT_FALSE(snap["x-medkit"].contains("source")); + EXPECT_EQ(snap["x-medkit"]["topic"], "/motor/temperature"); +} + // Conversion layer must emit an explicit "snapshot_type" discriminator so // downstream consumers (handler, SSE, MCP) can dispatch on a single key // regardless of which optional payload fields are present. diff --git a/src/ros2_medkit_gateway/test/test_gateway_node.cpp b/src/ros2_medkit_gateway/test/test_gateway_node.cpp index 2f81dbc6c..a92edcc4e 100644 --- a/src/ros2_medkit_gateway/test/test_gateway_node.cpp +++ b/src/ros2_medkit_gateway/test/test_gateway_node.cpp @@ -1058,10 +1058,14 @@ TEST(GatewayStartupSummary, CountPeerNodesExcludesOwnAndHidden) { } TEST(GatewayStartupSummary, CountPeerNodesZeroWhenOnlyOwnNodes) { + // The gateway node plus every helper it creates inside its own process. A + // gateway alone on the graph must report zero peers, so each of these has to + // be recognized as not-a-peer, the lifecycle reader included. const std::vector> nodes = { {"ros2_medkit_gateway", "/"}, {"ros2_medkit_gateway_sub", "/"}, {"ros2_medkit_gateway_fault_clients", "/"}, + {"ros2_medkit_gateway_lifecycle_state_reader", "/"}, }; // Zero peers is the condition that triggers the empty-graph warning. EXPECT_EQ(ros2_medkit_gateway::GatewayNode::count_peer_nodes(nodes, "/ros2_medkit_gateway"), 0u); diff --git a/src/ros2_medkit_gateway/test/test_graph_node_list.cpp b/src/ros2_medkit_gateway/test/test_graph_node_list.cpp new file mode 100644 index 000000000..b552cc22c --- /dev/null +++ b/src/ros2_medkit_gateway/test/test_graph_node_list.cpp @@ -0,0 +1,294 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// The node-list reader's decisions. The test_graph_leftover_nodes*.test.py suites in +// ros2_medkit_integration_tests cover a real graph. + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "ros2_medkit_gateway/ros2_common/graph_node_list.hpp" + +using ros2_medkit_gateway::ros2_common::graph_node_fqn; +using ros2_medkit_gateway::ros2_common::graph_node_has_endpoints; +using ros2_medkit_gateway::ros2_common::GraphNodeEntry; +using ros2_medkit_gateway::ros2_common::GraphNodeList; +using ros2_medkit_gateway::ros2_common::GraphNodeListReader; +using ros2_medkit_gateway::ros2_common::LeftoverNodeReporter; + +namespace { + +using NamePair = std::pair; +using Clock = GraphNodeListReader::Clock; +using std::chrono::seconds; + +/// Endpoint lookup over a fixed set of FQNs that resolve endpoints, counting every question asked. +struct EndpointTable { + std::set with_endpoints; + std::map asked; + + auto probe() { + return [this](const std::string & name, const std::string & ns) { + const auto fqn = graph_node_fqn(name, ns); + ++asked[fqn]; + return with_endpoints.count(fqn) > 0; + }; + } +}; + +const Clock::time_point kStart{}; + +GraphNodeEntry running(const std::string & name, const std::string & ns) { + return GraphNodeEntry{name, ns, "/"}; +} + +GraphNodeEntry no_enclave(const std::string & name, const std::string & ns) { + return GraphNodeEntry{name, ns, ""}; +} + +} // namespace + +TEST(GraphNodeList, FqnOfRootAndNestedNamespaces) { + EXPECT_EQ(graph_node_fqn("a", "/"), "/a"); + EXPECT_EQ(graph_node_fqn("a", ""), "/a"); + EXPECT_EQ(graph_node_fqn("a", "/ns/sub"), "/ns/sub/a"); +} + +TEST(GraphNodeListReader, EntryWithEnclaveIsListedWithoutAnEndpointQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({running("quiet", "/ns")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"quiet", "/ns"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); + EXPECT_EQ(reader.remembered(), 0u); +} + +TEST(GraphNodeListReader, EntryWithoutEnclaveOfANodeNeverSeenRunningIsListedWithoutAQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({no_enclave("far", "/router")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"far", "/router"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); + EXPECT_EQ(reader.remembered(), 0u); +} + +TEST(GraphNodeListReader, LeftoverOfANodeSeenRunningIsLeftOutUnlessItHasEndpoints) { + GraphNodeListReader reader; + EndpointTable table; + reader.filter({running("gone", "/ns"), running("bridged", "/ns")}, kStart, table.probe()); + + table.with_endpoints = {"/ns/bridged"}; + auto list = + reader.filter({no_enclave("gone", "/ns"), no_enclave("bridged", "/ns")}, kStart + seconds(1), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"bridged", "/ns"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/ns/gone"})); + EXPECT_EQ(table.asked, (std::map{{"/ns/bridged", 1}, {"/ns/gone", 1}})); +} + +TEST(GraphNodeListReader, EntryWithoutEnclaveNextToOneWithAnEnclaveIsDroppedWithoutAQuery) { + GraphNodeListReader reader; + EndpointTable table; + auto list = reader.filter({no_enclave("node", "/"), running("node", "/")}, kStart, table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"node", "/"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); +} + +TEST(GraphNodeListReader, EndpointsAreAskedOncePerNameAndGraphOrderIsKept) { + GraphNodeListReader reader; + EndpointTable table; + reader.filter({running("x", "/a"), running("x", "/b")}, kStart, table.probe()); + table.with_endpoints = {"/b/x"}; + auto list = reader.filter( + {no_enclave("x", "/a"), running("y", "/"), no_enclave("x", "/b"), no_enclave("x", "/a"), no_enclave("x", "/b")}, + kStart + seconds(1), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"y", "/"}, {"x", "/b"}, {"x", "/b"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/a/x"})); + EXPECT_EQ(table.asked, (std::map{{"/a/x", 1}, {"/b/x", 1}})); +} + +TEST(GraphNodeListReader, AbsenceIsCountedFromTheFirstReadThatFindsTheNameAbsent) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + // The next read comes long after the last one that listed the node; the absence starts here. + reader.filter({}, kStart + seconds(60), table.probe()); + reader.filter({}, kStart + seconds(69), table.probe()); + auto list = + reader.filter({no_enclave("n", "/")}, kStart + seconds(69) + std::chrono::milliseconds(500), table.probe()); + EXPECT_TRUE(list.nodes.empty()); + EXPECT_EQ(list.leftovers, (std::vector{"/n"})); +} + +TEST(GraphNodeListReader, NameAbsentLongerThanTheHoldIsForgottenAndItsLeftoverListed) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + reader.filter({}, kStart + seconds(1), table.probe()); + reader.filter({}, kStart + seconds(11), table.probe()); + EXPECT_EQ(reader.remembered(), 1u); + reader.filter({}, kStart + seconds(12), table.probe()); + EXPECT_EQ(reader.remembered(), 0u); + auto list = reader.filter({no_enclave("n", "/")}, kStart + seconds(13), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"n", "/"}})); + EXPECT_TRUE(list.leftovers.empty()); + EXPECT_TRUE(table.asked.empty()); +} + +TEST(GraphNodeListReader, ListedLeftoverIsRememberedForAsLongAsItIsListed) { + GraphNodeListReader reader(seconds(10), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + for (int i = 1; i <= 100; ++i) { + auto list = reader.filter({no_enclave("n", "/")}, kStart + seconds(i), table.probe()); + ASSERT_EQ(list.leftovers, (std::vector{"/n"})) << "read " << i; + } +} + +TEST(GraphNodeListReader, NamesRunningNowDoNotCountAgainstTheCapacity) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + std::vector graph; + graph.reserve(9); + for (int i = 0; i < 8; ++i) { + graph.push_back(running("live_" + std::to_string(i), "/")); + } + graph.push_back(running("gone", "/")); + reader.filter(graph, kStart, table.probe()); + reader.filter(graph, kStart + seconds(1), table.probe()); + EXPECT_EQ(reader.remembered(), 0u); + + graph.back() = no_enclave("gone", "/"); + for (int i = 2; i <= 20; ++i) { + auto list = reader.filter(graph, kStart + seconds(i), table.probe()); + ASSERT_EQ(list.leftovers, (std::vector{"/gone"})) << "read " << i; + ASSERT_EQ(list.nodes.size(), 8u) << "read " << i; + } + EXPECT_EQ(reader.remembered(), 1u); +} + +TEST(GraphNodeListReader, ANameThatRunsAgainIsNoLongerDeparted) { + GraphNodeListReader reader(seconds(100), 16); + EndpointTable table; + reader.filter({running("n", "/")}, kStart, table.probe()); + reader.filter({no_enclave("n", "/")}, kStart + seconds(1), table.probe()); + EXPECT_EQ(reader.remembered(), 1u); + auto list = reader.filter({no_enclave("n", "/"), running("n", "/")}, kStart + seconds(2), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"n", "/"}})); + EXPECT_EQ(reader.remembered(), 0u); + // It departs again: the leftover of the earlier run is left out once more. + list = reader.filter({no_enclave("n", "/")}, kStart + seconds(3), table.probe()); + EXPECT_TRUE(list.nodes.empty()); + EXPECT_EQ(list.leftovers, (std::vector{"/n"})); +} + +TEST(GraphNodeListReader, PastTheCapacityTheNameUnlistedLongestIsForgottenFirst) { + GraphNodeListReader reader(seconds(100), 3); + EndpointTable table; + reader.filter({running("listed", "/"), running("early", "/"), running("late", "/"), running("new", "/")}, kStart, + table.probe()); + // listed departs first and stays listed through a leftover; early and late depart with no entry left. + reader.filter({no_enclave("listed", "/"), running("early", "/"), running("late", "/"), running("new", "/")}, + kStart + seconds(1), table.probe()); + reader.filter({no_enclave("listed", "/"), running("late", "/"), running("new", "/")}, kStart + seconds(2), + table.probe()); + reader.filter({no_enclave("listed", "/"), running("new", "/")}, kStart + seconds(3), table.probe()); + EXPECT_EQ(reader.remembered(), 3u); + + // new departs past the capacity: early has had no entry for longest, so it is forgotten first, + // although listed departed before it. + auto list = reader.filter({no_enclave("listed", "/"), no_enclave("new", "/")}, kStart + seconds(4), table.probe()); + EXPECT_EQ(reader.remembered(), 3u); + EXPECT_EQ(list.leftovers, (std::vector{"/listed", "/new"})); + list = reader.filter( + {no_enclave("listed", "/"), no_enclave("early", "/"), no_enclave("late", "/"), no_enclave("new", "/")}, + kStart + seconds(5), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"early", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/listed", "/late", "/new"})); +} + +TEST(GraphNodeListReader, PastTheCapacityAmongListedNamesTheOneThatDepartedLongestAgoIsForgottenFirst) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + reader.filter({running("old", "/")}, kStart, table.probe()); + reader.filter({no_enclave("old", "/"), running("mid", "/")}, kStart + seconds(1), table.probe()); + reader.filter({no_enclave("old", "/"), no_enclave("mid", "/"), running("new", "/")}, kStart + seconds(2), + table.probe()); + EXPECT_EQ(reader.remembered(), 2u); + + auto list = reader.filter({no_enclave("old", "/"), no_enclave("mid", "/"), no_enclave("new", "/")}, + kStart + seconds(3), table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"old", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/mid", "/new"})); +} + +TEST(GraphNodeListReader, PastTheCapacityANameThatDepartsWithNoEntryLeftIsForgottenBeforeListedLeftovers) { + GraphNodeListReader reader(seconds(100), 2); + EndpointTable table; + reader.filter({running("a", "/"), running("b", "/"), running("c", "/")}, kStart, table.probe()); + reader.filter({no_enclave("a", "/"), no_enclave("b", "/"), running("c", "/")}, kStart + seconds(1), table.probe()); + // c departs and the graph lists no entry of it: of three departed names it is the one no entry lists. + auto list = reader.filter({no_enclave("a", "/"), no_enclave("b", "/")}, kStart + seconds(2), table.probe()); + EXPECT_EQ(reader.remembered(), 2u); + EXPECT_EQ(list.leftovers, (std::vector{"/a", "/b"})); + list = reader.filter({no_enclave("a", "/"), no_enclave("b", "/"), no_enclave("c", "/")}, kStart + seconds(3), + table.probe()); + EXPECT_EQ(list.nodes, (std::vector{{"c", "/"}})); + EXPECT_EQ(list.leftovers, (std::vector{"/a", "/b"})); +} + +TEST(LeftoverNodeReporter, ReportsOnTheReadWhereANodeStartsBeingLeftOut) { + LeftoverNodeReporter reporter(rclcpp::get_logger("test_graph_node_list")); + const GraphNodeList left_out{{}, {"/ns/gone"}}; + const GraphNodeList listed{{{"gone", "/ns"}}, {}}; + const GraphNodeList absent{}; + + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); + EXPECT_TRUE(reporter.report(left_out).empty()); + EXPECT_TRUE(reporter.report(left_out).empty()); + + EXPECT_TRUE(reporter.report(listed).empty()); + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); + + EXPECT_TRUE(reporter.report(absent).empty()); + EXPECT_EQ(reporter.report(left_out), (std::vector{"/ns/gone"})); +} + +// Against a real rcl graph: the one rclcpp error that means "the graph no longer lists this node" +// counts as no endpoints, and the error a shut-down context raises propagates. +TEST(GraphNodeHasEndpoints, NodeTheGraphDoesNotListHasNoneAndAShutDownContextThrows) { + auto context = std::make_shared(); + context->init(0, nullptr); + auto node = std::make_shared("_graph_node_list_probe", rclcpp::NodeOptions().context(context)); + const auto graph = node->get_node_graph_interface(); + + EXPECT_FALSE(graph_node_has_endpoints(*graph, "no_such_node", "/no_such_namespace")); + EXPECT_TRUE(graph_node_has_endpoints(*graph, node->get_name(), node->get_namespace())); + + context->shutdown("test"); + EXPECT_THROW(graph_node_has_endpoints(*graph, "no_such_node", "/no_such_namespace"), rclcpp::exceptions::RCLError); +} diff --git a/src/ros2_medkit_gateway/test/test_handler_context.cpp b/src/ros2_medkit_gateway/test/test_handler_context.cpp index 968f29469..f1dbbcd38 100644 --- a/src/ros2_medkit_gateway/test/test_handler_context.cpp +++ b/src/ros2_medkit_gateway/test/test_handler_context.cpp @@ -874,7 +874,7 @@ TEST(FilterInternalNodeAppsTest, FiltersLocalInternalNodes) { apps.push_back(another_internal); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 2u); ASSERT_EQ(apps.size(), 1u); @@ -896,7 +896,7 @@ TEST(FilterInternalNodeAppsTest, PreservesAllNormalNodes) { apps.push_back(a3); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); EXPECT_EQ(apps.size(), 3u); @@ -921,7 +921,7 @@ TEST(FilterInternalNodeAppsTest, FiltersPeerPrefixedInternalNodes) { routing["peer_subsystem___ros2cli_daemon"] = "peer_subsystem"; routing["peer_subsystem__lidar_driver"] = "peer_subsystem"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 1u); ASSERT_EQ(apps.size(), 1u); @@ -939,7 +939,7 @@ TEST(FilterInternalNodeAppsTest, DoesNotStripPrefixWithoutRoutingEntry) { apps.push_back(ambiguous); std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); ASSERT_EQ(apps.size(), 1u); @@ -950,7 +950,7 @@ TEST(FilterInternalNodeAppsTest, HandlesEmptyAppList) { std::vector apps; std::unordered_map routing; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); EXPECT_TRUE(apps.empty()); @@ -981,7 +981,7 @@ TEST(FilterInternalNodeAppsTest, MixedLocalAndRemoteInternalNodes) { routing["sub_b__actuator"] = "sub_b"; routing["sub_b___parameter_bridge"] = "sub_b"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 2u); ASSERT_EQ(apps.size(), 2u); @@ -1006,12 +1006,222 @@ TEST(FilterInternalNodeAppsTest, PeerPrefixMatchMustBeExact) { std::unordered_map routing; routing["my_peer__sensor"] = "my_peer"; - auto removed = filter_internal_node_apps(apps, routing); + auto removed = filter_internal_node_apps(apps, routing, "/ros2_medkit_gateway"); EXPECT_EQ(removed, 0u); ASSERT_EQ(apps.size(), 1u); } +namespace { + +App bound_app(const std::string & id, const std::string & fqn) { + App app; + app.id = id; + app.name = id; + app.bound_fqn = fqn; + return app; +} + +} // namespace + +TEST(FilterInternalNodeAppsTest, DropsTheGatewaysOwnHelperNodesButKeepsItsOwnNode) { + // The gateway creates "_sub", "_fault_clients" and + // "_lifecycle_state_reader" in its own process. None starts with '_', + // so runtime introspection returns them as ordinary apps and the gateway ends + // up listing its own plumbing as diagnosable. Its own node is a different + // case and stays: its ROS parameters are served as that App's configurations. + const std::string self_fqn = "/ros2_medkit_gateway"; + std::vector apps{ + bound_app("ros2_medkit_gateway", self_fqn), + bound_app("ros2_medkit_gateway_sub", self_fqn + "_sub"), + bound_app("ros2_medkit_gateway_fault_clients", self_fqn + "_fault_clients"), + bound_app("ros2_medkit_gateway_lifecycle_state_reader", self_fqn + "_lifecycle_state_reader"), + // Positive controls on the same harness: a similarly suffixed FOREIGN + // node, a node whose name merely extends the gateway's, and the + // fault_manager, which is a separate diagnosable component even when it + // shares the process. + bound_app("other_gateway_sub", "/other_gateway_sub"), + bound_app("ros2_medkit_gateway_monitor", self_fqn + "_monitor"), + bound_app("fault_manager", "/fault_manager"), + }; + + std::unordered_map routing; + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 3u); + std::set remaining; + for (const auto & app : apps) { + remaining.insert(app.id); + } + EXPECT_EQ(remaining, (std::set{"ros2_medkit_gateway", "other_gateway_sub", "ros2_medkit_gateway_monitor", + "fault_manager"})); +} + +TEST(FilterInternalNodeAppsTest, LeavesPeerHelperNodesToThePeer) { + // A remote entity carrying the same FQN belongs to the peer that reported + // it, so this gateway must not reach across and filter it. + const std::string self_fqn = "/ros2_medkit_gateway"; + std::vector apps{bound_app("sub_b__ros2_medkit_gateway_sub", self_fqn + "_sub")}; + + std::unordered_map routing; + routing["sub_b__ros2_medkit_gateway_sub"] = "sub_b"; + + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 0u); + ASSERT_EQ(apps.size(), 1u); +} + +TEST(IsOwnGatewayHelperNodeTest, MatchesTheThreeHelperSuffixesExactlyAndNothingElse) { + const std::string self_fqn = "/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_sub", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node(self_fqn + "_lifecycle_state_reader", self_fqn)); + + // The gateway's own node is not a helper: it carries the parameters SOVD + // serves as configurations, so it stays a diagnosable App. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, self_fqn)); + + // Prefix neighbours are genuine peers, not ours. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn + "_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn + "2", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/other" + self_fqn + "_sub", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/fault_manager", self_fqn)); + + // An unknown self FQN claims nothing. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, "")); + EXPECT_FALSE(is_own_gateway_helper_node("", self_fqn)); +} + +TEST(IsOwnGatewayHelperNodeTest, RecognizesHelpersOfANamespacedGateway) { + // A gateway moved on its own - `-r :__ns:=/subsystem_b` - keeps the + // subscription node with it, because the executor passes the gateway's + // namespace, while the fault-client and lifecycle-reader nodes are built from + // the node name alone and stay in the process default namespace. All three + // are still this gateway's plumbing. + const std::string self_fqn = "/subsystem_b/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_sub", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // A process-wide namespace remap moves all four together, so the in-namespace + // spelling has to keep working for the same two. + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // The subscription node always shares the gateway's namespace, so a + // root-namespace one belongs to a different gateway process and is that + // gateway's own filter's business. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_sub", self_fqn)); + + // Still nothing else: a peer in either namespace, and the gateway itself. + EXPECT_FALSE(is_own_gateway_helper_node(self_fqn, self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_b/ros2_medkit_gateway_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_monitor", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_b/fault_manager", self_fqn)); +} + +TEST(FilterInternalNodeAppsTest, ReportsOnlyDeclaredAppsItDropsAsHelpers) { + // A runtime-discovered helper app is what this filter exists to remove, and + // saying so on every refresh would be noise. A DECLARED one is a manifest or + // plugin entity being overridden, which the caller has to be able to log. + const std::string self_fqn = "/ros2_medkit_gateway"; + App runtime_helper = bound_app("ros2_medkit_gateway_sub", self_fqn + "_sub"); + runtime_helper.source = "heuristic"; + App declared_helper = bound_app("plc_bridge", self_fqn + "_fault_clients"); + declared_helper.source = "manifest"; + std::vector apps{runtime_helper, declared_helper}; + + std::unordered_map routing; + std::vector dropped_declared; + auto removed = filter_internal_node_apps(apps, routing, self_fqn, &dropped_declared); + + EXPECT_EQ(removed, 2u); + EXPECT_TRUE(apps.empty()); + ASSERT_EQ(dropped_declared.size(), 1u); + EXPECT_EQ(dropped_declared[0], "plc_bridge -> /ros2_medkit_gateway_fault_clients"); +} + +TEST(IsOwnGatewayHelperNodeTest, RecognizesAForeignRootHelperAsPlumbing) { + // Two gateways, both left at the default node name, one moved into a + // namespace: the namespaced one's root-dwelling helpers and the root one's + // are the SAME literal FQN, because both are built from the same node name in + // the same namespace. Nothing in the name can tell them apart, so a + // helper-named node in the root namespace is treated as plumbing whichever + // process created it. That is the deliberate trade: the alternative is + // serving and counting one's own helper nodes in every namespaced + // deployment, and a node named "_fault_clients" is plumbing in + // either case - what it is does not depend on who owns it. + const std::string self_fqn = "/subsystem_a/ros2_medkit_gateway"; + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_fault_clients", self_fqn)); + EXPECT_TRUE(is_own_gateway_helper_node("/ros2_medkit_gateway_lifecycle_state_reader", self_fqn)); + + // The subscription node is the one case the name DOES settle: it always + // follows its gateway's namespace, so a root-namespace one is another + // process's and stays visible. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway_sub", self_fqn)); + + // A foreign gateway's own node is never plumbing, in any namespace. + EXPECT_FALSE(is_own_gateway_helper_node("/ros2_medkit_gateway", self_fqn)); + EXPECT_FALSE(is_own_gateway_helper_node("/subsystem_c/ros2_medkit_gateway", self_fqn)); + + // A differently NAMED gateway's helpers are not ours: the bare name has to + // match, so the rule does not reach across to a peer that was renamed. + EXPECT_FALSE(is_own_gateway_helper_node("/other_gateway_fault_clients", self_fqn)); +} + +TEST(RememberDroppedDeclaredAppsTest, ReportsTheSetOnceUntilItChanges) { + // The condition is a static misconfiguration and the caller runs on every + // refresh, so the same set must be reported once, not once per second. + std::set remembered; + + const std::vector one{"plc_bridge -> /ros2_medkit_gateway_fault_clients"}; + EXPECT_TRUE(remember_dropped_declared_apps(one, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(one, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(one, remembered)); + + // A changed set is news again. + const std::vector two{"plc_bridge -> /ros2_medkit_gateway_fault_clients", + "aux_bridge -> /ros2_medkit_gateway_sub"}; + EXPECT_TRUE(remember_dropped_declared_apps(two, remembered)); + EXPECT_FALSE(remember_dropped_declared_apps(two, remembered)); + + // Order is not a change: the caller builds this list from a vector whose + // order follows discovery, which is not stable between refreshes. + const std::vector two_reordered{"aux_bridge -> /ros2_medkit_gateway_sub", + "plc_bridge -> /ros2_medkit_gateway_fault_clients"}; + EXPECT_FALSE(remember_dropped_declared_apps(two_reordered, remembered)); + + // Clearing says nothing, and it is remembered, so the same set coming back + // is reported again. + EXPECT_FALSE(remember_dropped_declared_apps({}, remembered)); + EXPECT_TRUE(remember_dropped_declared_apps(one, remembered)); +} + +TEST(FilterInternalNodeAppsTest, DropsTheHelpersOfANamespacedGateway) { + // The same split at the level callers see: two of the three helpers of a + // namespaced gateway live in the root namespace, and without them being + // recognised the gateway lists its own plumbing. + const std::string self_fqn = "/subsystem_b/ros2_medkit_gateway"; + std::vector apps{ + bound_app("ros2_medkit_gateway", self_fqn), + bound_app("ros2_medkit_gateway_sub", "/subsystem_b/ros2_medkit_gateway_sub"), + bound_app("ros2_medkit_gateway_fault_clients", "/ros2_medkit_gateway_fault_clients"), + bound_app("ros2_medkit_gateway_lifecycle_state_reader", "/ros2_medkit_gateway_lifecycle_state_reader"), + bound_app("other_gateway", "/subsystem_c/other_gateway"), + }; + + std::unordered_map routing; + auto removed = filter_internal_node_apps(apps, routing, self_fqn); + + EXPECT_EQ(removed, 3u); + std::set remaining; + for (const auto & app : apps) { + remaining.insert(app.id); + } + EXPECT_EQ(remaining, (std::set{"ros2_medkit_gateway", "other_gateway"})); +} + // ============================================================================= // Area fault/log aggregation handler tests (via REST API) // ============================================================================= diff --git a/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp b/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp index 9cbea1b35..08e611979 100644 --- a/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_lifecycle_handlers.cpp @@ -329,6 +329,9 @@ class EntityDetailStatusLinkTest : public ::testing::Test { } static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } } void SetUp() override { @@ -471,6 +474,9 @@ class LifecycleHandlersWithProviderTest : public ::testing::Test { } static void TearDownTestSuite() { + if (rclcpp::ok()) { + rclcpp::shutdown(); + } } void SetUp() override { diff --git a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp index e5ec51125..f95e73995 100644 --- a/src/ros2_medkit_gateway/test/test_operation_handlers.cpp +++ b/src/ros2_medkit_gateway/test/test_operation_handlers.cpp @@ -524,10 +524,14 @@ class OperationHandlersFixtureTest : public ::testing::Test { return async_ptr->id; } - ActionGoalInfo get_tracked_goal_or_fail(const std::string & execution_id) { + // Returns the optional, so a missing goal reaches the caller as an empty + // value and the caller's ASSERT stops the test with a message. A non-fatal + // expectation followed by an unconditional `*goal_info` here would be + // undefined behaviour. + std::optional get_tracked_goal_or_fail(const std::string & execution_id) { auto goal_info = gateway_node_->get_operation_manager()->get_tracked_goal(execution_id); EXPECT_TRUE(goal_info.has_value()); - return *goal_info; + return goal_info; } CorsConfig cors_{}; @@ -978,7 +982,15 @@ TEST_F(OperationHandlersFixtureTest, GetOperationResolvesAQualifiedIdToItsMember } TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocation) { - const auto execution_id = create_action_execution(20); + // The fixture server pushes one sequence element per 100 ms tick and succeeds + // at the requested length, so the order sets how long the goal stays + // cancellable: 20 finishes on its own in under two seconds, which a loaded or + // instrumented runner can spend on the create plus the cancel round trip, and + // the goal is then SUCCEEDED with nothing left to stop. 50 is the largest + // order handle_goal accepts and buys about five seconds, which is the whole + // point of asking for it - the assertions below are about an accepted stop, + // not about how fast the machine is. + const auto execution_id = create_action_execution(50); ASSERT_FALSE(execution_id.empty()); auto raw_req = @@ -989,7 +1001,9 @@ TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocati body.capability = "stop"; auto result = handlers_->update_execution(typed, body); - auto goal_info = get_tracked_goal_or_fail(execution_id); + auto tracked = get_tracked_goal_or_fail(execution_id); + ASSERT_TRUE(tracked.has_value()); + const auto & goal_info = *tracked; if (result.has_value()) { const auto & exec = result.value().first.value; @@ -1007,8 +1021,26 @@ TEST_F(OperationHandlersFixtureTest, UpdateExecutionStopReturnsAcceptedAndLocati EXPECT_TRUE(has_location); ASSERT_TRUE(exec.id.has_value()); EXPECT_EQ(*exec.id, execution_id); - EXPECT_EQ(exec.status, "running"); - EXPECT_EQ(goal_info.status, ActionGoalStatus::CANCELING); + + // An accepted stop promises the goal is on its way out: CANCELING while + // the server winds down, CANCELED once it has. It never promises which of + // the two the caller observes, and a server that cancels within the + // round trip lands on CANCELED directly. What it does rule out is a goal + // that is still running (ACCEPTED, EXECUTING) or one that completed + // anyway (SUCCEEDED): those mean the stop did not take. + EXPECT_TRUE(goal_info.status == ActionGoalStatus::CANCELING || goal_info.status == ActionGoalStatus::CANCELED) + << "tracked status after an accepted stop: " << ros2_medkit_gateway::action_status_to_string(goal_info.status); + + // The body renders the status the handler read, which is at or before the + // one read above - a goal only moves CANCELING -> CANCELED, never back. So + // a goal still CANCELING here cannot have been CANCELED when the handler + // looked, which pins the body exactly; a goal already CANCELED admits + // either rendering. + if (goal_info.status == ActionGoalStatus::CANCELING) { + EXPECT_EQ(exec.status, "running"); + } else { + EXPECT_TRUE(exec.status == "running" || exec.status == "failed") << "execution status in body: " << exec.status; + } } else { // The fixture's action server always ACCEPTS cancels, so the only // realistic failure here is a lost/late CancelGoal response whose diff --git a/src/ros2_medkit_gateway/test/test_runtime_linker.cpp b/src/ros2_medkit_gateway/test/test_runtime_linker.cpp index 2fc04afdc..f8ceef122 100644 --- a/src/ros2_medkit_gateway/test/test_runtime_linker.cpp +++ b/src/ros2_medkit_gateway/test/test_runtime_linker.cpp @@ -14,6 +14,13 @@ #include +#include +#include +#include +#include + +#include + #include "ros2_medkit_gateway/discovery/manifest/runtime_linker.hpp" using namespace ros2_medkit_gateway::discovery; @@ -804,6 +811,71 @@ TEST_F(RuntimeLinkerTest, MergedInput_PreservesOrphanRuntimeApps) { EXPECT_EQ(result.orphan_nodes[0], "/nav/planner"); } +// ============================================================================= +// The gateway's own helper nodes: skipped by the same setting that filters them +// out of the served apps +// ============================================================================= + +TEST(RuntimeLinkerHelperNodes, TheOrphanSkipFollowsTheAppFilterSetting) { + // discovery.runtime.filter_internal_nodes decides whether the gateway's own + // in-process helper nodes are served as apps. While it is on they are not + // entities, so listing them as unmanifested would tell the operator to declare + // nodes that can never become apps. Turn it off - which the config docs say + // re-exposes them - and they ARE served, so leaving them out of orphan_nodes + // hides from /health's unmanifested_nodes exactly the nodes the operator + // turned the filter off in order to see. + const bool owned_rclcpp = !rclcpp::ok(); + if (owned_rclcpp) { + rclcpp::init(0, nullptr); + } + auto node = std::make_shared("runtime_linker_helper_gateway"); + const std::string self_fqn = node->get_fully_qualified_name(); + // The subscription helper node the gateway creates in its own namespace. + const std::string helper_fqn = self_fqn + "_sub"; + + App helper; + helper.id = "helper"; + helper.name = "helper"; + helper.source = "heuristic"; + helper.is_online = true; + helper.bound_fqn = helper_fqn; + + App foreign; + foreign.id = "planner"; + foreign.name = "planner"; + foreign.source = "heuristic"; + foreign.is_online = true; + foreign.bound_fqn = "/nav/planner"; + + const std::vector runtime_apps{helper, foreign}; + ManifestConfig config; + + const auto contains = [](const std::vector & orphans, const std::string & fqn) { + return std::find(orphans.begin(), orphans.end(), fqn) != orphans.end(); + }; + + { + RuntimeLinker filtering(node.get(), /*filter_internal_nodes=*/true); + const auto result = filtering.link({}, runtime_apps, config); + EXPECT_FALSE(contains(result.orphan_nodes, helper_fqn)) + << "a helper node the app filter removes was reported as unmanifested"; + EXPECT_TRUE(contains(result.orphan_nodes, foreign.bound_fqn.value())) + << "the control: a real unmanifested node is still reported"; + } + { + RuntimeLinker serving(node.get(), /*filter_internal_nodes=*/false); + const auto result = serving.link({}, runtime_apps, config); + EXPECT_TRUE(contains(result.orphan_nodes, helper_fqn)) + << "with the app filter off the helper is served as an app, so it is unmanifested and must be counted"; + EXPECT_TRUE(contains(result.orphan_nodes, foreign.bound_fqn.value())); + } + + node.reset(); + if (owned_rclcpp) { + rclcpp::shutdown(); + } +} + int main(int argc, char ** argv) { testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_integration_tests/CMakeLists.txt b/src/ros2_medkit_integration_tests/CMakeLists.txt index dab635448..2345cbd00 100644 --- a/src/ros2_medkit_integration_tests/CMakeLists.txt +++ b/src/ros2_medkit_integration_tests/CMakeLists.txt @@ -162,6 +162,20 @@ install(TARGETS topics_test_plugin LIBRARY DESTINATION lib/${PROJECT_NAME}) if(BUILD_TESTING) find_package(launch_testing_ament_cmake REQUIRED) + # Graph fixtures for the test_graph_leftover_nodes suites, installed for tests in any package. + # ghost_node_injector writes rmw_dds_common's discovery message itself. + find_package(rcl REQUIRED) + find_package(rmw REQUIRED) + find_package(rmw_dds_common REQUIRED) + find_package(rosidl_typesupport_cpp REQUIRED) + add_executable(ghost_node_injector demo_nodes/ghost_node_injector.cpp) + target_include_directories(ghost_node_injector PRIVATE ${_demo_include_dir}) + medkit_target_dependencies(ghost_node_injector rclcpp rcl rmw rmw_dds_common rosidl_typesupport_cpp std_msgs) + add_executable(endpointless_node demo_nodes/endpointless_node.cpp) + target_include_directories(endpointless_node PRIVATE ${_demo_include_dir}) + medkit_target_dependencies(endpointless_node rclcpp rcl) + install(TARGETS ghost_node_injector endpointless_node DESTINATION lib/${PROJECT_NAME}) + # Lint the Python that makes up this package. Without this the whole package # sits outside `colcon test -L linter`, so the documented lint command passes # while none of these files are checked, and the pre-commit hooks are the @@ -348,6 +362,8 @@ if(BUILD_TESTING) test_graph_provider_sse 300 test_peer_recovery 300 test_triggers_restore_before_discovery 300 + # Seven cases share one gateway, and their scaled give-up bounds add up past 120s. + test_graph_leftover_nodes 300 # Three discovery budgets in setUpClass before the first case runs, and a # broken merge spends all of them. The glob default cuts that short and # reports a timeout with no test name, which is the one answer that says diff --git a/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp b/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp new file mode 100644 index 000000000..89fb1b599 --- /dev/null +++ b/src/ros2_medkit_integration_tests/demo_nodes/endpointless_node.cpp @@ -0,0 +1,72 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +/** + * @file endpointless_node.cpp + * @brief A running node with no endpoints of its own + * + * Test fixture: a bare rcl node with /rosout off, because an rclcpp::Node always + * subscribes to /parameter_events. Set the name with `-r __node:=... -r __ns:=...`. + * Runs until SIGINT or SIGTERM. + */ + +#include +#include + +#include +#include +#include + +#include "ros2_medkit_integration_tests/crash_backtrace.hpp" + +int main(int argc, char ** argv) { + ros2_medkit_integration_tests::install_crash_backtrace(); + + // Blocked before any thread exists, so every thread inherits the mask and the + // signal waits for sigwait() below instead of rclcpp's handler. + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + if (pthread_sigmask(SIG_BLOCK, &mask, nullptr) != 0) { + return 1; + } + rclcpp::init(argc, argv, rclcpp::InitOptions(), rclcpp::SignalHandlerOptions::None); + + rcl_node_t node = rcl_get_zero_initialized_node(); + rcl_node_options_t options = rcl_node_get_default_options(); + options.enable_rosout = false; + auto context = rclcpp::contexts::get_global_default_context()->get_rcl_context(); + if (rcl_node_init(&node, "endpointless_node", "", context.get(), &options) != RCL_RET_OK) { + std::fprintf(stderr, "rcl_node_init failed: %s\n", rcl_get_error_string().str); + rclcpp::shutdown(); + return 1; + } + + int signum = 0; + while (sigwait(&mask, &signum) != 0) { + // Interrupted before a signal was taken: wait again. + } + + int exit_code = 0; + if (rcl_node_fini(&node) != RCL_RET_OK) { + std::fprintf(stderr, "rcl_node_fini failed: %s\n", rcl_get_error_string().str); + exit_code = 1; + } + if (rcl_node_options_fini(&options) != RCL_RET_OK) { + exit_code = 1; + } + rclcpp::shutdown(); + return exit_code; +} diff --git a/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp b/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp new file mode 100644 index 000000000..c230551a0 --- /dev/null +++ b/src/ros2_medkit_integration_tests/demo_nodes/ghost_node_injector.cpp @@ -0,0 +1,577 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +/** + * @file ghost_node_injector.cpp + * @brief Puts nodes into the ROS graph that no running participant stands behind + * + * Test fixture. The DDS RMWs create a graph node entry for any participant GID a + * `ros_discovery_info` message names. + * + * --ghost a message for an unowned GID (starts 00 00) listing the node with + * no endpoints: the shape of a node behind a DDS router. + * --backed the same for a GID starting 00 01 that names a publisher this + * process owns: a node whose participant announces no enclave. + * --leftover leaves a leftover behind a real node: + * 1. creates the node in its own context and captures its participant's last + * discovery message, then prints `ghost_node_injector: leftover_ready + * participant_gid=`; + * 2. with --announce , on an `announce` stdin line, republishes it with + * more nodes `_` and prints `ghost_node_injector: + * leftover_announced count= acked=`. Send it only after the + * target graph lists the node, or the original message can replace it; + * 3. on a `leave` line, destroys the node and its context and waits until this + * process's graph drops it; + * 4. after --delay (default 0), or on a `publish` line after `leave`, + * republishes the message and prints `ghost_node_injector: + * leftover_published matched_subscriptions= acked=`. + * + * The participant GID is the node's publisher GUID prefix plus entity id 00 00 01 c1. + * Step 1 waits for a discovery message with that GID, so a wrong derivation fails there. + * --ghost and --backed print `ghost_node_injector: matched_subscriptions= entries= + * acked=`, and send one message per node in command-line order. Messages go + * through the rmw layer on a reliable, transient-local, keep-all writer, so late graphs + * receive them in that order and they outlive this process. rcl would rename the topic to + * `/ros_discovery_info`, which the RMWs do not read. + * + * Runs until SIGINT or SIGTERM. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_integration_tests/demo_node_main.hpp" + +namespace { + +using ParticipantEntitiesInfo = rmw_dds_common::msg::ParticipantEntitiesInfo; +using NodeEntitiesInfo = rmw_dds_common::msg::NodeEntitiesInfo; +using Gid = rmw_dds_common::msg::Gid; + +constexpr const char * kUsage = + "usage: ghost_node_injector [--ghost ]... [--backed ]... " + "[--leftover [--announce ] [--delay ]]"; + +struct InjectedNode { + std::string name; + std::string ns; + bool backed; +}; + +struct Arguments { + std::vector injected; + std::optional leftover; + size_t announce{0}; + double delay_sec{0.0}; +}; + +InjectedNode parse_fqn(const std::string & fqn, bool backed) { + const auto slash = fqn.rfind('/'); + if (fqn.empty() || fqn.front() != '/' || slash == std::string::npos || slash + 1 == fqn.size()) { + throw std::invalid_argument("not a fully qualified node name: '" + fqn + "'"); + } + std::string ns = slash == 0 ? "/" : fqn.substr(0, slash); + return InjectedNode{fqn.substr(slash + 1), std::move(ns), backed}; +} + +Arguments parse_arguments(const std::vector & args) { + Arguments parsed; + bool announce_given = false; + bool delay_given = false; + // args[0] is the program name. + for (size_t i = 1; i < args.size(); ++i) { + const auto & flag = args[i]; + if (i + 1 >= args.size()) { + throw std::invalid_argument(kUsage); + } + const auto & value = args[++i]; + if (flag == "--ghost" || flag == "--backed") { + parsed.injected.push_back(parse_fqn(value, flag == "--backed")); + } else if (flag == "--leftover" && !parsed.leftover) { + parsed.leftover = parse_fqn(value, false); + } else if (flag == "--announce") { + parsed.announce = std::stoul(value); + announce_given = true; + } else if (flag == "--delay") { + parsed.delay_sec = std::stod(value); + // In-range test negated so NaN fails it. Do not apply clang-tidy's De Morgan rewrite. + if (!(parsed.delay_sec >= 0.0 && parsed.delay_sec <= 3600.0)) { // NOLINT(readability-simplify-boolean-expr) + throw std::invalid_argument("--delay must be between 0 and 3600 seconds"); + } + delay_given = true; + } else { + throw std::invalid_argument(kUsage); + } + } + if ((parsed.injected.empty() && !parsed.leftover) || ((announce_given || delay_given) && !parsed.leftover)) { + throw std::invalid_argument(kUsage); + } + return parsed; +} + +std::string to_hex(const Gid & gid) { + static constexpr std::array kDigits{'0', '1', '2', '3', '4', '5', '6', '7', + '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}; + std::string hex; + for (const auto byte : gid.data) { + hex.push_back(kDigits[static_cast(byte >> 4U)]); + hex.push_back(kDigits[static_cast(byte & 0x0FU)]); + } + return hex; +} + +class GhostNodeInjector : public rclcpp::Node { + public: + explicit GhostNodeInjector(Arguments args) : Node("_ghost_node_injector"), args_(std::move(args)) { + backing_publisher_ = create_publisher("~/backing", rclcpp::QoS(1)); + + rmw_node_ = rcl_node_get_rmw_handle(get_node_base_interface()->get_rcl_node_handle()); + rmw_qos_profile_t qos = rmw_qos_profile_default; + qos.history = RMW_QOS_POLICY_HISTORY_KEEP_ALL; + qos.durability = RMW_QOS_POLICY_DURABILITY_TRANSIENT_LOCAL; + qos.reliability = RMW_QOS_POLICY_RELIABILITY_RELIABLE; + qos.avoid_ros_namespace_conventions = true; + const auto * type_support = + rosidl_typesupport_cpp::get_message_type_support_handle(); + rmw_publisher_options_t publisher_options = rmw_get_default_publisher_options(); + discovery_publisher_ = + rmw_create_publisher(rmw_node_, type_support, "ros_discovery_info", &qos, &publisher_options); + if (discovery_publisher_ == nullptr) { + throw std::runtime_error(std::string("rmw_create_publisher(ros_discovery_info) failed: ") + + rmw_get_error_string().str); + } + + if (args_.leftover) { + rmw_subscription_options_t subscription_options = rmw_get_default_subscription_options(); + discovery_subscription_ = + rmw_create_subscription(rmw_node_, type_support, "ros_discovery_info", &qos, &subscription_options); + if (discovery_subscription_ == nullptr) { + throw std::runtime_error(std::string("rmw_create_subscription(ros_discovery_info) failed: ") + + rmw_get_error_string().str); + } + start_leftover_node(); + if (fcntl(STDIN_FILENO, F_SETFL, fcntl(STDIN_FILENO, F_GETFL) | O_NONBLOCK) != 0) { + throw std::runtime_error("could not make stdin non-blocking"); + } + } + + timer_ = create_wall_timer(std::chrono::milliseconds(20), [this]() { + tick_injected(); + tick_leftover(); + }); + } + + ~GhostNodeInjector() override { + timer_.reset(); + stop_leftover_node(); + if (discovery_subscription_ != nullptr) { + if (rmw_destroy_subscription(rmw_node_, discovery_subscription_) != RMW_RET_OK) { + std::fprintf(stderr, "rmw_destroy_subscription(ros_discovery_info) failed: %s\n", rmw_get_error_string().str); + rmw_reset_error(); + } + discovery_subscription_ = nullptr; + } + if (discovery_publisher_ != nullptr) { + if (rmw_destroy_publisher(rmw_node_, discovery_publisher_) != RMW_RET_OK) { + std::fprintf(stderr, "rmw_destroy_publisher(ros_discovery_info) failed: %s\n", rmw_get_error_string().str); + rmw_reset_error(); + } + discovery_publisher_ = nullptr; + } + backing_publisher_.reset(); + } + + GhostNodeInjector(const GhostNodeInjector &) = delete; + GhostNodeInjector & operator=(const GhostNodeInjector &) = delete; + GhostNodeInjector(GhostNodeInjector &&) = delete; + GhostNodeInjector & operator=(GhostNodeInjector &&) = delete; + + private: + enum class LeftoverStep { kCapturing, kReady, kRemoving, kDelaying, kPublished, kFailed }; + + // ---- --ghost / --backed -------------------------------------------------------------- + + /// Publish once something is listening, then report once every matched reader has acknowledged. + void tick_injected() { + if (args_.injected.empty() || injected_reported_) { + return; + } + if (!injected_published_) { + size_t matched = 0; + if (rmw_publisher_count_matched_subscriptions(discovery_publisher_, &matched) != RMW_RET_OK || matched == 0 || + !resolve_backing_gid()) { + if (std::chrono::steady_clock::now() - started_ < kMatchTimeout) { + return; + } + report_injected(matched, false); + return; + } + publish_injected(); + injected_matched_ = matched; + injected_published_ = true; + injected_published_at_ = std::chrono::steady_clock::now(); + } + const bool acked = rmw_publisher_wait_for_all_acked(discovery_publisher_, rmw_time_t{0, 0}) == RMW_RET_OK; + if (acked || std::chrono::steady_clock::now() - injected_published_at_ >= kAckTimeout) { + report_injected(injected_matched_, acked); + } + } + + /// The backing publisher's graph GID. Not PublisherBase::get_gid(): Cyclone returns a local id there. + bool resolve_backing_gid() { + const auto gid = endpoint_gid(*this, backing_publisher_->get_topic_name()); + if (!gid) { + return false; + } + backing_gid_ = *gid; + return true; + } + + void publish_injected() { + std::random_device seed; + std::mt19937 generator(seed()); + std::uniform_int_distribution byte(0, 255); + + for (const auto & node : args_.injected) { + ParticipantEntitiesInfo message; + for (auto & b : message.gid.data) { + b = static_cast>(byte(generator)); + } + message.gid.data[0] = 0x00; + message.gid.data[1] = node.backed ? 0x01 : 0x00; + NodeEntitiesInfo info; + info.node_namespace = node.ns; + info.node_name = node.name; + if (node.backed) { + info.writer_gid_seq.push_back(backing_gid_); + } + message.node_entities_info_seq.push_back(std::move(info)); + publish(message); + } + } + + void report_injected(size_t matched, bool acked) { + std::printf("ghost_node_injector: matched_subscriptions=%zu entries=%zu acked=%s\n", matched, args_.injected.size(), + acked ? "true" : "false"); + std::fflush(stdout); + injected_reported_ = true; + } + + // ---- --leftover ---------------------------------------------------------------------- + + void start_leftover_node() { + rclcpp::InitOptions init_options; + init_options.auto_initialize_logging(false); + leftover_context_ = std::make_shared(); + leftover_context_->init(0, nullptr, init_options); + leftover_node_ = std::make_shared(args_.leftover->name, args_.leftover->ns, + rclcpp::NodeOptions().context(leftover_context_)); + leftover_marker_ = leftover_node_->create_publisher("~/leftover_marker", rclcpp::QoS(1)); + rclcpp::ExecutorOptions executor_options; + executor_options.context = leftover_context_; + leftover_executor_ = std::make_shared(executor_options); + leftover_executor_->add_node(leftover_node_); + leftover_spin_thread_ = std::thread([executor = leftover_executor_]() { + executor->spin(); + }); + } + + /// Destroy the node, which removes its participant, and shut its context down. + void stop_leftover_node() { + if (!leftover_context_) { + return; + } + leftover_executor_->cancel(); + if (leftover_spin_thread_.joinable()) { + leftover_spin_thread_.join(); + } + leftover_executor_->remove_node(leftover_node_); + leftover_executor_.reset(); + leftover_marker_.reset(); + leftover_node_.reset(); + leftover_context_->shutdown("leftover node leaves"); + leftover_context_.reset(); + } + + void tick_leftover() { + if (!args_.leftover) { + return; + } + const auto & target = *args_.leftover; + if (leftover_step_ != LeftoverStep::kCapturing) { + // Keep the keep-all reader empty; only the capture step reads what it takes. + drain_discovery_messages(); + } + switch (leftover_step_) { + case LeftoverStep::kCapturing: + capture_leftover_message(); + if (captured_) { + std::printf("ghost_node_injector: leftover_ready participant_gid=%s\n", to_hex(captured_->gid).c_str()); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kReady; + } else if (std::chrono::steady_clock::now() - started_ >= kMatchTimeout) { + fail("no discovery message for participant GID " + (derived_gid_ ? to_hex(*derived_gid_) : "(none)") + + " lists " + target.ns + "/" + target.name + " with its marker publisher"); + } + break; + case LeftoverStep::kReady: + read_stdin(); + if (!announced_ && args_.announce > 0 && stdin_buffer_.find("announce\n") != std::string::npos) { + announce(); + // No graph in this process sees its own participant's writes; reader acks stand in. + std::printf("ghost_node_injector: leftover_announced count=%zu acked=%s\n", args_.announce, + wait_for_acks() ? "true" : "false"); + std::fflush(stdout); + announced_ = true; + } + if (stdin_buffer_.find("leave\n") != std::string::npos) { + stop_leftover_node(); + left_at_ = std::chrono::steady_clock::now(); + leftover_step_ = LeftoverStep::kRemoving; + } + break; + case LeftoverStep::kRemoving: + if (!own_graph_lists(target)) { + removed_at_ = std::chrono::steady_clock::now(); + std::printf("ghost_node_injector: leftover_removed after_ms=%lld\n", + static_cast( + std::chrono::duration_cast(removed_at_ - left_at_).count())); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kDelaying; + } else if (std::chrono::steady_clock::now() - left_at_ >= kMatchTimeout) { + fail("this process's graph still lists the node after its context was shut down"); + } + break; + case LeftoverStep::kDelaying: + read_stdin(); + if (std::chrono::steady_clock::now() - removed_at_ >= std::chrono::duration(args_.delay_sec) || + stdin_buffer_.find("publish\n", stdin_buffer_.find("leave\n")) != std::string::npos) { + size_t matched = 0; + if (rmw_publisher_count_matched_subscriptions(discovery_publisher_, &matched) != RMW_RET_OK) { + matched = 0; + } + publish(stale_message_); + const bool acked = wait_for_acks(); + std::printf("ghost_node_injector: leftover_published matched_subscriptions=%zu acked=%s\n", matched, + acked ? "true" : "false"); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kPublished; + } + break; + case LeftoverStep::kPublished: + case LeftoverStep::kFailed: + break; + } + } + + /// Take the discovery messages received so far; keep the leftover participant's one that lists + /// the marker publisher, the node's last endpoint. + void capture_leftover_message() { + if (!derived_gid_) { + marker_gid_ = endpoint_gid(*leftover_node_, leftover_marker_->get_topic_name()); + if (!marker_gid_) { + return; + } + Gid participant_gid; + std::fill(participant_gid.data.begin(), participant_gid.data.end(), 0); + std::copy_n(marker_gid_->data.begin(), kGuidPrefixSize, participant_gid.data.begin()); + std::copy(kParticipantEntityId.begin(), kParticipantEntityId.end(), + participant_gid.data.begin() + kGuidPrefixSize); + derived_gid_ = participant_gid; + } + const auto & target = *args_.leftover; + while (true) { + ParticipantEntitiesInfo message; + bool taken = false; + if (rmw_take(discovery_subscription_, &message, &taken, nullptr) != RMW_RET_OK) { + rmw_reset_error(); + return; + } + if (!taken) { + return; + } + if (message.gid != *derived_gid_) { + continue; + } + const auto & nodes = message.node_entities_info_seq; + const bool lists_target = std::any_of(nodes.begin(), nodes.end(), [this, &target](const NodeEntitiesInfo & info) { + const auto & writers = info.writer_gid_seq; + return info.node_name == target.name && info.node_namespace == target.ns && + std::find(writers.begin(), writers.end(), *marker_gid_) != writers.end(); + }); + if (lists_target && !captured_) { + captured_ = message; + stale_message_ = message; + } + } + } + + void drain_discovery_messages() { + ParticipantEntitiesInfo message; + bool taken = true; + while (taken) { + if (rmw_take(discovery_subscription_, &message, &taken, nullptr) != RMW_RET_OK) { + rmw_reset_error(); + return; + } + } + } + + void announce() { + stale_message_ = *captured_; + for (size_t i = 0; i < args_.announce; ++i) { + NodeEntitiesInfo info; + info.node_namespace = args_.leftover->ns; + info.node_name = announced_name(i); + stale_message_.node_entities_info_seq.push_back(std::move(info)); + } + publish(stale_message_); + } + + std::string announced_name(size_t index) const { + auto digits = std::to_string(index); + if (digits.size() < kAnnouncedIndexWidth) { + digits.insert(0, kAnnouncedIndexWidth - digits.size(), '0'); + } + return args_.leftover->name + "_" + digits; + } + + /// Whether this process's graph lists the node with an enclave. + bool own_graph_lists(const InjectedNode & node) { + for (const auto & [name, ns, enclave] : get_node_graph_interface()->get_node_names_with_enclaves()) { + if (name == node.name && ns == node.ns && !enclave.empty()) { + return true; + } + } + return false; + } + + void read_stdin() { + std::array buffer{}; + while (true) { + const auto count = ::read(STDIN_FILENO, buffer.data(), buffer.size()); + if (count <= 0) { + return; + } + stdin_buffer_.append(buffer.data(), static_cast(count)); + } + } + + /// Whether every matched reader acknowledged what was published, within kAckTimeout. + bool wait_for_acks() { + return rmw_publisher_wait_for_all_acked(discovery_publisher_, + rmw_time_t{static_cast(kAckTimeout.count()), 0}) == RMW_RET_OK; + } + + void fail(const std::string & reason) { + std::printf("ghost_node_injector: leftover_failed %s\n", reason.c_str()); + std::fflush(stdout); + leftover_step_ = LeftoverStep::kFailed; + } + + // ---- shared -------------------------------------------------------------------------- + + /// The graph GID of the publisher `node` owns on `topic`, as `node`'s own graph keys it. + static std::optional endpoint_gid(rclcpp::Node & node, const std::string & topic) { + for (const auto & info : node.get_publishers_info_by_topic(topic)) { + if (info.node_name() == node.get_name() && info.node_namespace() == node.get_namespace()) { + Gid gid; + std::fill(gid.data.begin(), gid.data.end(), 0); + const auto & endpoint = info.endpoint_gid(); + std::copy_n(endpoint.begin(), std::min(endpoint.size(), gid.data.size()), gid.data.begin()); + return gid; + } + } + return std::nullopt; + } + + void publish(const ParticipantEntitiesInfo & message) { + if (rmw_publish(discovery_publisher_, &message, nullptr) != RMW_RET_OK) { + RCLCPP_ERROR(get_logger(), "rmw_publish failed: %s", rmw_get_error_string().str); + rmw_reset_error(); + } + } + + static constexpr std::chrono::seconds kMatchTimeout{30}; + static constexpr std::chrono::seconds kAckTimeout{10}; + static constexpr size_t kGuidPrefixSize = 12; + static constexpr size_t kAnnouncedIndexWidth = 5; + static constexpr std::array kParticipantEntityId{0x00, 0x00, 0x01, 0xc1}; + + Arguments args_; + rclcpp::Publisher::SharedPtr backing_publisher_; + Gid backing_gid_; + rmw_node_t * rmw_node_{nullptr}; + rmw_publisher_t * discovery_publisher_{nullptr}; + rmw_subscription_t * discovery_subscription_{nullptr}; + rclcpp::TimerBase::SharedPtr timer_; + const std::chrono::steady_clock::time_point started_{std::chrono::steady_clock::now()}; + + std::chrono::steady_clock::time_point injected_published_at_; + size_t injected_matched_{0}; + bool injected_published_{false}; + bool injected_reported_{false}; + + rclcpp::Context::SharedPtr leftover_context_; + rclcpp::Node::SharedPtr leftover_node_; + rclcpp::Publisher::SharedPtr leftover_marker_; + std::shared_ptr leftover_executor_; + std::thread leftover_spin_thread_; + LeftoverStep leftover_step_{LeftoverStep::kCapturing}; + std::optional marker_gid_; + std::optional derived_gid_; + std::optional captured_; + bool announced_{false}; + ParticipantEntitiesInfo stale_message_; + std::string stdin_buffer_; + std::chrono::steady_clock::time_point left_at_; + std::chrono::steady_clock::time_point removed_at_; +}; + +} // namespace + +int main(int argc, char ** argv) { + Arguments args; + try { + args = parse_arguments(rclcpp::remove_ros_arguments(argc, argv)); + } catch (const std::exception & e) { + std::fprintf(stderr, "%s\n", e.what()); + return 2; + } + return ros2_medkit_integration_tests::run_demo_node(argc, argv, [&args]() -> std::shared_ptr { + return std::make_shared(std::move(args)); + }); +} diff --git a/src/ros2_medkit_integration_tests/package.xml b/src/ros2_medkit_integration_tests/package.xml index 3ae4e8ae0..d2edc11b4 100644 --- a/src/ros2_medkit_integration_tests/package.xml +++ b/src/ros2_medkit_integration_tests/package.xml @@ -33,6 +33,10 @@ launch_testing launch_ros ament_index_python + rcl + rmw + rmw_dds_common + rosidl_typesupport_cpp python3-requests python3-jsonschema ament_cmake_flake8 diff --git a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py new file mode 100644 index 000000000..6a0e0deb7 --- /dev/null +++ b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/graph_fixtures.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Drive ``ghost_node_injector`` (demo_nodes/ghost_node_injector.cpp) and read a ROS graph. + +``--leftover`` leaves a real node's leftover in every graph on the domain; ``--ghost`` and +``--backed`` build the same empty-enclave shape for a node no graph saw running. +""" + +import os +import queue +import re +import signal +import subprocess +import threading +import time + +from ament_index_python.packages import get_package_prefix + +PACKAGE = 'ros2_medkit_integration_tests' + + +def fixture_path(name): + """Absolute path of an executable this package installs next to its demo nodes.""" + path = os.path.join(get_package_prefix(PACKAGE), 'lib', PACKAGE, name) + if not os.path.isfile(path): + raise FileNotFoundError(f'test fixture not installed: {path}') + return path + + +def split_fqn(fqn): + """(name, namespace) of a fully qualified node name, as a ROS graph reports them.""" + namespace, _, name = fqn.rpartition('/') + return name, namespace or '/' + + +def observed_enclaves(node, fqn): + """Enclave of every entry `node`'s ROS graph lists for `fqn`, in graph order.""" + name, namespace = split_fqn(fqn) + entries = node.get_node_names_and_namespaces_with_enclaves() + return [ + enclave + for node_name, node_namespace, enclave in entries + if node_name == name and node_namespace == namespace + ] + + +def wait_observed(node, fqn, predicate, timeout): + """Poll `node`'s graph until `predicate(enclaves of fqn)` holds; return whether it did.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(observed_enclaves(node, fqn)): + return True + time.sleep(0.05) + return False + + +def stop_process(proc, timeout): + """SIGTERM `proc` and wait for it, killing it if it outlives `timeout`.""" + if proc is None or proc.poll() is not None: + return + proc.send_signal(signal.SIGTERM) + try: + proc.wait(timeout=timeout) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=10) + + +class _FixtureProcess: + """One ghost_node_injector process, its output collected line by line.""" + + def __init__(self, args, env=None): + self.proc = subprocess.Popen( + [fixture_path('ghost_node_injector')] + args, env=env, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + self._lines = queue.Queue() + self._output = [] + threading.Thread(target=self._read, daemon=True).start() + + def _read(self): + for line in self.proc.stdout: + self._output.append(line) + self._lines.put(line) + + def wait_line(self, pattern, timeout): + """Return the first match of `pattern` on a line printed from now on, or None.""" + regex = re.compile(pattern) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + line = self._lines.get(timeout=0.1) + except queue.Empty: + if self.proc.poll() is not None and self._lines.empty(): + return None + continue + match = regex.search(line) + if match: + return match + return None + + def send(self, command): + self.proc.stdin.write(command + '\n') + self.proc.stdin.flush() + + def output(self): + return ''.join(self._output) + + def stop(self, timeout=30.0): + stop_process(self.proc, timeout) + + +class GhostInjection(_FixtureProcess): + """``--ghost`` and ``--backed`` nodes for GIDs no participant owns. + + The backed nodes go out before the ghosts, so a ROS graph that lists a ghost of this + injection also lists every backed node of it. + """ + + def __init__(self, ghosts=(), backed=(), env=None): + args = [] + for fqn in backed: + args += ['--backed', fqn] + for fqn in ghosts: + args += ['--ghost', fqn] + super().__init__(args, env) + self.entries = len(ghosts) + len(backed) + + def wait_status(self, timeout): + """(matched_subscriptions, entries, acked) once the messages went out, or None.""" + match = self.wait_line( + r'ghost_node_injector: matched_subscriptions=(\d+) entries=(\d+) acked=(true|false)', + timeout) + if match is None: + return None + return int(match.group(1)), int(match.group(2)), match.group(3) == 'true' + + +class LeftoverNode(_FixtureProcess): + """A real node that, on `leave()`, leaves a leftover of itself in every graph. + + `announce_nodes()` adds `announce` nodes ``_`` to its participant; they are + left over with it. + """ + + def __init__(self, fqn, delay_sec, announce=0, env=None): + args = ['--leftover', fqn, '--delay', f'{delay_sec:.3f}'] + if announce: + args += ['--announce', str(announce)] + super().__init__(args, env) + self.fqn = fqn + self.announce = announce + + def announced_fqns(self): + return [f'{self.fqn}_{index:05d}' for index in range(self.announce)] + + def wait_ready(self, timeout): + """Whether the node runs and its participant's discovery message was captured.""" + match = self.wait_line(r'ghost_node_injector: leftover_ready participant_gid=', timeout) + return match is not None + + def announce_nodes(self, timeout): + """Publish the announced nodes; whether every matched reader acknowledged them.""" + self.send('announce') + match = self.wait_line( + r'ghost_node_injector: leftover_announced count=\d+ acked=(true|false)', timeout) + return match is not None and match.group(1) == 'true' + + def leave(self): + """Remove the node and its participant; the late sample follows after the delay.""" + self.send('leave') + + def publish(self): + """Send the late sample now, after `leave()`, rather than when the delay runs out.""" + self.send('publish') diff --git a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py index f36a329de..198028f69 100644 --- a/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py +++ b/src/ros2_medkit_integration_tests/ros2_medkit_test_utils/launch_helpers.py @@ -101,7 +101,7 @@ # --------------------------------------------------------------------------- def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', - extra_params=None, coverage=True, extra_env=None, + extra_params=None, parameter_files=(), coverage=True, extra_env=None, respawn=False, respawn_delay=1.0): """Create a ``gateway_node`` launch action with standard config. @@ -114,6 +114,10 @@ def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', so their names do not collide (e.g. ``gateway_with_scripts``). extra_params : dict or None Additional ROS parameters merged into the node config. + parameter_files : sequence of str + Parameter files passed after those parameters, for values a Python dict cannot + spell the way a user's file does (``.nan``, ``.inf``). A key set in both takes the + file's value. coverage : bool If True, set GCOV_PREFIX env vars for code coverage collection. extra_env : dict or None @@ -152,7 +156,7 @@ def create_gateway_node(*, port=DEFAULT_PORT, name='ros2_medkit_gateway', executable='gateway_node', name=name, output='screen', - parameters=[params], + parameters=[params, *parameter_files], additional_env=env, respawn=respawn, respawn_delay=respawn_delay, diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py new file mode 100644 index 000000000..8bc5104fa --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes.test.py @@ -0,0 +1,561 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Discovery hides the leftover of a node it saw running, and lists everything else. + +``ghost_node_injector --leftover`` leaves a real node's leftover (empty enclave, no +endpoints) in the graph. Each case first checks that the node ran and that this process's +graph then lists only the leftover. ``--ghost`` and ``--backed`` build the same shape for +names the gateway never saw, which stay listed. + +All cases share one gateway and use names of their own. The last case rechecks the earlier +leftovers after the hold, so it must run last. +""" + +import os +import signal +import subprocess +import time +import unittest + +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + GhostInjection, + LeftoverNode, + observed_enclaves, + split_fqn, + stop_process, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() + +# Backstop refresh, so the observation windows below count refreshes that ran. +REFRESH_INTERVAL_MS = 500 +REFRESH_DEBOUNCE_MS = 500 +# How long a claim is watched: three refreshes. A window, not a give-up bound, so not scaled. +OBSERVE_SEC = 3 * REFRESH_INTERVAL_MS / 1000.0 +# Give-up bounds for the fixture (30 s of its own), DDS discovery and process start or exit. +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +# From this process's graph listing a node to GET /apps listing it: the debounce, the 100 ms +# graph poll, and up to 4 s by which the gateway's discovery can trail on a loaded host. +LISTED_AFTER_OBSERVED_SEC = ((REFRESH_DEBOUNCE_MS + 100) / 1000.0 + 4.0) * TIME_SCALE +# GraphNodeListReader::kDefaultHold. +LEFTOVER_HOLD_SEC = 10.0 +# Delay of the late sample for leftovers that arrive while the gateway remembers the node. +LEFTOVER_DELAY_SEC = 1.0 +# Inside the hold but six refreshes after the removal: catches forgetting after a few refreshes. +INSIDE_HOLD_DELAY_SEC = 3.0 +# From GET /apps first not listing a departed node until a refresh has run past the hold. +PAST_THE_HOLD_SEC = LEFTOVER_HOLD_SEC + LISTED_AFTER_OBSERVED_SEC +# A --delay longer than any run of this file: the late sample goes out on `publish`. +PUBLISH_ON_COMMAND_DELAY_SEC = 3600.0 +# How many leftovers the many-leftovers case leaves behind next to the live nodes. +MANY_LEFTOVERS = 50 + +# Departs in setUpClass; the last case sends its late sample after the hold. +AFTER_HOLD_FQN = '/leftover_hold_ns/leftover_after' + + +def generate_test_description(): + gateway_node = create_gateway_node( + extra_params={ + 'refresh_interval_ms': REFRESH_INTERVAL_MS, + 'discovery.refresh_debounce_ms': REFRESH_DEBOUNCE_MS, + }, + ) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +def _app_id(fqn): + return split_fqn(fqn)[0] + + +def _app_ids(): + response = requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=10) + response.raise_for_status() + return [item.get('id') for item in response.json().get('items', [])] + + +def _poll_app_listed(app_id, timeout, listed=True): + """Time at which GET /apps first matched `listed` for `app_id`, or None.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if (app_id in _app_ids()) == listed: + return time.monotonic() + except requests.exceptions.RequestException: + pass + time.sleep(0.1) + return None + + +class TestGraphLeftoverNodes(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_observer') + # (fqn, App id, when GET /apps first stopped listing it) per hidden leftover. + cls.hidden_leftovers = [] + + # Departs now with no entry left; its late sample waits for the last case. + cls.after_hold = LeftoverNode( + AFTER_HOLD_FQN, PUBLISH_ON_COMMAND_DELAY_SEC, env=_fixture_env()) + if not cls.after_hold.wait_ready(INJECTION_TIMEOUT_SEC): + raise AssertionError( + f'ghost_node_injector never got {AFTER_HOLD_FQN} running:\n' + f'{cls.after_hold.output()}') + if _poll_app_listed(_app_id(AFTER_HOLD_FQN), APPEAR_TIMEOUT_SEC) is None: + raise AssertionError(f'{AFTER_HOLD_FQN} runs but never appeared in GET /apps') + cls.after_hold.leave() + cls.after_hold_departed = _poll_app_listed( + _app_id(AFTER_HOLD_FQN), LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC, + listed=False) + if cls.after_hold_departed is None: + raise AssertionError(f'{AFTER_HOLD_FQN} left, but GET /apps kept listing it') + + @classmethod + def tearDownClass(cls): + cls.after_hold.stop(PROCESS_EXIT_TIMEOUT_SEC) + cls.observer.destroy_node() + rclpy.shutdown() + + def setUp(self): + self._processes = [] + self._fixtures = [] + + def tearDown(self): + for fixture in self._fixtures: + fixture.stop(PROCESS_EXIT_TIMEOUT_SEC) + for proc in self._processes: + stop_process(proc, PROCESS_EXIT_TIMEOUT_SEC) + + # ---- fixtures ----------------------------------------------------------------------- + + def _start_node(self, fqn, executable='demo_rpm_sensor'): + name, namespace = split_fqn(fqn) + proc = subprocess.Popen( + [fixture_path(executable), '--ros-args', '-r', f'__ns:={namespace}', + '-r', f'__node:={name}'], + env=_fixture_env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + self._processes.append(proc) + return proc + + def _start_leftover(self, fqn, delay_sec=LEFTOVER_DELAY_SEC, announce=0): + """Start a node that can leave a leftover, and wait until GET /apps lists it.""" + leftover = LeftoverNode(fqn, delay_sec, announce=announce, env=_fixture_env()) + self._fixtures.append(leftover) + self.assertTrue( + leftover.wait_ready(INJECTION_TIMEOUT_SEC), + f'ghost_node_injector never got {fqn} running:\n{leftover.output()}') + self.assertTrue( + self._wait_listed(_app_id(fqn), APPEAR_TIMEOUT_SEC), + f'{fqn} runs but never appeared in GET /apps, so the gateway never saw it running') + return leftover + + def _wait_left_over(self, fqn, timeout): + """Assert this process's graph lists `fqn` only as a leftover (empty enclave).""" + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], timeout), + f'after its participant left, the test process graph lists {fqn} with enclaves ' + f'{observed_enclaves(self.observer, fqn)} rather than once with an empty one, so ' + 'no leftover was reproduced') + + def _hidden_since(self, fqn, app_id): + """Wait until GET /apps does not list `app_id`; record the leftover for the last case.""" + hidden_at = _poll_app_listed(app_id, LISTED_AFTER_OBSERVED_SEC, listed=False) + self.assertIsNotNone(hidden_at, f'the leftover of {fqn} is listed as {app_id}') + type(self).hidden_leftovers.append((fqn, app_id, hidden_at)) + + # ---- observations ------------------------------------------------------------------- + + @staticmethod + def _ids(collection): + response = requests.get(f'{DEFAULT_BASE_URL}/{collection}', timeout=10) + response.raise_for_status() + return [item.get('id') for item in response.json().get('items', [])] + + @staticmethod + def _wait_listed(app_id, timeout, listed=True): + return _poll_app_listed(app_id, timeout, listed) is not None + + @staticmethod + def _warnings_for(proc_output, gateway_node, fqn): + text = ''.join( + output.text.decode(errors='replace') for output in proc_output[gateway_node]) + return text.count(f"Node '{fqn}' is not exposed") + + def _wait_warnings(self, proc_output, gateway_node, fqn, expected, + timeout=LISTED_AFTER_OBSERVED_SEC): + deadline = time.monotonic() + timeout + count = self._warnings_for(proc_output, gateway_node, fqn) + while count < expected and time.monotonic() < deadline: + time.sleep(0.2) + count = self._warnings_for(proc_output, gateway_node, fqn) + return count + + @staticmethod + def _observe(check): + """Run `check` every 250 ms for OBSERVE_SEC; it fails the test on the first bad sample.""" + deadline = time.monotonic() + OBSERVE_SEC + while time.monotonic() < deadline: + check() + time.sleep(0.25) + + # ---- cases -------------------------------------------------------------------------- + + def test_01_leftover_of_a_listed_node_inside_the_hold_is_hidden( + self, proc_output, gateway_node): + fqn = '/leftover_solo_ns/leftover_solo' + leftover = self._start_leftover(fqn, delay_sec=INSIDE_HOLD_DELAY_SEC) + self.assertIn( + 'leftover_solo_ns', self._ids('functions'), + f'no Function was derived from the namespace of the running {fqn}, so its absence ' + 'below would show nothing') + + leftover.leave() + self.assertTrue( + self._wait_listed('leftover_solo', LISTED_AFTER_OBSERVED_SEC, listed=False), + f'{fqn} left, but GET /apps kept listing it, so the gateway never read the graph ' + 'without it') + self._wait_left_over(fqn, INSIDE_HOLD_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self._hidden_since(fqn, 'leftover_solo') + + def check(): + app_ids = self._ids('apps') + self.assertNotIn( + 'leftover_solo', app_ids, + f'the late sample for {fqn} arrived {INSIDE_HOLD_DELAY_SEC:.1f} s after its ' + f'participant left, inside the {LEFTOVER_HOLD_SEC:.0f} s hold, and is listed') + detail = requests.get(f'{DEFAULT_BASE_URL}/apps/leftover_solo', timeout=10) + self.assertEqual( + detail.status_code, 404, + f'GET /apps/leftover_solo answered {detail.status_code} for a leftover') + self.assertNotIn( + 'leftover_solo_ns', self._ids('functions'), + f'a Function is still derived from the namespace only the leftover of {fqn} ' + 'occupies') + + self._observe(check) + self.assertEqual( + observed_enclaves(self.observer, fqn), [''], + f'the leftover of {fqn} left the test process graph during the observation, so the ' + 'gateway was not holding it out') + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway must warn exactly once about the leftover of {fqn}') + + def test_02_nodes_the_gateway_never_saw_leave_are_listed(self, proc_output, gateway_node): + far = '/ghost_far_ns/ghost_far' + twin = '/ghost_twin_ns/ghost_twin' + quiet = '/quiet_ns/quiet' + # A never-seen node without an enclave; the same next to a backed twin whose GID sorts + # after the ghost's; and a running node with no endpoints. + injection = GhostInjection(ghosts=[far, twin], backed=[twin], env=_fixture_env()) + self._fixtures.append(injection) + self._start_node(quiet, executable='endpointless_node') + self.assertIsNotNone( + injection.wait_status(INJECTION_TIMEOUT_SEC), + f'ghost_node_injector never published:\n{injection.output()}') + for fqn, enclaves in ((far, ['']), (twin, ['', '']), (quiet, ['/'])): + self.assertTrue( + wait_observed(self.observer, fqn, lambda found, want=enclaves: found == want, + INJECTION_TIMEOUT_SEC), + f'the test process graph lists {fqn} with {observed_enclaves(self.observer, fqn)}' + f', not {enclaves}') + name, namespace = split_fqn(quiet) + endpoints = ( + self.observer.get_publisher_names_and_types_by_node(name, namespace, True) + + self.observer.get_subscriber_names_and_types_by_node(name, namespace, True)) + self.assertEqual( + endpoints, [], + f'the fixture {quiet} is supposed to have no endpoints of its own, so this case ' + f'would not show that endpoints are not the rule: {endpoints}') + ids = {'ghost_far', 'ghost_twin', 'quiet'} + deadline = time.monotonic() + LISTED_AFTER_OBSERVED_SEC + while ids - set(self._ids('apps')) and time.monotonic() < deadline: + time.sleep(0.1) + self.assertEqual( + ids - set(self._ids('apps')), set(), + 'these nodes are not leftovers of nodes the gateway saw running, and never appeared ' + 'in GET /apps') + + def check(): + self.assertEqual(ids - set(self._ids('apps')), set(), + 'a node dropped out of GET /apps') + + self._observe(check) + for fqn in (far, twin, quiet): + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 0) + + def test_03_leftover_then_the_node_runs_again_is_listed_once( + self, proc_output, gateway_node): + fqn = '/leftover_again_ns/leftover_again' + leftover = self._start_leftover(fqn) + leftover.leave() + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertTrue( + self._wait_listed('leftover_again', LISTED_AFTER_OBSERVED_SEC, listed=False), + f'the leftover of {fqn} is listed') + self.assertEqual(self._wait_warnings(proc_output, gateway_node, fqn, 1), 1) + + self._start_node(fqn) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: sorted(enclaves) == ['', '/'], + APPEAR_TIMEOUT_SEC), + f'the test process graph does not list {fqn} running next to its leftover: ' + f'{observed_enclaves(self.observer, fqn)}') + self.assertTrue( + self._wait_listed('leftover_again', LISTED_AFTER_OBSERVED_SEC), + f'{fqn} runs again but never appeared in GET /apps') + + def check(): + app_ids = self._ids('apps') + self.assertEqual( + app_ids.count('leftover_again'), 1, + f'{fqn} runs next to its leftover and must be listed exactly once: {app_ids}') + + self._observe(check) + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 1) + + def test_04_leftover_does_not_rename_a_live_node_sharing_its_bare_name( + self, proc_output, gateway_node): + live = '/leftover_collide_b/leftover_collider' + fqn = '/leftover_collide_a/leftover_collider' + self._start_node(live) + self.assertTrue(self._wait_listed('leftover_collider', APPEAR_TIMEOUT_SEC), + f'the live node {live} never appeared in GET /apps') + leftover = LeftoverNode(fqn, LEFTOVER_DELAY_SEC, env=_fixture_env()) + self._fixtures.append(leftover) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertTrue( + self._wait_listed('leftover_collide_a_leftover_collider', APPEAR_TIMEOUT_SEC), + 'while both nodes run, the gateway never gave them namespace-prefixed ids, so the ' + 'collision rule this case depends on did not engage') + + leftover.leave() + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertTrue( + self._wait_listed('leftover_collider', LISTED_AFTER_OBSERVED_SEC), + f'{live} never got its un-prefixed id back after the other node left') + # Only the gateway's own read logs this, so its graph holds the leftover too. + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway never warned about the leftover of {fqn}, so its graph may never have ' + 'listed it and the ids below would show nothing') + hidden_at = time.monotonic() + + def check(): + app_ids = self._ids('apps') + self.assertIn( + 'leftover_collider', app_ids, + f'{live} lost its un-prefixed App id to the leftover of {fqn}: {app_ids}') + self.assertFalse( + [app_id for app_id in app_ids if app_id.endswith('_leftover_collider')], + f'a namespace-prefixed id was derived for a collision with a leftover: {app_ids}') + + self._observe(check) + # Once the live node is gone, the leftover would take the un-prefixed id. + type(self).hidden_leftovers.append((fqn, 'leftover_collider', hidden_at)) + + def test_05_node_seen_running_that_stays_behind_an_endpoint_is_listed( + self, proc_output, gateway_node): + fqn = '/bridged_ns/bridged' + marker = '/bridged_ns/bridged_marker' + node = self._start_node(fqn) + self.assertTrue(self._wait_listed('bridged', APPEAR_TIMEOUT_SEC), + f'{fqn} never appeared in GET /apps') + # A backed entry of the same name, published while the node runs; it is all that is + # left once the node exits. The marker ghost goes out after it. + injection = GhostInjection(ghosts=[marker], backed=[fqn], env=_fixture_env()) + self._fixtures.append(injection) + self.assertIsNotNone(injection.wait_status(INJECTION_TIMEOUT_SEC), injection.output()) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: sorted(enclaves) == ['', '/'], + INJECTION_TIMEOUT_SEC), + f'the test process graph does not list {fqn} both running and without an enclave: ' + f'{observed_enclaves(self.observer, fqn)}') + # The gateway's discovery of the injector can trail this process's by seconds. + self.assertTrue( + self._wait_listed(_app_id(marker), APPEAR_TIMEOUT_SEC), + f'GET /apps never listed {marker}, so the gateway graph may not hold the backed ' + f'entry of {fqn} when the node exits') + + node.send_signal(signal.SIGTERM) + node.wait(timeout=PROCESS_EXIT_TIMEOUT_SEC) + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], + APPEAR_TIMEOUT_SEC), + f'after the node exited, the test process graph lists {fqn} with ' + f'{observed_enclaves(self.observer, fqn)}') + name, namespace = split_fqn(fqn) + self.assertTrue( + self.observer.get_publisher_names_and_types_by_node(name, namespace, True), + f'{fqn} resolves no endpoint in the test process graph, so this case would not show ' + 'that an endpoint keeps it listed') + + def check(): + self.assertIn( + 'bridged', self._ids('apps'), + f'{fqn} was seen running and has an endpoint, but dropped out of GET /apps') + + self._observe(check) + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 0) + + def test_06_many_leftovers_next_to_live_nodes(self, proc_output, gateway_node): + live = [f'/leftover_many_live/live_{index}' for index in range(3)] + for fqn in live: + self._start_node(fqn) + for fqn in live: + self.assertTrue(self._wait_listed(_app_id(fqn), APPEAR_TIMEOUT_SEC), + f'the live node {fqn} never appeared in GET /apps') + + leftover = self._start_leftover( + '/leftover_many_ns/leftover_many', announce=MANY_LEFTOVERS - 1) + # Announce after GET /apps lists the node, or its own discovery message can replace it. + self.assertTrue(leftover.announce_nodes(INJECTION_TIMEOUT_SEC), leftover.output()) + fqns = [leftover.fqn] + leftover.announced_fqns() + ids = {_app_id(fqn) for fqn in fqns} + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while ids - set(self._ids('apps')) and time.monotonic() < deadline: + time.sleep(0.2) + self.assertEqual( + ids - set(self._ids('apps')), set(), + 'the gateway never listed every announced node running, so it did not see them run') + + leftover.leave() + for fqn in fqns: + self._wait_left_over(fqn, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + live_ids = {_app_id(fqn) for fqn in live} + + def check(): + app_ids = set(self._ids('apps')) + self.assertEqual(live_ids - app_ids, set(), 'a live node dropped out of GET /apps') + self.assertEqual(app_ids & ids, set(), 'leftovers are listed') + self.assertNotIn('leftover_many_ns', self._ids('functions')) + + self._observe(check) + + late = '/leftover_many_live/late' + self._start_node(late) + self.assertTrue( + wait_observed(self.observer, late, lambda enclaves: '/' in enclaves, + APPEAR_TIMEOUT_SEC), + f'the test process graph never listed {late}') + observed = time.monotonic() + self.assertTrue( + self._wait_listed('late', APPEAR_TIMEOUT_SEC), + f'{late} never appeared in GET /apps next to {MANY_LEFTOVERS} leftovers') + latency = time.monotonic() - observed + self.assertLessEqual( + latency, LISTED_AFTER_OBSERVED_SEC, + f'{late} took {latency:.2f} s to reach GET /apps after the graph listed it, next ' + f'to {MANY_LEFTOVERS} leftovers') + + for fqn in fqns: + self.assertEqual( + self._wait_warnings(proc_output, gateway_node, fqn, 1), 1, + f'the gateway must warn exactly once about the leftover of {fqn}') + + def test_07_the_hold_runs_out_only_for_a_name_no_entry_lists( + self, proc_output, gateway_node): + # Leftovers hidden by earlier cases, listed for longer than the hold, stay hidden: a + # listed leftover keeps its name remembered. + self.assertTrue(self.hidden_leftovers, 'no earlier case recorded a hidden leftover') + oldest = min(hidden_at for _, _, hidden_at in self.hidden_leftovers) + remaining = PAST_THE_HOLD_SEC - (time.monotonic() - oldest) + if remaining > 0: + time.sleep(remaining) + now = time.monotonic() + past_the_hold = [ + (fqn, app_id) for fqn, app_id, hidden_at in self.hidden_leftovers + if now - hidden_at >= PAST_THE_HOLD_SEC] + app_ids = set(self._ids('apps')) + for fqn, app_id in past_the_hold: + self.assertEqual( + observed_enclaves(self.observer, fqn), [''], + f'the test process graph no longer lists {fqn} as a leftover only') + self.assertNotIn( + app_id, app_ids, + f'the leftover of {fqn} was hidden and is listed again as {app_id} while the ' + 'graph still lists it: the gateway forgot the node after the hold') + self.assertEqual(self._warnings_for(proc_output, gateway_node, fqn), 1) + + # With no entry left, the first refresh past the hold forgets the name (refreshes run + # every 0.5 s here), so a later late sample is listed like a node never seen running. + remaining = PAST_THE_HOLD_SEC - (time.monotonic() - self.after_hold_departed) + if remaining > 0: + time.sleep(remaining) + self.after_hold.publish() + self._wait_left_over(AFTER_HOLD_FQN, INJECTION_TIMEOUT_SEC) + after_id = _app_id(AFTER_HOLD_FQN) + self.assertTrue( + self._wait_listed(after_id, LISTED_AFTER_OBSERVED_SEC), + f'the late sample for {AFTER_HOLD_FQN} arrived more than {PAST_THE_HOLD_SEC:.1f} s ' + f'after it departed, past the {LEFTOVER_HOLD_SEC:.0f} s hold, so the gateway no ' + 'longer remembers the node; it never appeared in GET /apps') + + def after_listed(): + self.assertIn(after_id, self._ids('apps'), + f'the leftover of {AFTER_HOLD_FQN}, past the hold, dropped out of ' + 'GET /apps') + + self._observe(after_listed) + self.assertEqual(self._warnings_for(proc_output, gateway_node, AFTER_HOLD_FQN), 0) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py new file mode 100644 index 000000000..1fb1ea4c1 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_scale.test.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Discovery remembers a bounded number of departed nodes, and running nodes cost nothing. + +One participant announces more nodes than discovery remembers. Checks how many leftovers stay +hidden, that a node departing with no entry left is forgotten first past the capacity, and +that new nodes are still listed. Own gateway: a graph this size slows later reads. +""" + +import os +import subprocess +import time +import unittest + +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + LeftoverNode, + observed_enclaves, + split_fqn, + stop_process, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() + +REFRESH_INTERVAL_MS = 500 +REFRESH_DEBOUNCE_MS = 500 +# How long a claim is watched: three refreshes. An observation window, so not scaled. +OBSERVE_SEC = 3 * REFRESH_INTERVAL_MS / 1000.0 +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +# Listing bound of test_graph_leftover_nodes.test.py doubled: each remembered leftover costs +# endpoint queries on every read. +LISTED_AFTER_OBSERVED_SEC = 2 * ((REFRESH_DEBOUNCE_MS + 100) / 1000.0 + 4.0) * TIME_SCALE +LEFTOVER_DELAY_SEC = 1.0 +# A --delay longer than any run of this file: the late sample goes out on `publish`. +PUBLISH_ON_COMMAND_DELAY_SEC = 3600.0 +# GraphNodeListReader::kDefaultCapacity. +CAPACITY = 1024 +# Announced nodes: with the announcing node, more than discovery remembers. +ANNOUNCED = CAPACITY + 76 + +CAP_FQN = '/leftover_cap_ns/leftover_cap' +DEPART_FQN = '/leftover_depart_ns/leftover_depart' +RECENT_FQN = '/leftover_recent_ns/leftover_recent' +LATE_FQN = '/leftover_cap_live/late' + + +def generate_test_description(): + gateway_node = create_gateway_node( + extra_params={ + 'refresh_interval_ms': REFRESH_INTERVAL_MS, + 'discovery.refresh_debounce_ms': REFRESH_DEBOUNCE_MS, + }, + ) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +class TestGraphLeftoverNodesScale(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_scale_observer') + + @classmethod + def tearDownClass(cls): + cls.observer.destroy_node() + rclpy.shutdown() + + def setUp(self): + self._stop = [] + + def tearDown(self): + for stop in reversed(self._stop): + stop() + + @staticmethod + def _app_ids(): + response = requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=30) + response.raise_for_status() + return {item.get('id') for item in response.json().get('items', [])} + + def _wait_ids(self, predicate, timeout, interval=0.5): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + ids = self._app_ids() + if predicate(ids): + return ids + except requests.exceptions.RequestException: + pass + time.sleep(interval) + return None + + def _observe(self, check, interval=0.5): + deadline = time.monotonic() + OBSERVE_SEC + while time.monotonic() < deadline: + check(self._app_ids()) + time.sleep(interval) + + def _leftover(self, fqn, delay_sec, announce=0): + leftover = LeftoverNode(fqn, delay_sec, announce=announce, env=_fixture_env()) + self._stop.append(lambda: leftover.stop(PROCESS_EXIT_TIMEOUT_SEC)) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertIsNotNone( + self._wait_ids(lambda ids: split_fqn(fqn)[0] in ids, APPEAR_TIMEOUT_SEC), + f'{fqn} runs but never appeared in GET /apps') + return leftover + + def _wait_left_over(self, fqn, timeout): + self.assertTrue( + wait_observed(self.observer, fqn, lambda enclaves: enclaves == [''], timeout), + f'the test process graph lists {fqn} with {observed_enclaves(self.observer, fqn)} ' + 'rather than as a leftover') + + def _warned(self, proc_output, gateway_node, fqns): + text = ''.join( + output.text.decode(errors='replace') for output in proc_output[gateway_node]) + return sum(1 for fqn in fqns if f"Node '{fqn}' is not exposed" in text) + + def _wait_warned(self, proc_output, gateway_node, fqns, expected): + deadline = time.monotonic() + LISTED_AFTER_OBSERVED_SEC + while (self._warned(proc_output, gateway_node, fqns) < expected + and time.monotonic() < deadline): + time.sleep(0.2) + return self._warned(proc_output, gateway_node, fqns) + + def test_more_nodes_than_discovery_remembers(self, proc_output, gateway_node): + cap = self._leftover(CAP_FQN, PUBLISH_ON_COMMAND_DELAY_SEC, announce=ANNOUNCED) + self.assertTrue(cap.announce_nodes(INJECTION_TIMEOUT_SEC), cap.output()) + cap_fqns = [CAP_FQN] + cap.announced_fqns() + cap_ids = {split_fqn(fqn)[0] for fqn in cap_fqns} + self.assertIsNotNone( + self._wait_ids(lambda ids: cap_ids <= ids, 2 * APPEAR_TIMEOUT_SEC), + f'the gateway never listed all {len(cap_ids)} announced nodes running') + + # Running nodes do not count against the capacity: this node's leftover is hidden. + depart_id = split_fqn(DEPART_FQN)[0] + depart = self._leftover(DEPART_FQN, LEFTOVER_DELAY_SEC) + depart.leave() + self._wait_left_over(DEPART_FQN, LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC) + self.assertEqual( + self._wait_warned(proc_output, gateway_node, [DEPART_FQN], 1), 1, + f'the gateway never warned about the leftover of {DEPART_FQN} next to ' + f'{len(cap_ids)} running nodes, so it forgot the node') + self.assertIsNotNone( + self._wait_ids(lambda ids: depart_id not in ids and cap_ids <= ids, + LISTED_AFTER_OBSERVED_SEC), + f'the leftover of {DEPART_FQN} is listed next to {len(cap_ids)} running nodes') + + # Every announced node departs with no entry of it left, then leaves a leftover. + cap.leave() + self.assertIsNotNone( + self._wait_ids(lambda ids: not (ids & cap_ids), + LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC), + 'the announced nodes left, but GET /apps kept listing some of them') + cap.publish() + namespace = split_fqn(CAP_FQN)[1] + + def left_over(entries): + enclaves = [enclave for _, node_namespace, enclave in entries + if node_namespace == namespace] + return len(enclaves) == len(cap_fqns) and set(enclaves) == {''} + + deadline = time.monotonic() + INJECTION_TIMEOUT_SEC + while (not left_over(self.observer.get_node_names_and_namespaces_with_enclaves()) + and time.monotonic() < deadline): + time.sleep(0.2) + self.assertTrue( + left_over(self.observer.get_node_names_and_namespaces_with_enclaves()), + f'the test process graph does not list all {len(cap_fqns)} nodes as leftovers') + + # As many leftovers as the capacity allows stay hidden; the rest are listed again. + listed_count = len(cap_fqns) - (CAPACITY - 1) + listed = self._wait_ids( + lambda ids: len(ids & cap_ids) == listed_count, LISTED_AFTER_OBSERVED_SEC) + self.assertIsNotNone( + listed, + f'of {len(cap_ids)} leftovers GET /apps lists {len(self._app_ids() & cap_ids)}; with ' + f'a capacity of {CAPACITY} and one more departed node remembered it must list ' + f'{listed_count}') + + def cap_check(ids): + self.assertEqual(len(ids & cap_ids), listed_count) + self.assertNotIn(depart_id, ids, f'the leftover of {DEPART_FQN} is listed') + + self._observe(cap_check) + self.assertEqual( + self._wait_warned(proc_output, gateway_node, cap_fqns, CAPACITY - 1), CAPACITY - 1) + + # Past the capacity a node with no entry left is forgotten first: its late sample is + # listed, and no older leftover takes its place. + recent_id = split_fqn(RECENT_FQN)[0] + recent = self._leftover(RECENT_FQN, PUBLISH_ON_COMMAND_DELAY_SEC) + recent.leave() + self.assertIsNotNone( + self._wait_ids(lambda ids: recent_id not in ids, + LISTED_AFTER_OBSERVED_SEC + PROCESS_EXIT_TIMEOUT_SEC), + f'{RECENT_FQN} left, but GET /apps kept listing it') + recent.publish() + self._wait_left_over(RECENT_FQN, INJECTION_TIMEOUT_SEC) + self.assertIsNotNone( + self._wait_ids(lambda ids: recent_id in ids, LISTED_AFTER_OBSERVED_SEC), + f'the leftover of {RECENT_FQN} is hidden: past the capacity discovery forgot an ' + 'older, listed leftover rather than the name no entry listed') + + def recent_check(ids): + self.assertIn(recent_id, ids, f'the leftover of {RECENT_FQN} dropped out of GET /apps') + self.assertEqual( + len(ids & cap_ids), listed_count, + 'an older leftover was listed in place of the name no entry listed') + self.assertNotIn(depart_id, ids, f'the leftover of {DEPART_FQN} is listed') + + self._observe(recent_check) + self.assertEqual(self._warned(proc_output, gateway_node, [RECENT_FQN]), 0) + + name, namespace = split_fqn(LATE_FQN) + late = subprocess.Popen( + [fixture_path('demo_rpm_sensor'), '--ros-args', '-r', f'__ns:={namespace}', + '-r', f'__node:={name}'], + env=_fixture_env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + self._stop.append(lambda: stop_process(late, PROCESS_EXIT_TIMEOUT_SEC)) + self.assertTrue( + wait_observed(self.observer, LATE_FQN, lambda enclaves: '/' in enclaves, + APPEAR_TIMEOUT_SEC), + f'the test process graph never listed {LATE_FQN}') + observed = time.monotonic() + self.assertIsNotNone( + self._wait_ids(lambda ids: name in ids, APPEAR_TIMEOUT_SEC, interval=0.1), + f'{LATE_FQN} never appeared in GET /apps next to {len(cap_ids)} leftovers') + latency = time.monotonic() - observed + self.assertLessEqual( + latency, LISTED_AFTER_OBSERVED_SEC, + f'{LATE_FQN} took {latency:.2f} s to reach GET /apps after the graph listed it, ' + f'next to {len(cap_ids)} leftovers') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py new file mode 100644 index 000000000..c8bcadff6 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_graph_leftover_nodes_startup.test.py @@ -0,0 +1,283 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The graph readers outside discovery apply the same leftover rule. + +Covers the startup summary's peer count (discovery's reader) and parameter_beacon (a reader +of its own). Before the gateway starts, the graph lists a ``--ghost`` node, a ``--backed`` +node, a running node, and a running twin whose name is also listed without an enclave. The +``--backed`` node in GET /apps shows the gateway got the injector's transient-local history. +""" + +import os +import re +import time +import unittest + +from ament_index_python.packages import get_package_prefix +from launch import LaunchDescription +from launch.actions import ExecuteProcess, TimerAction +import launch_ros.actions +import launch_testing +import launch_testing.actions +import rclpy +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + DEFAULT_BASE_URL, + DEFAULT_DOMAIN_ID, + get_time_scale, +) +from ros2_medkit_test_utils.graph_fixtures import ( + fixture_path, + LeftoverNode, + observed_enclaves, + split_fqn, + wait_observed, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node, get_coverage_env + +TIME_SCALE = get_time_scale() +PACKAGE = 'ros2_medkit_integration_tests' + +GHOST = '/ghost_boot_ns/ghost_boot' +BACKED_CONTROL = '/ghost_boot_control/ghost_boot_control' +LIVE = '/ghost_boot_live/ghost_boot_live' +TWIN = '/ghost_boot_twin/ghost_boot_twin' +LEFTOVER = '/leftover_beacon_ns/leftover_beacon' +# Ghost, backed, live and twin (once): the injector's node is hidden, the gateway's are not peers. +EXPECTED_PEERS = 4 + +# The gateway starts after the injected entries are in every graph. +GATEWAY_START_DELAY_SEC = 3.0 +STARTUP_TIMEOUT_SEC = 60.0 * TIME_SCALE +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +INJECTION_TIMEOUT_SEC = 45.0 * TIME_SCALE +PROCESS_EXIT_TIMEOUT_SEC = 30.0 * TIME_SCALE +LEFTOVER_DELAY_SEC = 1.0 +POLL_INTERVAL_SEC = 1.0 +# Three 1 s poll cycles of watching parameter_beacon's clients. A window, so not scaled. +POLL_HOLD_SEC = 3.0 + + +def _node_action(fqn): + namespace, _, name = fqn.rpartition('/') + return launch_ros.actions.Node( + package=PACKAGE, + executable='demo_rpm_sensor', + name=name, + namespace=namespace, + output='screen', + additional_env=get_coverage_env(PACKAGE), + sigterm_timeout='30', + sigkill_timeout='15', + ) + + +def generate_test_description(): + injector = ExecuteProcess( + cmd=[fixture_path('ghost_node_injector'), '--ghost', GHOST, '--backed', BACKED_CONTROL, + '--ghost', TWIN], + output='screen', + sigterm_timeout='30', + sigkill_timeout='15', + ) + plugin_path = os.path.join( + get_package_prefix('ros2_medkit_param_beacon'), 'lib', 'ros2_medkit_param_beacon', + 'libparam_beacon_plugin.so') + gateway_node = create_gateway_node( + extra_params={ + 'plugins': ['parameter_beacon'], + 'plugins.parameter_beacon.path': plugin_path, + 'plugins.parameter_beacon.poll_interval_sec': POLL_INTERVAL_SEC, + 'plugins.parameter_beacon.param_timeout_sec': 1.0, + }, + ) + return ( + LaunchDescription([ + injector, + _node_action(LIVE), + _node_action(TWIN), + TimerAction(period=GATEWAY_START_DELAY_SEC, actions=[gateway_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'gateway_node': gateway_node, 'injector': injector}, + ) + + +def _output(proc_output, process): + return ''.join(output.text.decode(errors='replace') for output in proc_output[process]) + + +def _fixture_env(): + env = os.environ.copy() + env['ROS_DOMAIN_ID'] = str(DEFAULT_DOMAIN_ID) + env.update(get_coverage_env()) + return env + + +class TestGraphLeftoverNodesAtStartup(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + STARTUP_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.observer = rclpy.create_node('_graph_leftover_startup_observer') + + @classmethod + def tearDownClass(cls): + cls.observer.destroy_node() + rclpy.shutdown() + + @staticmethod + def _app_ids(): + return [item.get('id') for item in + requests.get(f'{DEFAULT_BASE_URL}/apps', timeout=10).json().get('items', [])] + + def _assert_injected(self, proc_output, injector): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + match = None + while match is None and time.monotonic() < deadline: + match = re.search(r'ghost_node_injector: matched_subscriptions=(\d+)', + _output(proc_output, injector)) + time.sleep(0.2) + self.assertIsNotNone(match, 'ghost_node_injector printed no status line') + self.assertGreater( + int(match.group(1)), 0, + 'ghost_node_injector timed out waiting for a matched subscription on ' + 'ros_discovery_info and never published') + for fqn, enclaves in ((GHOST, ['']), (TWIN, ['', '/'])): + self.assertTrue( + wait_observed(self.observer, fqn, + lambda found, want=enclaves: sorted(found) == want, + APPEAR_TIMEOUT_SEC), + f'the test process graph lists {fqn} with ' + f'{observed_enclaves(self.observer, fqn)}, not {enclaves}') + control_id = split_fqn(BACKED_CONTROL)[0] + while control_id not in self._app_ids() and time.monotonic() < deadline: + time.sleep(0.2) + self.assertIn(control_id, self._app_ids(), + f'{BACKED_CONTROL} never appeared in GET /apps, so the gateway never ' + 'received the injection') + + def _beacon_clients(self): + try: + return [name for name, _ in self.observer.get_client_names_and_types_by_node( + '_param_beacon_node', '/')] + except Exception: + # rclpy raises for a node this graph does not list yet. + return [] + + def _wait_polled(self, fqn, timeout, polled=True): + prefix = f'{fqn}/' + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if any(name.startswith(prefix) for name in self._beacon_clients()) == polled: + return True + time.sleep(0.2) + return False + + def test_01_startup_peer_count_counts_what_discovery_exposes( + self, proc_output, gateway_node, injector): + self._assert_injected(proc_output, injector) + deadline = time.monotonic() + STARTUP_TIMEOUT_SEC + summary = None + while summary is None and time.monotonic() < deadline: + summary = re.search(r'Discovery summary: (\d+) peer node\(s\)', + _output(proc_output, gateway_node)) + time.sleep(0.2) + self.assertIsNotNone(summary, 'the gateway never logged its startup discovery summary') + self.assertEqual( + int(summary.group(1)), EXPECTED_PEERS, + f'the startup summary must count {LIVE}, {BACKED_CONTROL}, {GHOST} and {TWIN} once') + + def test_02_parameter_beacon_polls_a_node_it_never_saw_running(self, proc_output, injector): + self._assert_injected(proc_output, injector) + self.assertTrue( + self._wait_polled(GHOST, APPEAR_TIMEOUT_SEC), + f'parameter_beacon never created a parameter client for {GHOST}: it has no enclave ' + 'and no endpoints, but the plugin never saw it running, so it is not a leftover') + + def _participant_prefix(self, node_name): + """GUID prefix of the participant `node_name` publishes /rosout from, or None.""" + for info in self.observer.get_publishers_info_by_topic('/rosout'): + if info.node_name == node_name and info.node_namespace == '/': + return bytes(info.endpoint_gid[:12]) + return None + + def test_03_parameter_beacon_does_not_poll_the_leftover_of_a_node_it_polled( + self, proc_output, gateway_node): + leftover = LeftoverNode(LEFTOVER, LEFTOVER_DELAY_SEC, env=_fixture_env()) + self.addCleanup(leftover.stop, PROCESS_EXIT_TIMEOUT_SEC) + self.assertTrue(leftover.wait_ready(INJECTION_TIMEOUT_SEC), leftover.output()) + self.assertTrue( + self._wait_polled(LEFTOVER, APPEAR_TIMEOUT_SEC), + f'parameter_beacon never polled {LEFTOVER} while it ran, so it never saw it running') + + leftover.leave() + self.assertTrue( + wait_observed(self.observer, LEFTOVER, lambda enclaves: enclaves == [''], + LEFTOVER_DELAY_SEC + INJECTION_TIMEOUT_SEC), + f'the test process graph lists {LEFTOVER} with ' + f'{observed_enclaves(self.observer, LEFTOVER)} rather than as a leftover') + # The plugin's node shares the gateway's participant, so discovery's warning shows the + # plugin's graph lists the leftover too. + gateway_prefix = self._participant_prefix('ros2_medkit_gateway') + self.assertIsNotNone(gateway_prefix, 'the gateway node publishes no /rosout') + self.assertEqual( + self._participant_prefix('_param_beacon_node'), gateway_prefix, + "parameter_beacon's node does not share the gateway node's participant, so the " + 'discovery warning would say nothing about the graph the plugin reads') + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + warning = f"Node '{LEFTOVER}' is not exposed" + while warning not in _output(proc_output, gateway_node) and time.monotonic() < deadline: + time.sleep(0.2) + self.assertTrue( + warning in _output(proc_output, gateway_node), + f'discovery never warned about the leftover of {LEFTOVER}, so the graph the plugin ' + 'reads may never have listed it') + + self.assertTrue( + self._wait_polled(LEFTOVER, APPEAR_TIMEOUT_SEC, polled=False), + f'parameter_beacon still has a parameter client for the leftover of {LEFTOVER}') + deadline = time.monotonic() + POLL_HOLD_SEC + while time.monotonic() < deadline: + clients = [name for name in self._beacon_clients() + if name.startswith(f'{LEFTOVER}/')] + self.assertFalse( + clients, + f'parameter_beacon polls the leftover of {LEFTOVER}, a node it polled while it ' + f'ran: {clients}') + time.sleep(0.2) + self.assertEqual(observed_enclaves(self.observer, LEFTOVER), ['']) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + launch_testing.asserts.assertExitCodes( + proc_info, allowable_exit_codes=ALLOWED_EXIT_CODES + ) diff --git a/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py b/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py index aa9755188..9f201f93e 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_openapi_contract.test.py @@ -33,7 +33,7 @@ import launch_testing.actions import requests -from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, DISCOVERY_TIMEOUT from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase from ros2_medkit_test_utils.launch_helpers import ( create_test_launch, @@ -148,6 +148,16 @@ class TestOpenApiContract(GatewayTestCase): MIN_EXPECTED_APPS = 2 REQUIRED_APPS = {'calibration', 'temp_sensor'} + # The app entities appearing is not enough for this file. A node is listed + # in the ROS graph before its service endpoints have propagated, so a + # discovery sweep can build the App with an empty service list, and the + # cache-derived operation items in `/docs` are built from exactly that + # list. Until one service is in the cache every operations sub-document + # publishes only projections, and the comparison over `operations` in + # `test_a_scoped_item_says_what_its_templated_sibling_says` has nothing to + # compare. Waiting for the capability the assertion reads is what makes + # the file independent of how fast the runner propagates a service. + REQUIRED_OPERATIONS = {'/apps/calibration': 'calibrate'} _spec = None @@ -1241,6 +1251,7 @@ def test_every_advertised_collection_is_served(self): """ offenders = [] covered = {} + covered_built = {} for entity_type in ('areas', 'components', 'apps', 'functions'): items = self.get_json(f'/{entity_type}').get('items', []) if not items: @@ -1249,24 +1260,45 @@ def test_every_advertised_collection_is_served(self): # the per-type lists themselves are pinned by the # `EntityCapabilities` unit tests. continue - entity_id = items[0]['id'] - detail = self.get_json(f'/{entity_type}/{entity_id}') - subtree = self.get_json(f'/{entity_type}/{entity_id}/docs') - advertised = {c['href'] for c in detail.get('capabilities', [])} - advertised |= {f'/api/v1{p}' - for p, item in subtree['paths'].items() - if 'get' in item} followed = 0 - for href in sorted(advertised): - if '{' in href: - # A templated path names no concrete resource to fetch. - continue - resp = requests.get( - f'{self.BASE_URL}{href[len("/api/v1"):]}', timeout=10) - followed += 1 - if resp.status_code == 404: - offenders.append(f'{entity_type}: {href}') + built = 0 + for entity_id in [item['id'] for item in items]: + detail = self.get_json(f'/{entity_type}/{entity_id}') + subtree = self.get_json(f'/{entity_type}/{entity_id}/docs') + advertised = {c['href'] for c in detail.get('capabilities', [])} + advertised |= {f'/api/v1{p}' + for p, item in subtree['paths'].items() + if 'get' in item} + # `x-sovd-name` marks a path the cache built out of a concrete + # resource id, which is the only kind whose href can 404 from an + # id that does not resolve. They live in the collection + # sub-documents, so the capability list and the entity subtree + # alone never reach one. + built_hrefs = set() + for collection in ('data', 'operations'): + doc = requests.get( + f'{self.BASE_URL}/{entity_type}/{entity_id}/' + f'{collection}/docs', timeout=10) + if doc.status_code != 200: + continue + built_hrefs |= {f'/api/v1{p}' + for p, item in doc.json().get( + 'paths', {}).items() + if 'get' in item and 'x-sovd-name' in item} + advertised |= built_hrefs + for href in sorted(advertised): + if '{' in href: + # A templated path names no concrete resource to fetch. + continue + resp = requests.get( + f'{self.BASE_URL}{href[len("/api/v1"):]}', timeout=10) + followed += 1 + if href in built_hrefs: + built += 1 + if resp.status_code == 404: + offenders.append(f'{entity_type}: {href}') covered[entity_type] = followed + covered_built[entity_type] = built self.assertEqual(offenders, [], f'advertised but 404: {offenders}') # Guard against a vacuous pass: an entity type that advertised nothing, # or a listing that came back empty, must not read as green. @@ -1274,6 +1306,16 @@ def test_every_advertised_collection_is_served(self): self.assertGreater( covered.get(entity_type, 0), 8, f'{entity_type}: only {covered.get(entity_type, 0)} hrefs followed') + # The capability list and the docs subtree are the same shape for every + # entity, so following them proves nothing about an id that has to + # resolve. A cache-built href is the only kind that can 404 from an id + # the cache does not hold, and every entity type that had a listing + # aggregates at least one data point or operation in this fixture. + for entity_type, built in sorted(covered_built.items()): + self.assertGreater( + built, 0, + f'{entity_type}: no cache-built href was followed, so no ' + f'advertised resource id was resolved') def test_the_root_list_and_the_document_agree(self): """`GET /api/v1` and `GET /api/v1/docs` describe the same gateway. @@ -1347,37 +1389,58 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): inherited from the projected route they describe, because they *are* that route. - Both scopes are compared, not merely visited. ``project()`` substitutes - the ids it was given into the path **keys**, so at collection scope the - sibling is still templated (``/data/{data_id}``) while at - specific-resource scope it has already become concrete - (``/data/temperature``). An earlier version used the collection - document only as a source of siblings, which meant every - collection-scope item could be published completely un-inherited and - this test stayed green. + Both scopes are compared. ``project()`` substitutes the ids it was + given into the path **keys**, so at collection scope the sibling is + still templated (``/data/{data_id}``) while at specific-resource scope + it has already become concrete (``/data/temperature``). The collection + document is both the source of siblings and a publisher of built + items, so its own items are compared here too. Read from the document alone: the sibling states the contract and the built item must match it, so no second source is needed and none is trusted. + + Every entity in both listings is swept, and the count is kept per + entity type, per collection and per scope: an app whose node exposes no + service publishes no built operation item at all, and the two scopes + are separate branches of the same producer, so each of those counts is + what makes its own branch falsifiable. + + The fixture is pinned on the capabilities this reads. + `REQUIRED_OPERATIONS` holds the class wait until calibration's + `calibrate` is in the cache; the poll below does the same for + temp_sensor's data, which the base class has no equivalent for. A node + is listed in the ROS graph before its services and topics have + propagated, so without both the counters are whatever the graph had + reached at the instant the request went out. """ + self.poll_endpoint_until( + '/apps/temp_sensor/data', + lambda d: d if d.get('items') else None, + timeout=DISCOVERY_TIMEOUT, + ) compared = 0 - built_items = {'data': 0, 'operations': 0} + built_items = {(entity_type, collection, scope): 0 + for entity_type in ('apps', 'components') + for collection in ('data', 'operations') + for scope in ('collection', 'resource')} offenders = [] for entity_type in ('apps', 'components'): items = self.get_json(f'/{entity_type}').get('items', []) if not items: continue - entity_id = items[0]['id'] - for collection in ('data', 'operations'): + entity_ids = [item['id'] for item in items] + pairs = [(entity_id, collection) + for entity_id in entity_ids + for collection in ('data', 'operations')] + for entity_id, collection in pairs: base = f'/{entity_type}/{entity_id}/{collection}' collection_doc = self.get_json(f'{base}/docs') collection_paths = collection_doc.get('paths', {}) # The item parameter is read from the served document, the way - # `CapabilityGenerator` reads it from the registry. Spelling - # `data_id`/`operation_id` here was a second copy of the fact - # that fix removed from production: renaming the registry - # parameter left production working and this guard green having - # compared nothing. + # `CapabilityGenerator` reads it from the registry, so the + # parameter name lives in one place and a rename there is + # followed here. template = self._item_template(collection_paths, base) self.assertIsNotNone( template, @@ -1389,7 +1452,17 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): for key, path_item in collection_paths.items(): if 'x-sovd-name' not in path_item: continue - built_items[collection] += 1 + # The key a built item is published under names the item + # the request resolves to. + self.assertTrue( + key.startswith(f'{base}/'), + f'{key}: a built item published outside {base}/') + self.assertEqual( + self._item_half(key[len(base) + 1:]), + self._item_half(path_item['x-sovd-name']), + f'{key}: the built item names ' + f'{path_item["x-sovd-name"]}') + built_items[(entity_type, collection, 'collection')] += 1 for method, operation in path_item.items(): if method not in HTTP_METHODS: continue @@ -1403,21 +1476,47 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): listing = requests.get(f'{self.BASE_URL}{base}', timeout=10) if listing.status_code != 200: continue - for entry in listing.json().get('items', []): - resource_id = entry['id'] + listed_ids = [entry['id'] + for entry in listing.json().get('items', [])] + # An operation id that names more than one operation answers + # 400 and gets no built item. The listing shows such an id + # once per operation it names, which is how it is told apart + # here without a second copy of the producer's rule. + ambiguous = {resource_id for resource_id in listed_ids + if listed_ids.count(resource_id) > 1} + for resource_id in listed_ids: scoped = requests.get( f'{self.BASE_URL}{base}/{resource_id}/docs', timeout=10) - if scoped.status_code != 200: - continue + # Every id the collection listed has a sub-document. + self.assertEqual( + scoped.status_code, 200, + f'{base}/{resource_id}/docs answered ' + f'{scoped.status_code}; the collection lists that id') key = f'{base}/{resource_id.lstrip("/")}' path_item = scoped.json().get('paths', {}).get(key, {}) + if resource_id in ambiguous: + self.assertNotIn( + 'x-sovd-name', path_item, + f'{key}: a built item for an id that names more ' + f'than one operation describes a request the ' + f'gateway refuses') + continue # `x-sovd-name` is written only by `PathBuilder`, so it is # what tells a *built* item from the projection that sits - # at the same key at this scope. Counting the key alone - # made this test unfalsifiable. - if 'x-sovd-name' not in path_item: - continue - built_items[collection] += 1 + # at the same key at this scope. The listing and the + # producer read the same cache, so every id listed here + # carries one. + self.assertIn( + 'x-sovd-name', path_item, + f'{key}: the collection lists this id and its scoped ' + f'document carries no built item for it') + self.assertEqual( + self._item_half(path_item['x-sovd-name']), + self._item_half(resource_id), + f'{key}: the built item names ' + f'{path_item["x-sovd-name"]}, the request named ' + f'{resource_id}') + built_items[(entity_type, collection, 'resource')] += 1 for method, operation in path_item.items(): if method not in HTTP_METHODS: continue @@ -1429,27 +1528,36 @@ def test_a_scoped_item_says_what_its_templated_sibling_says(self): self.assertEqual( offenders, [], f'built items contradicting their route: {offenders[:12]}') - # A cache-derived item must *exist*, and a comparison must actually have - # happened. `compared` counts comparisons performed, not operations - # visited: a sibling that is missing is a miss, not a pass, so a guard - # that found no sibling can no longer satisfy this by counting the + # A cache-derived item must exist, and a comparison must have happened. + # `compared` counts comparisons performed: a missing sibling is a miss, + # so a guard that found no sibling cannot satisfy this by counting the # operations it skipped. - # Per collection, not in total. Both are built by the same code down - # different branches, so one can vanish entirely while the other keeps - # the count above zero - which is what happened when a built verb the - # sibling lacked made every *data* item get discarded and this stayed - # green on operations alone. - for collection, count in sorted(built_items.items()): + # Per type, collection and scope. Data and operations are built by the + # same code down different branches, and the two scopes are separate + # branches again, so any one of them can stop publishing while a merged + # count stays above zero. + for (entity_type, collection, scope), count in sorted(built_items.items()): self.assertGreater( count, 0, - f'no {collection} sub-document published a cache-derived item ' - f'(none carried x-sovd-name); every comparison over ' - f'{collection} was vacuous') + f'no {entity_type} {collection} sub-document published a ' + f'cache-derived item at {scope} scope (none carried ' + f'x-sovd-name); every comparison over {entity_type} ' + f'{collection} at {scope} scope was vacuous') self.assertGreater( compared, 0, 'no built operation was compared against a sibling; the guard ran ' 'over nothing') + @staticmethod + def _item_half(item_id): + """Return the item half of a possibly member-qualified id, no leading slash. + + A short name more than one member of an entity carries is addressed + ``:``; ``x-sovd-name`` carries the item half alone. + """ + _, sep, item = item_id.partition(':') + return (item if sep else item_id).lstrip('/') + @staticmethod def _item_template(collection_paths, base): """Return the templated item route under `base`, or None. @@ -1470,9 +1578,9 @@ def _framework_mismatch(self, collection_paths, template, key, method, operation): """Compare one built operation against its templated sibling. - Returns ``(problems, compared)``. A missing sibling is reported rather - than skipped: it used to return no problems, so a lookup that found - nothing counted as a pass everywhere it was called. + Returns ``(problems, compared)``. A missing sibling is reported as a + problem, so a lookup that finds nothing is a miss everywhere this is + called. """ sibling = collection_paths.get(template, {}).get(method) if sibling is None: @@ -1483,11 +1591,9 @@ def _framework_mismatch(self, collection_paths, template, key, method, problems.append( f'{method.upper()} {key}: security ' f'{operation.get("security")} != {sibling.get("security")}') - # Every status, 2xx included. An earlier version carved 2xx out as - # "the payload, meant to differ", which is true of a request body and - # false of a response: the gateway envelopes every read - `DataValue`, - # `OperationDetail` - so a built 200 was a second, contradictory answer - # for one route rather than a more specific one. + # Every status, 2xx included: the gateway envelopes every read - + # `DataValue`, `OperationDetail` - so a built 200 is a second answer + # for one route, and it has to be the same answer. built_statuses = set(operation.get('responses', {})) sibling_statuses = set(sibling.get('responses', {})) if built_statuses != sibling_statuses: diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py new file mode 100644 index 000000000..7ace8ac09 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps.test.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""What of the gateway's own process shows up in ``/apps``. + +The gateway runs four ROS nodes in one process: itself, the subscription +executor's ``_sub``, the fault-service transport's +``_fault_clients`` and the lifecycle reader's +``_lifecycle_state_reader``. None of the three helper names begins +with ``_``, so the ROS 2 hidden-node convention leaves them in the graph and +runtime discovery would turn each into an App - the gateway advertising its own +plumbing as something an operator can diagnose. + +The gateway's own node is the opposite case and stays an App: its ROS +parameters are what SOVD serves as that App's configurations, so +``/apps//configurations`` is the only place a caller can read or write +``aggregation.peer_auth_header`` and its neighbours. Two gateways watching one +graph also have to agree on what is on it, which they cannot do if each hides +a different node. + +This fixture launches the gateway with no ``__node`` remap, the way ``ros2 run`` +and the container images start it. ``launch_ros``' ``name=`` applies +``-r __node:=`` to the whole process, which renames all four nodes to the +same string - so under the suite's usual launch the helper names do not exist +and nothing here could be observed. + +An absence assertion is only worth reading if the thing could have been there. +The gateway's helper nodes are created at three different points of start-up, +two of them after the first ``refresh_cache()`` and after ``/health`` starts +answering, so a naive "wait until /apps is non-empty" can read a list built +from a graph that did not yet contain them - and then the absences below hold +whatever the filter does. The settle sequence therefore proves the order it +needs: all three helpers on the graph FIRST, then a witness node created after +them, then an ``/apps`` snapshot that contains the witness. Such a snapshot was +built from a graph that held the helpers too. +""" + +import time +import unittest + +import launch +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NODE = 'ros2_medkit_gateway' +HELPER_NODES = ( + f'{GATEWAY_NODE}_sub', + f'{GATEWAY_NODE}_fault_clients', + f'{GATEWAY_NODE}_lifecycle_state_reader', +) +WITNESS_NODE = 'own_node_apps_refresh_witness' + +# The three stages of the settle sequence. They run one after another, so their +# sum plus launch and teardown has to stay inside this file's ctest TIMEOUT +# (the feature default, 120 s): a run killed by ctest reports a timeout and no +# test name, which hides whichever assertion actually failed. +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +REFRESH_WITNESS_BUDGET = 25.0 + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one bare gateway, keeping its process's real node names.""" + gateway_node = create_gateway_node( + port=GATEWAY_PORT, + name=None, + extra_params={'server.host': '127.0.0.1', 'refresh_interval_ms': 1000}, + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +class TestOwnNodeApps(unittest.TestCase): + """The gateway is a diagnosable App; its in-process helpers are not.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway, then settle /apps against the helper nodes.""" + cls.session = requests.Session() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node('own_node_apps_graph_probe', context=cls.context) + cls.witness = None + cls.apps = set() + cls.graph_fqns = set() + cls.witness_seen = False + + cls._wait_for_health() + # Stage 1: every helper the absences below are about must be on the + # graph before anything reads /apps. + cls.graph_fqns = cls._graph_fqns_until( + {f'/{helper}' for helper in HELPER_NODES}, HELPERS_ON_GRAPH_BUDGET) + if not {f'/{helper}' for helper in HELPER_NODES} <= cls.graph_fqns: + return + + # Stage 2: a node created strictly after the helpers appeared. Its own + # arrival in /apps dates the snapshot: the gateway cannot have seen the + # witness without having seen the helpers. + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.witness_seen, cls.apps = cls._apps_until_contains( + WITNESS_NODE, REFRESH_WITNESS_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + return + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') + + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Node FQNs on the graph, polled until *awaited* is a subset. + + The graph query reads the discovery database directly, so polling is + what this needs. Returns the last set seen even on timeout - the caller + asserts on it, so a timeout cannot pass silently. + """ + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + + @classmethod + def _apps_until_contains(cls, app_id, budget): + """Poll /apps until *app_id* is listed. Returns (seen, last snapshot).""" + deadline = time.monotonic() + budget * get_time_scale() + apps = set() + while True: + body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() + apps = {item['id'] for item in body.get('items', [])} + if app_id in apps: + return True, apps + if time.monotonic() >= deadline: + return False, apps + time.sleep(0.5) + + def test_the_gateways_own_node_is_an_app(self): + """The gateway's own ROS node is listed, addressable and configurable. + + @verifies REQ_INTEROP_003 + """ + self.assertIn( + GATEWAY_NODE, self.apps, + f'the gateway node must be a diagnosable App. Listed: {sorted(self.apps)}') + + detail = self.session.get(f'{BASE_URL}/apps/{GATEWAY_NODE}', timeout=10) + self.assertEqual(detail.status_code, 200, detail.text) + self.assertEqual(detail.json()['id'], GATEWAY_NODE) + + configurations = self.session.get( + f'{BASE_URL}/apps/{GATEWAY_NODE}/configurations', timeout=10) + self.assertEqual(configurations.status_code, 200, configurations.text) + items = configurations.json().get('items', []) + self.assertTrue( + items, + 'the gateway App must serve its own ROS parameters as configurations; ' + 'they are reachable nowhere else') + + def test_the_in_process_helper_nodes_are_not_apps(self): + """The gateway's own helper nodes stay out of the app list. + + @verifies REQ_INTEROP_003 + """ + # Anti-vacuity, first half: each helper really is a node on this graph, + # so the absences below are about the filter and not about names that + # were never there. + awaited = {f'/{helper}' for helper in HELPER_NODES} + self.assertTrue( + awaited <= self.graph_fqns, + f'helper nodes missing from the graph: {sorted(awaited - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + + # Anti-vacuity, second half: the snapshot was rebuilt after they + # appeared. Without this the assertions below can be read from a list + # the gateway built before it could have listed a helper at all. + self.assertTrue( + self.witness_seen, + f'/apps never listed "{WITNESS_NODE}", so no snapshot is known to ' + f'post-date the helper nodes and these absences prove nothing. ' + f'Listed: {sorted(self.apps)}') + + for helper in HELPER_NODES: + self.assertNotIn( + helper, self.apps, + f'in-process helper "{helper}" must not be a diagnosable App. ' + f'Listed: {sorted(self.apps)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py new file mode 100644 index 000000000..941dbe3ec --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_apps_namespaced.test.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The helper-node filter holds when the gateway alone is moved to a namespace. + +``test_own_node_apps.test.py`` covers the case where all four of the gateway's +nodes share one namespace, where the three helper names are simply the +gateway's plus a suffix. That is not the only shape they take. The subscription +node is created with the gateway's namespace, but the fault-client and +lifecycle-reader nodes are created from the gateway's node NAME alone, so they +take the process default namespace - and a remap naming the gateway alone, +``-r ros2_medkit_gateway:__ns:=/subsystem_b``, moves the gateway and the +subscription node while leaving those two in the root namespace. + +The gateway must still recognise all three. This launch is the split +invocation, with one gateway on the graph so nothing else can supply a node by +those names. +""" + +import time +import unittest + +import launch +import launch_ros.actions +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.coverage import get_coverage_env + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NAME = 'ros2_medkit_gateway' +GATEWAY_NS = '/subsystem_b' +GATEWAY_FQN = f'{GATEWAY_NS}/{GATEWAY_NAME}' +# The split: one helper follows the gateway, two stay at the root. +NAMESPACED_HELPER_FQN = f'{GATEWAY_NS}/{GATEWAY_NAME}_sub' +ROOT_HELPER_FQNS = ( + f'/{GATEWAY_NAME}_fault_clients', + f'/{GATEWAY_NAME}_lifecycle_state_reader', +) +ALL_HELPER_FQNS = (NAMESPACED_HELPER_FQN,) + ROOT_HELPER_FQNS +WITNESS_NODE = 'own_node_apps_ns_witness' +PROBE_NODE = 'own_node_apps_ns_probe' +# This file's own two nodes are ordinary ROS nodes, so the gateway lists them +# as apps like any other. Naming them keeps the assertion an exact set rather +# than a membership test that would not notice a helper slipping back in. +TEST_OWN_NODES = {WITNESS_NODE, PROBE_NODE} + +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +REFRESH_WITNESS_BUDGET = 25.0 + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one gateway with only its own node moved into a namespace. + + Built here because the remap has to name the gateway node, which + ``create_gateway_node`` cannot express: a namespace on the launch action + moves all four nodes together, and that case is the sibling file's. + """ + gateway_node = launch_ros.actions.Node( + package='ros2_medkit_gateway', + executable='gateway_node', + name=None, + output='screen', + parameters=[{ + 'server.host': '127.0.0.1', + 'server.port': GATEWAY_PORT, + 'refresh_interval_ms': 1000, + }], + ros_arguments=['-r', f'{GATEWAY_NAME}:__ns:={GATEWAY_NS}'], + additional_env=dict(get_coverage_env()), + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +class TestOwnNodeAppsNamespaced(unittest.TestCase): + """A gateway moved on its own still filters all three of its helpers.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway, then settle /apps against the helper nodes.""" + cls.session = requests.Session() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node(PROBE_NODE, context=cls.context) + cls.witness = None + cls.apps = set() + cls.graph_fqns = set() + cls.witness_seen = False + + cls._wait_for_health() + cls.graph_fqns = cls._graph_fqns_until(set(ALL_HELPER_FQNS), HELPERS_ON_GRAPH_BUDGET) + if not set(ALL_HELPER_FQNS) <= cls.graph_fqns: + return + + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.witness_seen, cls.apps = cls._apps_until_contains( + WITNESS_NODE, REFRESH_WITNESS_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + return + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') + + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Poll the graph until *awaited* is a subset of the node FQNs.""" + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + + @classmethod + def _apps_until_contains(cls, app_id, budget): + """Poll /apps until *app_id* is listed. Returns (seen, last snapshot).""" + deadline = time.monotonic() + budget * get_time_scale() + apps = set() + while True: + body = cls.session.get(f'{BASE_URL}/apps', timeout=10).json() + apps = {item['id'] for item in body.get('items', [])} + if app_id in apps: + return True, apps + if time.monotonic() >= deadline: + return False, apps + time.sleep(0.5) + + def test_the_split_helper_nodes_are_all_on_the_graph(self): + """The remap really does split them, so the absences below mean something. + + Without this the next test would pass on a launch where the namespace + remap silently did nothing and the helpers never existed under these + names. + """ + self.assertTrue( + set(ALL_HELPER_FQNS) <= self.graph_fqns, + f'expected the gateway and its subscription node in {GATEWAY_NS} and the ' + f'other two helpers at the root, missing: ' + f'{sorted(set(ALL_HELPER_FQNS) - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + self.assertIn( + GATEWAY_FQN, self.graph_fqns, + f'the gateway node is not in {GATEWAY_NS}, so the remap did not apply') + + def test_apps_lists_the_gateway_and_none_of_its_helpers(self): + """A namespaced gateway serves itself and none of its plumbing. + + @verifies REQ_INTEROP_003 + """ + self.assertTrue( + self.witness_seen, + f'/apps never listed "{WITNESS_NODE}", so no snapshot is known to ' + f'post-date the helper nodes. Listed: {sorted(self.apps)}') + + self.assertEqual( + self.apps, {GATEWAY_NAME} | TEST_OWN_NODES, + f'a gateway alone on the graph must serve its own node and nothing ' + f"of its plumbing, beside this test's own two nodes. " + f'Listed: {sorted(self.apps)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py new file mode 100644 index 000000000..aafd08ac9 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_own_node_undeclared.test.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The gateway does not ask the operator to declare its own plumbing. + +In hybrid mode with ``unmanifested_nodes: error``, every running node that no +manifest app binds is reported on ``GET /health`` as an ``unmanifested_nodes`` +warning, and the message is an instruction: "Declare them in the manifest". + +The gateway's three in-process helper nodes must not be in that list. They can +never become apps - the app filter removes them - so declaring them would +silence the warning and produce manifest entities the gateway then deletes. The +two halves have to agree: what is never an App is never something to declare. + +Launched without a ``__node`` remap for the same reason as +``test_own_node_apps.test.py``: ``launch_ros``' ``name=`` renames every node in +the process, so under the suite's usual launch the helper names do not exist +and nothing here could be observed. +""" + +import os +import tempfile +import time +import unittest + +import launch +import launch_testing +import launch_testing.actions +import pytest +import rclpy +from rclpy.context import Context +from rclpy.node import Node +import requests + +from ros2_medkit_test_utils.constants import ( + ALLOWED_EXIT_CODES, + API_BASE_PATH, + get_test_port, + get_time_scale, +) +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +GATEWAY_PORT = get_test_port() +BASE_URL = f'http://127.0.0.1:{GATEWAY_PORT}{API_BASE_PATH}' + +GATEWAY_NODE = 'ros2_medkit_gateway' +HELPER_FQNS = ( + f'/{GATEWAY_NODE}_sub', + f'/{GATEWAY_NODE}_fault_clients', + f'/{GATEWAY_NODE}_lifecycle_state_reader', +) +# An undeclared node of the test's own, so the warning below is known to have +# something to list. +WITNESS_NODE = 'own_node_undeclared_witness' + +WARN_UNMANIFESTED_NODES = 'unmanifested_nodes' + +HEALTH_BUDGET = 30.0 +HELPERS_ON_GRAPH_BUDGET = 20.0 +WARNING_BUDGET = 25.0 + +_MANIFEST_DIR = tempfile.mkdtemp(prefix='medkit-own-node-undeclared-') +_MANIFEST_PATH = os.path.join(_MANIFEST_DIR, 'manifest.yaml') + +# One app bound to a node this launch never starts: the manifest is valid and +# loaded, nothing links, so every running node is undeclared and the policy has +# something to report. +with open(_MANIFEST_PATH, 'w') as _manifest: + _manifest.write("""\ +manifest_version: "1.0" +metadata: + name: "Own node undeclared test vehicle" + version: "1.0.0" +config: + unmanifested_nodes: "error" +areas: + - id: test_area + name: "Test Area" +components: + - id: test_ecu + name: "Test ECU" + area: test_area +apps: + - id: absent_app + name: "An app whose node is not running" + is_located_on: test_ecu + ros_binding: + node_name: absent_node + namespace: /nowhere +""") + + +@pytest.mark.launch_test +def generate_test_description(): + """Launch one hybrid gateway keeping its process's real node names.""" + gateway_node = create_gateway_node( + port=GATEWAY_PORT, + name=None, + extra_params={ + 'server.host': '127.0.0.1', + 'refresh_interval_ms': 1000, + 'discovery.mode': 'hybrid', + 'discovery.manifest_path': _MANIFEST_PATH, + }, + ) + + return launch.LaunchDescription([ + gateway_node, + launch_testing.actions.ReadyToTest(), + ]), {'gateway_node': gateway_node} + + +class TestOwnNodeUndeclared(unittest.TestCase): + """The undeclared-node warning and the app filter agree on the helpers.""" + + @classmethod + def setUpClass(cls): + """Wait for the gateway, then settle the warning against the helpers.""" + cls.session = requests.Session() + cls.context = Context() + rclpy.init(context=cls.context) + cls.probe = Node('own_node_undeclared_probe', context=cls.context) + cls.witness = None + cls.graph_fqns = set() + cls.warning = None + + cls._wait_for_health() + cls.graph_fqns = cls._graph_fqns_until(set(HELPER_FQNS), HELPERS_ON_GRAPH_BUDGET) + if not set(HELPER_FQNS) <= cls.graph_fqns: + return + + # Created after the helpers, and undeclared like them. Waiting for the + # warning to name it dates the report: a report that has seen the + # witness has seen the helpers, which appeared earlier. + cls.witness = Node(WITNESS_NODE, context=cls.context) + cls.warning = cls._warning_until_lists(f'/{WITNESS_NODE}', WARNING_BUDGET) + + @classmethod + def tearDownClass(cls): + if cls.witness is not None: + cls.witness.destroy_node() + cls.probe.destroy_node() + rclpy.shutdown(context=cls.context) + cls.session.close() + + @classmethod + def _wait_for_health(cls): + deadline = time.monotonic() + HEALTH_BUDGET * get_time_scale() + last = None + while time.monotonic() < deadline: + try: + response = cls.session.get(f'{BASE_URL}/health', timeout=5) + if response.status_code == 200: + return + last = response.status_code + except requests.RequestException as exc: + last = str(exc) + time.sleep(0.5) + raise AssertionError( + f'gateway not ready within {HEALTH_BUDGET}s (last: {last})') + + @classmethod + def _graph_fqns_until(cls, awaited, budget): + """Node FQNs on the graph, polled until *awaited* is a subset.""" + deadline = time.monotonic() + budget * get_time_scale() + while True: + fqns = { + (namespace.rstrip('/') + '/' + name) + for name, namespace in cls.probe.get_node_names_and_namespaces() + } + if awaited <= fqns or time.monotonic() >= deadline: + return fqns + time.sleep(0.2) + + @classmethod + def _warning_until_lists(cls, node_fqn, budget): + """Poll /health until the unmanifested_nodes warning names *node_fqn*.""" + deadline = time.monotonic() + budget * get_time_scale() + warning = None + while True: + body = cls.session.get(f'{BASE_URL}/health', timeout=10).json() + for candidate in body.get('warnings', []): + if candidate.get('code') == WARN_UNMANIFESTED_NODES: + warning = candidate + break + if warning is not None and node_fqn in warning.get('ros_node_fqns', []): + return warning + if time.monotonic() >= deadline: + return warning + time.sleep(0.5) + + def test_the_undeclared_warning_is_reporting(self): + """The policy is active and the warning lists the nodes it found. + + Without this the absence check below would pass on a gateway that + reported nothing at all. + """ + self.assertTrue( + set(HELPER_FQNS) <= self.graph_fqns, + f'helper nodes missing from the graph: ' + f'{sorted(set(HELPER_FQNS) - self.graph_fqns)}. ' + f'Nodes seen: {sorted(self.graph_fqns)}') + + self.assertIsNotNone( + self.warning, + f'no "{WARN_UNMANIFESTED_NODES}" warning on /health, so this file ' + f'cannot tell a filtered helper from an unreported one') + self.assertIn( + f'/{WITNESS_NODE}', self.warning.get('ros_node_fqns', []), + f'the warning never named "{WITNESS_NODE}", so no report is known ' + f'to post-date the helper nodes. Listed: ' + f'{sorted(self.warning.get("ros_node_fqns", []))}') + + def test_the_helper_nodes_are_not_reported_as_undeclared(self): + """The gateway's own helper nodes are not something to declare.""" + self.assertIsNotNone(self.warning, 'no unmanifested_nodes warning to read') + reported = set(self.warning.get('ros_node_fqns', [])) + self.assertFalse( + reported & set(HELPER_FQNS), + f'the gateway asked the operator to declare its own in-process ' + f'helper nodes: {sorted(reported & set(HELPER_FQNS))}. Declaring ' + f'them produces entities the app filter then removes. Reported: ' + f'{sorted(reported)}') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """Check all processes exited cleanly (SIGTERM allowed).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py b/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py new file mode 100644 index 000000000..9edd4d43f --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_param_beacon_out_of_range_config.test.py @@ -0,0 +1,624 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""parameter_beacon duration and max_hints bounds, and polling of nodes that answer badly or never. + +One gateway loads the plugin once per instance, each with its own prefix. The sweep sets +durations to NaN, -inf, below the minimum, the minimum, the maximum, just above it, +inf and +3e9; the steady instance has a working configuration. More instances take one max_hints point +each: 0, -5, 1, 2147483647, 2147483648, 2^32 + 1, 1e12 and NaN. The instance loaded first +answers the beacon endpoint and has NaN TTL and expiry. Polled nodes: answering, unserved +(never answers), get_silent (answers list only), typed_unset (rclpy, a typed parameter without +a value, so its get answers carry no values), two hint sources that give each max_hints +instance a hint of its own, and lifetime_beacon, which gives the first instance a hint for +itself until the test stops it. +""" + +import collections +import os +import tempfile +import threading +import time +import unittest + +from ament_index_python.packages import get_package_prefix +from launch import LaunchDescription +import launch_testing +import launch_testing.actions +from rcl_interfaces.msg import ParameterType, ParameterValue +from rcl_interfaces.srv import GetParameters, ListParameters +import rclpy +from rclpy.executors import SingleThreadedExecutor +from rclpy.parameter import Parameter +import requests + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, DEFAULT_BASE_URL, get_time_scale +from ros2_medkit_test_utils.launch_helpers import create_gateway_node + +TIME_SCALE = get_time_scale() +APPEAR_TIMEOUT_SEC = 30.0 * TIME_SCALE +# Give-up bound for anything the instances do; every wait ends as soon as its condition holds. +POLL_TIMEOUT_SEC = 30.0 * TIME_SCALE +# A thread busier than this over the CPU window is spinning. +SPIN_CPU_FRACTION = 0.5 +CPU_WINDOW_SEC = 3.0 + +MAX_SECONDS = '2147483647' +DURATIONS = ('poll_interval_sec', 'poll_budget_sec', 'param_timeout_sec', 'beacon_ttl_sec', + 'beacon_expiry_sec') +MINIMUM = dict.fromkeys(DURATIONS, '0.1') +MINIMUM['beacon_expiry_sec'] = '1' +BELOW = dict.fromkeys(DURATIONS, '0.05') +BELOW['beacon_expiry_sec'] = '0.5' + + +def _point(name, key): + """(YAML spelling, how the plugin prints it, what it clamps to or None) of a sweep point.""" + return { + 'nan': ('.nan', 'nan', MINIMUM[key]), + 'neg_inf': ('-.inf', '-inf', MINIMUM[key]), + 'below': (BELOW[key], BELOW[key], MINIMUM[key]), + 'min': (MINIMUM[key] if key != 'beacon_expiry_sec' else '1.0', MINIMUM[key], None), + 'max': (MAX_SECONDS + '.0', MAX_SECONDS, None), + 'above': ('2147483648.0', '2147483648', MAX_SECONDS), + 'inf': ('.inf', 'inf', MAX_SECONDS), + 'huge': ('3000000000.0', '3000000000', MAX_SECONDS), + }[name] + + +LOW_POINTS = ('nan', 'neg_inf', 'below', 'min') +HIGH_POINTS = ('max', 'above', 'inf', 'huge') +# Every duration at a low point: the instance keeps cycling with 0.1 s timeouts. +CYCLING = {f'low_{p}': {key: p for key in DURATIONS} for p in LOW_POINTS} +# Timeout, budget, TTL and expiry at a high point, the interval at its minimum: the instance +# reaches a node that never answers and waits there. +WAITING = {f'high_{p}': dict({key: p for key in DURATIONS[1:]}, poll_interval_sec='min') + for p in HIGH_POINTS} +# The interval at a high point: the instance runs one cycle. +ONE_CYCLE = {f'interval_{p}': {'poll_interval_sec': p} for p in HIGH_POINTS} +SWEEP = dict(**CYCLING, **WAITING, **ONE_CYCLE) +STEADY = 'steady' +STEADY_CONFIG = {'poll_interval_sec': '0.1', 'poll_budget_sec': '30.0', 'param_timeout_sec': '0.3'} +# Bound for the steady counts to settle: a list answer of get_silent is counted before its get. +SETTLE_TIMEOUT_SEC = 1.0 * TIME_SCALE + +# max_hints points: instance -> (YAML spelling, warning or None, hints kept of the two sources) +HINT_INSTANCES = { + 'hints_zero': ('0', 'max_hints clamped from 0 to 1', 1), + 'hints_negative': ('-5', 'max_hints clamped from -5 to 1', 1), + 'hints_min': ('1', None, 1), + 'hints_max': ('2147483647', None, 2), + 'hints_above': ('2147483648', 'max_hints clamped from 2147483648 to 2147483647', 2), + 'hints_wrapping': ('4294967297', 'max_hints clamped from 4294967297 to 2147483647', 2), + 'hints_double': ('1.0e12', 'max_hints 1e+12 is not an integer, using 10000', 2), + 'hints_nan': ('.nan', 'max_hints nan is not an integer, using 10000', 2), +} +HINT_CONFIG = {'poll_interval_sec': '1.0', 'param_timeout_sec': '0.1'} +HINT_SOURCES = ('a', 'b') +CAPACITY_WARNING = 'BeaconHintStore capacity reached (max_hints=1)' + +# Loaded first, so its route answers the beacon endpoint. TTL becomes 3 * 0.1 s, expiry 1 s. +LIFETIME = 'lifetime' +LIFETIME_CONFIG = {'poll_interval_sec': '0.1', 'param_timeout_sec': '0.3', + 'beacon_ttl_sec': '.nan', 'beacon_expiry_sec': '.nan'} +LIFETIME_TTL_SEC = 0.3 +LIFETIME_EXPIRY_SEC = 1.0 +# How late the test may see the beacon go stale or be removed. +LIFETIME_LATENESS_SEC = 0.3 * TIME_SCALE + + +def _prefix(instance): + return f'oor_{instance}' + + +def _parameter_file(): + plugin_path = os.path.join( + get_package_prefix('ros2_medkit_param_beacon'), 'lib', 'ros2_medkit_param_beacon', + 'libparam_beacon_plugin.so') + settings = {LIFETIME: LIFETIME_CONFIG} + for instance, points in SWEEP.items(): + settings[instance] = {key: _point(point, key)[0] for key, point in points.items()} + settings[STEADY] = STEADY_CONFIG + for instance, (value, _, _) in HINT_INSTANCES.items(): + settings[instance] = dict(HINT_CONFIG, max_hints=value) + lines = [ + 'ros2_medkit_gateway:', + ' ros__parameters:', + ' plugins: [' + ', '.join(settings) + ']', + ] + for instance, values in settings.items(): + lines.append(f' plugins.{instance}.path: "{plugin_path}"') + lines.append(f' plugins.{instance}.parameter_prefix: "{_prefix(instance)}"') + for key, value in values.items(): + lines.append(f' plugins.{instance}.{key}: {value}') + handle = tempfile.NamedTemporaryFile( + 'w', prefix='param_beacon_bounds_', suffix='.yaml', delete=False) + with handle: + handle.write('\n'.join(lines) + '\n') + return handle.name + + +PARAMETER_FILE = _parameter_file() + + +def generate_test_description(): + gateway_node = create_gateway_node(parameter_files=[PARAMETER_FILE]) + return ( + LaunchDescription([gateway_node, launch_testing.actions.ReadyToTest()]), + {'gateway_node': gateway_node}, + ) + + +def _prefix_of(name): + return name.split('.', 1)[0] + + +class _Counter: + """Thread-safe request counts per (node, service, prefix).""" + + def __init__(self): + self._counts = collections.Counter() + self._lock = threading.Lock() + + def add(self, node, service, prefixes): + with self._lock: + for prefix in prefixes: + self._counts[(node, service, prefix)] += 1 + + def get(self, node, service, instance): + with self._lock: + return self._counts[(node, service, _prefix(instance))] + + def snapshot(self): + with self._lock: + return collections.Counter(self._counts) + + +class _UnansweredServices: + """Takes requests from services no executor serves, counts them and never answers.""" + + def __init__(self, counter): + self._counter = counter + self._services = [] + self._stop = threading.Event() + self._thread = threading.Thread(target=self._run, daemon=True) + + def add(self, node_label, service): + self._services.append((node_label, service)) + + def start(self): + self._thread.start() + + def stop(self): + self._stop.set() + self._thread.join(timeout=10) + + def _run(self): + while not self._stop.is_set(): + for node_label, service in self._services: + while True: + with service.handle: + request, _header = service.handle.service_take_request( + service.srv_type.Request) + if request is None: + break + if service.srv_type is ListParameters: + self._counter.add(node_label, 'list', request.prefixes) + else: + self._counter.add(node_label, 'get', + {_prefix_of(name) for name in request.names}) + self._stop.wait(0.02) + + +def _thread_cpu_ticks(pid): + ticks = {} + for tid in os.listdir(f'/proc/{pid}/task'): + try: + with open(f'/proc/{pid}/task/{tid}/stat') as stat: + fields = stat.read().rsplit(')', 1)[1].split() + except OSError: + continue + ticks[tid] = int(fields[11]) + int(fields[12]) + return ticks + + +class TestParamBeaconBounds(unittest.TestCase): + + @classmethod + def setUpClass(cls): + deadline = time.monotonic() + APPEAR_TIMEOUT_SEC + while True: + try: + if requests.get(f'{DEFAULT_BASE_URL}/health', timeout=2).status_code == 200: + break + except requests.exceptions.RequestException: + pass + if time.monotonic() > deadline: + raise AssertionError('the gateway never answered GET /health') + time.sleep(0.2) + rclpy.init() + cls.counter = _Counter() + cls.nodes = [] + + # The max_hints instances get their hints from the hint sources only. + hint_prefixes = {_prefix(instance) for instance in HINT_INSTANCES} + answering = cls._node('answering_beacon') + answering.create_service( + ListParameters, '~/list_parameters', cls._answer_list('answering', hint_prefixes)) + answering.create_service(GetParameters, '~/get_parameters', cls._answer_get) + + get_silent = cls._node('get_silent') + get_silent.create_service( + ListParameters, '~/list_parameters', cls._answer_list('get_silent')) + # Its get service lives on a hidden node no executor spins. + get_backend = cls._node('_get_silent_backend') + get_silent_get = get_backend.create_service( + GetParameters, '/get_silent/get_parameters', cls._answer_get) + + cls.unserved = cls._node('unserved_parameters') + unserved_list = cls.unserved.create_service( + ListParameters, '~/list_parameters', cls._answer_list('unserved')) + unserved_get = cls.unserved.create_service( + GetParameters, '~/get_parameters', cls._answer_get) + + typed_unset = rclpy.create_node('typed_unset') + cls.nodes.append(typed_unset) + typed_unset.declare_parameter(f'{_prefix(STEADY)}.entity_id', 'typed_unset') + typed_unset.declare_parameter(f'{_prefix(STEADY)}.display_name', 'Typed Unset') + typed_unset.declare_parameter(f'{_prefix(STEADY)}.stable_id', Parameter.Type.STRING) + get_service = next( + service for service in typed_unset.services + if service.srv_name.endswith('/get_parameters')) + rclpy_get = get_service.callback + + def counting_get(request, response): + cls.counter.add('typed_unset', 'get', {_prefix_of(name) for name in request.names}) + return rclpy_get(request, response) + + get_service.callback = counting_get + + sources = [] + for label in HINT_SOURCES: + source = cls._node(f'hint_source_{label}') + source.create_service(ListParameters, '~/list_parameters', + cls._answer_list_for(hint_prefixes)) + source.create_service(GetParameters, '~/get_parameters', cls._answer_hint(label)) + sources.append(source) + + cls.lifetime_serving = threading.Event() + cls.lifetime_serving.set() + cls.lifetime_last_get = None + lifetime = cls._node('lifetime_beacon') + lifetime.create_service(ListParameters, '~/list_parameters', cls._answer_lifetime_list) + lifetime.create_service(GetParameters, '~/get_parameters', cls._answer_lifetime_get) + + cls.unanswered = _UnansweredServices(cls.counter) + cls.unanswered.add('unserved', unserved_list) + cls.unanswered.add('unserved', unserved_get) + cls.unanswered.add('get_silent', get_silent_get) + cls.unanswered.start() + + cls.executor = SingleThreadedExecutor() + for node in (answering, get_silent, typed_unset, *sources, lifetime): + cls.executor.add_node(node) + cls.spin_thread = threading.Thread(target=cls.executor.spin, daemon=True) + cls.spin_thread.start() + + @classmethod + def tearDownClass(cls): + cls.unanswered.stop() + cls.executor.shutdown() + cls.spin_thread.join(timeout=10) + for node in cls.nodes: + node.destroy_node() + rclpy.shutdown() + os.remove(PARAMETER_FILE) + + @classmethod + def _node(cls, name): + node = rclpy.create_node(name, start_parameter_services=False) + cls.nodes.append(node) + return node + + @classmethod + def _answer_list(cls, label, skipped=frozenset()): + def answer(request, response): + cls.counter.add(label, 'list', request.prefixes) + response.result.names = [ + f'{prefix}.entity_id' for prefix in request.prefixes if prefix not in skipped] + return response + return answer + + @staticmethod + def _answer_get(request, response): + response.values = [ + ParameterValue(type=ParameterType.PARAMETER_STRING, string_value='answering_beacon') + for _ in request.names] + return response + + @staticmethod + def _answer_list_for(prefixes): + def answer(request, response): + response.result.names = [ + f'{prefix}.entity_id' for prefix in request.prefixes if prefix in prefixes] + return response + return answer + + @staticmethod + def _answer_hint(label): + """Entity id '_