From edecb9549ee8aa54136e7f6e7507610de8a405f0 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Fri, 21 Aug 2026 17:31:08 +0100 Subject: [PATCH 1/6] DS ADR: low-level detector modules --- .../ADRs/010_detector_settings.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md new file mode 100644 index 0000000000..6a151df0cc --- /dev/null +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -0,0 +1,91 @@ +# 10 - handling low-level detector settings + +## Status + +- Provisional until agreed with DSG +- Requires prioritisation against other streaming work + +## Context + +`kafka_dae_control` primarily talks to the streaming control board (SCB). All user-facing operations are routed through this board, for example: +- Beginning and ending runs +- Configuring hardware vetoes +- Configuring time-sync source + +We want all **user-facing** workflows to remain possible purely via the streaming control board. For example, we do **not** want to end up in a situation where we need to tell each individual detector to begin a run. The streaming control board should remain the common control interface to all streaming instruments. + +However, for 'diagnostic' workflows, Detector Systems Group (DSG) would also like to be able to read and write some parameters on individual detector boards through IBEX. This may include workflows like: +- Configuring detector-specific parameters, for example discriminator thresholds or maximum event-rates before triggering a local 'overcount' veto. +- Verifying that a detector's configuration is as expected after a hardware reboot or a swap. +- Monitoring detector-specific diagnostic parameters, for example temperatures or local event rates. + +The set of 'individual' detector boards is likely to be large, varied, and different on different instruments. For example, HRPD-X will have ~80 detector modules; SANDALS-2 will use an entirely different type of detector modules. For this reason, it is not sustainable for IBEX to individually cater for the quirks of every detector board on every instrument. + +## Decision + +### Detector board self-description + +Individual detector boards which want to participate in this scheme will: +- Publish a UDP comms interface using the same request/response mechanism as the SCB +- In a register which is common across all detector boards, publish an identifier which uniquely identifies a specific memory map. +- This memory map will specify a mapping of `name <-> register` for all parameters this board can expose to IBEX. + +### Configuration + +We would add a section in [`kafka_dae_control`'s config file](https://github.com/ISISComputingGroup/kafka_dae_control/blob/main/config.example.toml), which looks like: + +```toml +[diagnostic_modules.mod1] +ip = "192.168.1.21" +pv_suffix = "MOD1" +parameters = [ + { reg_name = "temp", "pv_name" = "TEMP", write = false }, + { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, + { reg_name = "super_special_parameter_for_mod1", "pv_name" = "SUPER_SPECIAL", write = true }, +] + +[diagnostic_modules.mod2] +ip = "192.168.1.22" +pv_suffix = "MOD2" +parameters = [ + { reg_name = "temp", "pv_name" = "TEMP", write = false }, + { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, +] +``` + +### Runtime + +When `kafka_dae_control` starts, it will create a thread dedicated to communication with 'diagnostic' modules. + +It would then: +- Read register `0` of each configured diagnostic module to retrieve a memory-map identifier +- Look up that register map identifier in a central store to retrieve a 'full' memory map for this board + - The central store could be cached locally on startup to ensure `kafka_dae_control` still boots correctly if the central store is offline. + - In any case, a failure in the 'diagnostic' functionality should not prevent the critical functionality of `kafka_dae_control` from working. +- Use the retrieved mappings to map each configured register (in the `parameters` of the `config.toml` to a numeric address) +- A dedicated thread in `kafka_dae_control` would attempt to poll each diagnostic register in turn, looping for the lifetime of the program. +- The updated numbers would be served in PVs of the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL`. This allows them to be accessible to IBEX, monitored by Nagios, or consumed by DSG's monitoring infrastructure. + +### Writing + +`kafka_dae_control` would also create standard setpoint PVs for each writeable parameter, in the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL:SP`. + +## Alternatives + +- In the first instance, we could avoid the architectural complexity of a central memory-map store and self-describing boards by requiring a `reg_address` in the `config.toml`. + - Mapping via `reg_name` and self-description could still be added later if desired +- We could make this an entirely separate process from `kafka_dae_control`, which happens to be implemented in a similar way. + - Advantage: This would better insulate the *critical* functionality in `kafka_dae_control` from the non-critical functionality of providing diagnostics on individual detector modules. + - Disadvantage: There would be some duplication between `kafka_dae_control` and this new process; both would be doing UDP comms to boards with a similar interface, and serving PVs over PVAccess. + +## Risks + +Providing a route through to individual detectors from IBEX may encourage architectural shortcuts to be taken later. For example, a parameter that should conceptually be set via the SCB may get set on individual detectors instead of adding the relevant functionality to the SCB. Over time, these shortcuts may accumulate and increase system maintenance burden to an unsustainable level. + +If instruments begin accumulating scripts or workflows which involve 'fiddling' with detector parameters directly, those instruments will become much harder to migrate to different detectors in future. It will reduce the commonality between instruments, which will increase system maintenance burden. + +## Consequences + +- It is possible to read and write a specified set of registers from individual detector modules from EPICS PVs. This set is statically configurable per-instrument. +- We increase the risk of architectural 'shortcuts' being taken later which, if taken, would adversely impact maintainability. +- `kafka_dae_control` becomes more complicated. From 5f60c9d9b7bf511255d51b586e3400223ebca565 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Mon, 24 Aug 2026 10:40:16 +0100 Subject: [PATCH 2/6] Comments --- .../ADRs/010_detector_settings.md | 28 +++++++++++++++---- doc/spelling_wordlist.txt | 1 + 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md index 6a151df0cc..6cea14a1e7 100644 --- a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -35,22 +35,34 @@ Individual detector boards which want to participate in this scheme will: We would add a section in [`kafka_dae_control`'s config file](https://github.com/ISISComputingGroup/kafka_dae_control/blob/main/config.example.toml), which looks like: ```toml +# 'Parameter groups' define shared sets of parameters which may exist on +# multiple boards, to help reduce repetition. +[diagnostic_parameter_groups.temperature] +parameters = [ + { reg_name = "temp1", "pv_name" = "TEMP1", write = false }, + { reg_name = "temp2", "pv_name" = "TEMP2", write = false }, +] + +[diagnostic_parameter_groups.event_rate] +parameters = [ + { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, +] + +# Each board defines it's key parameters (ip, PV name). +# It then defines any parameters and parameter_groups which it uses; these are merged +# to form the list of parameters read by this board. [diagnostic_modules.mod1] ip = "192.168.1.21" pv_suffix = "MOD1" +parameter_groups = ["temperature", "event_rate"] parameters = [ - { reg_name = "temp", "pv_name" = "TEMP", write = false }, - { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, { reg_name = "super_special_parameter_for_mod1", "pv_name" = "SUPER_SPECIAL", write = true }, ] [diagnostic_modules.mod2] ip = "192.168.1.22" pv_suffix = "MOD2" -parameters = [ - { reg_name = "temp", "pv_name" = "TEMP", write = false }, - { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, -] +parameter_groups = ["temperature", "event_rate"] ``` ### Runtime @@ -66,10 +78,14 @@ It would then: - A dedicated thread in `kafka_dae_control` would attempt to poll each diagnostic register in turn, looping for the lifetime of the program. - The updated numbers would be served in PVs of the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL`. This allows them to be accessible to IBEX, monitored by Nagios, or consumed by DSG's monitoring infrastructure. +Every parameter would be exposed as an integer, with no parameter-specific logic inside `kafka_dae_control`. + ### Writing `kafka_dae_control` would also create standard setpoint PVs for each writeable parameter, in the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL:SP`. +Every parameter would be written as an integer, with no parameter-specific logic inside `kafka_dae_control`. + ## Alternatives - In the first instance, we could avoid the architectural complexity of a central memory-map store and self-describing boards by requiring a `reg_address` in the `config.toml`. diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt index 76420db5b2..d3c46472af 100644 --- a/doc/spelling_wordlist.txt +++ b/doc/spelling_wordlist.txt @@ -622,6 +622,7 @@ opi opis Ordela ORed +overcount overspeed packagename Panchaud From 5e7a15aa63c4dc230841c3e6b2e77955ed408555 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Mon, 24 Aug 2026 11:10:48 +0100 Subject: [PATCH 3/6] reg_name -> reg_id --- .../datastreaming/ADRs/010_detector_settings.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md index 6cea14a1e7..ba7bb498bc 100644 --- a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -39,13 +39,13 @@ We would add a section in [`kafka_dae_control`'s config file](https://github.com # multiple boards, to help reduce repetition. [diagnostic_parameter_groups.temperature] parameters = [ - { reg_name = "temp1", "pv_name" = "TEMP1", write = false }, - { reg_name = "temp2", "pv_name" = "TEMP2", write = false }, + { reg_id = "temp1", "pv_name" = "TEMP1", write = false }, + { reg_id = "temp2", "pv_name" = "TEMP2", write = false }, ] [diagnostic_parameter_groups.event_rate] parameters = [ - { reg_name = "event_rate", "pv_name" = "EVENTRATE", write = false }, + { reg_id = "event_rate", "pv_name" = "EVENTRATE", write = false }, ] # Each board defines it's key parameters (ip, PV name). @@ -56,7 +56,7 @@ ip = "192.168.1.21" pv_suffix = "MOD1" parameter_groups = ["temperature", "event_rate"] parameters = [ - { reg_name = "super_special_parameter_for_mod1", "pv_name" = "SUPER_SPECIAL", write = true }, + { reg_id = "super_special_parameter_for_mod1", "pv_name" = "SUPER_SPECIAL", write = true }, ] [diagnostic_modules.mod2] @@ -89,7 +89,7 @@ Every parameter would be written as an integer, with no parameter-specific logic ## Alternatives - In the first instance, we could avoid the architectural complexity of a central memory-map store and self-describing boards by requiring a `reg_address` in the `config.toml`. - - Mapping via `reg_name` and self-description could still be added later if desired + - Mapping via `reg_id` and self-description could still be added later if desired - We could make this an entirely separate process from `kafka_dae_control`, which happens to be implemented in a similar way. - Advantage: This would better insulate the *critical* functionality in `kafka_dae_control` from the non-critical functionality of providing diagnostics on individual detector modules. - Disadvantage: There would be some duplication between `kafka_dae_control` and this new process; both would be doing UDP comms to boards with a similar interface, and serving PVs over PVAccess. From f7c8d21cb5259e183252337785b5c91fa97b02ab Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Mon, 24 Aug 2026 11:44:15 +0100 Subject: [PATCH 4/6] Implement it as a separate process, in the same repo as kafka_dae_control --- .../ADRs/010_detector_settings.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md index ba7bb498bc..4c14590f3d 100644 --- a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -30,9 +30,16 @@ Individual detector boards which want to participate in this scheme will: - In a register which is common across all detector boards, publish an identifier which uniquely identifies a specific memory map. - This memory map will specify a mapping of `name <-> register` for all parameters this board can expose to IBEX. +### New process + +We will add a separate process to read and write diagnostics from individual boards. The code will be in the same repository as +`kafka_dae_control` to allow re-use of shared infrastructure (for example, UDP comms logic), but will be a separate runtime +process. + ### Configuration -We would add a section in [`kafka_dae_control`'s config file](https://github.com/ISISComputingGroup/kafka_dae_control/blob/main/config.example.toml), which looks like: +The new process would be configured using a `config.toml` in a similar style to the existing `kafka_dae_control` config file. +An example of a configuration file is: ```toml # 'Parameter groups' define shared sets of parameters which may exist on @@ -67,32 +74,27 @@ parameter_groups = ["temperature", "event_rate"] ### Runtime -When `kafka_dae_control` starts, it will create a thread dedicated to communication with 'diagnostic' modules. - -It would then: +When the new process starts, it will: +- Update it's local cache of memory maps (e.g. `git pull`), with a timeout. If the pull fails or times out, the most recent set of memory maps will continue to be used, with a warning. - Read register `0` of each configured diagnostic module to retrieve a memory-map identifier -- Look up that register map identifier in a central store to retrieve a 'full' memory map for this board - - The central store could be cached locally on startup to ensure `kafka_dae_control` still boots correctly if the central store is offline. - - In any case, a failure in the 'diagnostic' functionality should not prevent the critical functionality of `kafka_dae_control` from working. -- Use the retrieved mappings to map each configured register (in the `parameters` of the `config.toml` to a numeric address) -- A dedicated thread in `kafka_dae_control` would attempt to poll each diagnostic register in turn, looping for the lifetime of the program. +- Use the retrieved mappings to map each configured register (in the `parameters` and `parameter_groups` sections of the `config.toml`) to a numeric address +- Attempt to poll each diagnostic register in turn, looping for the lifetime of the program. - The updated numbers would be served in PVs of the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL`. This allows them to be accessible to IBEX, monitored by Nagios, or consumed by DSG's monitoring infrastructure. -Every parameter would be exposed as an integer, with no parameter-specific logic inside `kafka_dae_control`. +Every parameter would be exposed as an integer, with no parameter-specific logic. ### Writing -`kafka_dae_control` would also create standard setpoint PVs for each writeable parameter, in the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL:SP`. +The new process would also create standard setpoint PVs for each writeable parameter, in the form `IN:INST:DAE:DIAG:MOD1:SUPER_SPECIAL:SP`. -Every parameter would be written as an integer, with no parameter-specific logic inside `kafka_dae_control`. +Every parameter would be written as an integer, with no parameter-specific logic. ## Alternatives - In the first instance, we could avoid the architectural complexity of a central memory-map store and self-describing boards by requiring a `reg_address` in the `config.toml`. - Mapping via `reg_id` and self-description could still be added later if desired -- We could make this an entirely separate process from `kafka_dae_control`, which happens to be implemented in a similar way. - - Advantage: This would better insulate the *critical* functionality in `kafka_dae_control` from the non-critical functionality of providing diagnostics on individual detector modules. - - Disadvantage: There would be some duplication between `kafka_dae_control` and this new process; both would be doing UDP comms to boards with a similar interface, and serving PVs over PVAccess. +- We could embed this within the `kafka_dae_control` process at runtime, for example as a separate thread. + - This would increase the risk that a failure in 'diagnostic' functionality could affect the core 'control' functionality. ## Risks @@ -104,4 +106,4 @@ If instruments begin accumulating scripts or workflows which involve 'fiddling' - It is possible to read and write a specified set of registers from individual detector modules from EPICS PVs. This set is statically configurable per-instrument. - We increase the risk of architectural 'shortcuts' being taken later which, if taken, would adversely impact maintainability. -- `kafka_dae_control` becomes more complicated. +- The `kafka_dae_control` repository becomes more complicated, as it now hosts the source code for two independent processes, which happen to share some functionality and architectural approaches. From 901f48c678249c2e9137fa275659f77c04fd1973 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Mon, 24 Aug 2026 11:50:23 +0100 Subject: [PATCH 5/6] Make write= optional --- .../datastreaming/ADRs/010_detector_settings.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md index 4c14590f3d..e8dbd4c8db 100644 --- a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -36,6 +36,9 @@ We will add a separate process to read and write diagnostics from individual boa `kafka_dae_control` to allow re-use of shared infrastructure (for example, UDP comms logic), but will be a separate runtime process. +This process is less critical than `kafka_dae_control`, in the sense that an instrument should be able to _run_ and perform all +routine DAE operations without this process available, albeit with reduced diagnostic visibility. + ### Configuration The new process would be configured using a `config.toml` in a similar style to the existing `kafka_dae_control` config file. @@ -72,6 +75,8 @@ pv_suffix = "MOD2" parameter_groups = ["temperature", "event_rate"] ``` +If the `write=` parameter is omitted, it will default to what is specified in the memory-map. + ### Runtime When the new process starts, it will: From 7d5d0a3a33af40658d528cf2d1cee6a2b4e8ec55 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Mon, 24 Aug 2026 11:53:45 +0100 Subject: [PATCH 6/6] Add isolation to consequences --- doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md index e8dbd4c8db..b2f8e42bde 100644 --- a/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md +++ b/doc/specific_iocs/datastreaming/ADRs/010_detector_settings.md @@ -112,3 +112,4 @@ If instruments begin accumulating scripts or workflows which involve 'fiddling' - It is possible to read and write a specified set of registers from individual detector modules from EPICS PVs. This set is statically configurable per-instrument. - We increase the risk of architectural 'shortcuts' being taken later which, if taken, would adversely impact maintainability. - The `kafka_dae_control` repository becomes more complicated, as it now hosts the source code for two independent processes, which happen to share some functionality and architectural approaches. +- Diagnostic functionality is isolated from core DAE control functionality through process-level separation, reducing the likelihood that detector diagnostic failures will impact routine instrument operations.