From dd36f12aab58aee243fc45951b87cb5e83a9bde9 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 26 Aug 2026 14:05:40 +0100 Subject: [PATCH] Initial options analysis --- filewriter_evaluation/ess.md | 125 ++++++++++++++++++++++++++ filewriter_evaluation/greenfield.md | 112 +++++++++++++++++++++++ filewriter_evaluation/supermusr.md | 132 ++++++++++++++++++++++++++++ filewriter_evaluation/template.md | 55 ++++++++++++ 4 files changed, 424 insertions(+) create mode 100644 filewriter_evaluation/ess.md create mode 100644 filewriter_evaluation/greenfield.md create mode 100644 filewriter_evaluation/supermusr.md create mode 100644 filewriter_evaluation/template.md diff --git a/filewriter_evaluation/ess.md b/filewriter_evaluation/ess.md new file mode 100644 index 0000000..73a7fba --- /dev/null +++ b/filewriter_evaluation/ess.md @@ -0,0 +1,125 @@ +## ESS Filewriter + +### Background + +The [ESS filewriter](https://github.com/ess-dmsc/kafka-to-nexus) is the ESS' filewriter process. + +### Fit with requirements + +#### NeXus file format + +In general: the biggest architectural hurdle is that ESS writer modules expect, for the most part, a one-to-one mapping between input schemas and writer modules. + +We will have many different nexus datasets which would need to depend on the data contained in a single message. The analysis below *assumes* that the ESS writer-module framework can allow for both: +- A single message to be delivered to multiple writer modules. This appears to already be true, but not the 'usual' case. +- A single writer module to be able to subscribe to multiple message types. This appears to be the 'harder' refactor, and would need architectural attention. +Without these assumptions, many of the arguments below that depend on new writer modules being written collapse. + +- [Neutron event data](https://github.com/isisComputingGroup/datastreaming/issues/83) + * **Generally supported** + * To fully support this, we would need to add support for ISIS-specific quirks of our Nexus files. This is expected to be a relatively modest, localised task - not a rewrite. +- [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) + * **Unsupported** + * To support this, we would need to: + * Add a new writer module for a histogrammed representation of events. + * Ensure that events can be written to both a histogrammed representation and an event representation if both modules are present in the `nexus_structure`. +- [Static metadata](https://github.com/isisComputingGroup/datastreaming/issues/87) + * **Mostly supported** + * The vast majority of the listed datasets could be included in the `nexus_structure` JSON, with no code changes required. + * The `mdat` writer module has specific support for start and end time. + * The only requirement that is potentially unclear is metadata which can update *during* a run, and should have only its most recent value written to the file (for example run title). + * To fully support this, we may need to add a new writer module which updates a single value in-place (rather than appending new values to a log). This is a relatively minor change, localised to a single new writer module. +- [Dynamic metadata](https://github.com/isisComputingGroup/datastreaming/issues/88) + * **Unsupported** + * There is no obvious support for the concept of 'dynamic' metadata in the ESS filewriter, except for a very limited `mdat` writer which only handles start/stop times. + * To fully support this, we would need to: + * Add new writer module(s), which *compute* or *accumulate* data, before writing them to the file. There is limited precedent for this in the `mdat` writer. We would likely need a new writer module for each type of 'dynamic' metadata. +- [SELog / blocks](https://github.com/isisComputingGroup/datastreaming/issues/89) - ensure to include the 'going back in time' requirement + * **Mostly supported** + * Support for most datasets of interest is present. + * Units are [statically supported](https://github.com/ess-dmsc/kafka-to-nexus/blob/232c7173d2e15bebcc4365accd6c32d616148eb9/src/WriterModule/f144/f144_Writer.h#L84); we would need to add support for `un00` unit updates via Kafka. + * The ESS filewriter has [a configurable 'back-in-time' parameter](https://github.com/ess-dmsc/kafka-to-nexus/blob/232c7173d2e15bebcc4365accd6c32d616148eb9/apps/kafka-to-nexus.cpp#L257) and [has logic to buffer `f144` updates](https://github.com/ess-dmsc/kafka-to-nexus/blob/232c7173d2e15bebcc4365accd6c32d616148eb9/src/Stream/SourceFilter.h#L35). + * To fully support this, we would need to add support for units from Kafka `_sampleEnv` topic, as opposed to from the run start. This should be a relatively small change, localised to the `f144` writer-module. +- [EPICS PutLog](https://github.com/isisComputingGroup/datastreaming/issues/91) + * **Unsupported** + * String logs are not supported; the reasoning was that storing strings in `NXlog` is not nexus-compliant. `NXtextlog` is now available for this purpose. + * ESS schemas already include a `vs00` schema suitable for emitting string-typed logs through the Kafka layer. + * To support this, we would need to implement a writer module for the existing `vs00` ESS schema to `NXtextlog`. This would follow broadly the same shape as the existing `f144` writer module. +- [`runlog/icp_event`](https://github.com/isisComputingGroup/datastreaming/issues/92) + * **Unsupported** + * There isn't currently a concept of internally-generated logs in the ESS filewriter - especially not ones which depend on multiple input streams. + * We would need to carefully assess how best to fit this into the ESS filewriter architecture. There is no clear precedent. +- [Period-specific metadata](https://github.com/isisComputingGroup/datastreaming/issues/93) + * **Partially supported** today, **Supported** alongside "dynamic metadata" work + * Some datasets can be statically known at runstart: these are trivial to add as part of the nexus structure. + * Datasets which depend on the event data inherit the same problem as the "dynamic metadata" section. + * To support this, we would use the existing `nexus_structure` support for any static metadata, and implement "dynamic metadata" writer-modules for any dynamic elements. +- [`isis_vms_compat`](https://github.com/isisComputingGroup/datastreaming/issues/94) + * **Partially supported** today, **Supported** alongside "dynamic metadata" work + * Some datasets can be statically known at runstart: these are trivial to add as part of the nexus structure. + * Datasets which depend on the event data inherit the same problem as the "dynamic metadata" section. + * To support this, we would use the existing `nexus_structure` support for any static metadata, and implement "dynamic metadata" writer-modules for any dynamic elements. +- [Monitor data](https://github.com/isisComputingGroup/datastreaming/issues/95) + * **Partially supported** today, **Supported** alongside histogram-generation work + * Monitor events should be trivially supportable, via nexus_structure defining appropriate locations for `ev44` writer modules. + * Generating nexus histograms from Kafka events inherit the same problems as 'neutron histogram data' + * Monitor histograms `hs00`/`hs01` don't appear to be supported (are ESS doing all monitors in event mode?) + * To support this, we would need to: + * Do the 'histogram writer module' work + * If we need streamed-histogram support (`hs00`/`hs01`) later, implement a new writer-module for these. This can likely be deferred for HRPD-X. +- [`instrument` group](https://github.com/isisComputingGroup/datastreaming/issues/97) + * **Supported** - via `nexus_structure`. +- [Instrument-specific nexus extras e.g. `instrument_components.nxs`](https://github.com/ISISComputingGroup/DataStreaming/issues/106) + * **Supported** - via `nexus_structure`. + +#### Downstream workflows + +The ESS filewriter emits a `wrdn` on completion of a file, and this should be suitable as a 'hook' on which to attach workflows such as file-archiving and journal-writing, implemented in separate processes. + +One area that would need attention is that the journal wants to write fields that *depend on the content of the file*, for example total counts or good uAh. At present, the metadata emitted in the `wrdn` is a copy of the metadata from the runStart. This means that consumers would either have to inspect the just-written file to pull out this data, or the `wrdn` machinery could be modified to (optionally) include some of the fields derived from the event data during the run. + +Files [are marked as read-only on close](https://github.com/ess-dmsc/kafka-to-nexus/blob/232c7173d2e15bebcc4365accd6c32d616148eb9/src/HDFFile.cpp#L41). + +Therefore for our requirements: + +- [File archiving workflows](https://github.com/ISISComputingGroup/DataStreaming/issues/85) + * **Supported**, in the sense that appropriate messages are emited which could be picked up by an external archiving process. +- [Generating journals](https://github.com/isisComputingGroup/datastreaming/issues/86) + * **Partially supported**; we would need to extend the content of the `wrdn` message to do this optimally. + +#### [Intermediate and autosave files](https://github.com/isisComputingGroup/datastreaming/issues/96) + +- **Unsupported**; it is unclear how this would be implemented. + - The ESS filewriter *does* use HDF5 SWMR support, so a separate/cooperating process may be able to take a 'snapshot' of an in-progress file, fix-up the metadata, and then emit that as an intermediate file. + - We could emit a new runStart/runStop pair, which would be picked up by another filewriter in the job-pool. This is an option because the ESS filewriter is inherently pooled and parallelizable by adding more writer processes. However, this would cause an expensive re-read of all of the data in the run from Kafka, and may cause our intermediate files to then be emitted so slowly that they are not useful to scientists. + - We could add full 'snapshotting' support to the filewriter, via a dedicated Kafka message, but this may be a rather invasive change. + +#### [Deployment topology](https://github.com/isisComputingGroup/datastreaming/issues/98) + +The ESS filewriter is deployed in a pooled architecture: a set of independent filewriter processes, each one capable of picking up a writing job from any instrument. Worker selection is by filewriters subscribing using the same Kafka consumer group while idle. Workers may freely join the pool. `SIGHUP` is used to notify a filewriter to complete it's current task and then shut down. + +Deployment infrastructure is not in the repository (beyond a minimal Dockerfile), and the filewriter makes no assumptions about deployment technology (beyond a Linux OS). + +### Code quality + +- Major dependencies are listed at https://github.com/ess-dmsc/kafka-to-nexus/tree/232c7173d2e15bebcc4365accd6c32d616148eb9#building-the-applications : Linux, Conan, CMake, C++17, Ninja +- A large number of [unit tests](https://github.com/ess-dmsc/kafka-to-nexus/tree/232c7173d2e15bebcc4365accd6c32d616148eb9/tests), [integration tests](https://github.com/ess-dmsc/kafka-to-nexus/tree/232c7173d2e15bebcc4365accd6c32d616148eb9/integration-tests) and [domain tests](https://github.com/ess-dmsc/kafka-to-nexus/tree/232c7173d2e15bebcc4365accd6c32d616148eb9/domain-tests) are included in the repository. +- The codebase is C++17. + * This deviates from our usual convention of using Rust for performance-sensitive processes and Python otherwise elsewhere in the streaming stack. + * We will need to accept the higher maintenance cost of a C++ codebase compared to other languages, if we select this option. + * Some of the correctness risks of using C++ are mitigated by the good test coverage and [routine use of sanitizers](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/conanfile.py#L34) +- The codebase is in a standalone repository that is reasonably well decoupled from other ESS-specific infrastructure. +- Historic collaboration with ESS has been somewhat patchy - we may struggle to reliably upstream all changes we want to make in a timely manner for HRPD-X, given our relatively tight implementation deadline. Therefore, a local fork, upstreaming changes *when possible*, seems like the most likely option. This creates the risk of significant later divergence unless we make a strong attempt to reconcile after HRPD-X implementation. + * [`CONTRIBUTING.MD` suggests that all changes would need to go through an ESS steering board](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/CONTRIBUTING.md#the-project); it is highly unlikely that this would be a viable approach in time for HRPD-X commissioning, which strengthens the argument that we would be developing a local fork rather than upstream-first. +- Some documentation exists, but correctness is patchy: for example [the `f144` documentation actually describes an `NXevent_data` structure](https://github.com/ess-dmsc/kafka-to-nexus/blob/232c7173d2e15bebcc4365accd6c32d616148eb9/documentation/writer_module_f144_logdata.md?plain=1#L42). + +### Supportability + +- Reasonable logging is present. +- The most important metrics are present. +- Periodic status reports via an `x5f2` schema are emitted. +- [Invalid messages are filtered](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/src/Stream/SourceFilter.cpp) and published to metrics - including corrupt messages and messages with too-old timestamps +- Some invalid data is detected, but then written anyway. For example, `ev44` messages with mismatched `pixel_id` and `time_of_flight` array lengths are [logged but *then written anyway*](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/src/WriterModule/ev44/ev44_Writer.cpp#L103), which would cause data corruption. + * We would need to carefully review other writer modules for these kinds of failure paths, and would likely want to drop these messages (with logs/metrics) to avoid corrupting the entire file. +- It is not clear how the filewriter recovers if it exits non-gracefully during writing (e.g. segfault, forceful shutdown, server power-off); in principle this should be *detectable* from an external process due to a lack of `x5f2` status reports. + * We may need some external 'watchdog' service which detects file-writing tasks which failed non-gracefully, and re-queues those writing jobs with a limited number of retries. It would also need to 'clean up' incomplete partially-written files from a non-graceful termination. Not immediately clear whether ESS already have such a service. diff --git a/filewriter_evaluation/greenfield.md b/filewriter_evaluation/greenfield.md new file mode 100644 index 0000000..2f6953b --- /dev/null +++ b/filewriter_evaluation/greenfield.md @@ -0,0 +1,112 @@ +## Greenfield development, ISIS-specific filewriter + +### Background + +This filewriter option proposes a brand-new development, specific to ISIS. + +This would be implemented in Rust, as per other performance-sensitive components of the datastreaming pipeline. + +Unlike the SuperMuSR and ESS options, every requirement below is unimplemented. Therefore, this document focuses on *how* we would propose to implement each requirement in a greenfield system. + +### Fit with requirements + +The overall target architecture is *similar* to the architecture of the ESS filewriter, but explicitly allowing *multiple* writer modules to each subscribe to the same input Kafka messages; i.e. rather than having an `ev44_writer`, we would have an `event_data` writer, which happens to subscribe to `ev44` and/or other message types as required. + +Architecturally: +- Configurable nexus_template, defined in runStart message, comparable to ESS filewriter. +- Writer modules for one Nexus dataset or group of related datasets (e.g. `NXevent_data` writer, `NXlog` writer, `isis_vms_compat` writer). Each writer module has complete, exclusive, ownership of one dataset - multiple writer modules will not update the same dataset. +- Each writer module can register to be notified of a specified set of message schemas, on one or more topics. +- When a Kafka message arrives, it is deserialised once centrally, and then every writer module that has registered for that schema + topic combination is passed a reference to that (typed) message + * This will happen sequentially, in a fixed order. i.e. Message N will be delivered to Writer1, Writer2, Writer3, before message N+1 is delivered to Writer1, Writer2, Writer3. This is to reduce error-recovery and consistency hazards where two writers see messages in a different order than each other. + * Where messages are on the same topic, message offset N will be received before message N+1. Where messages are on different Kafka topics or partitions, the order is undefined but consistent between any two writer modules. +- Writer modules may be stateful, for example if they need to accumulate data into a histogram or running counter, or if they need to keep track of the period-number for the current frame. + +#### Core infrastructure + +We will need to: +- Implement a writer-module framework with some concept of writer-modules subscribing to a writer-defined set of `(topic, schema)` combinations (the topics may only be known after the `nexus_structure` is parsed, so this needs to be dynamic). +- Define how our `nexus_structure` will look - wherever possible, following the ESS approach, but minor modifications may be required. +- Implement logic to parse the provided `nexus_structure`, write out static data immediately, and dynamically create the relevant writer-modules. +- Implement a Kafka-listening and message-routing layer, which subscribes to the set of topics needed by any writer module, decodes messages, and routes `(topic, schema)` combinations to the writer-modules interested in those messages. +- Implement a (minimal) framework or abstraction over the HDF5 libraries to ease writer-module implementation. +- Implement logic supporting the run start / run stop / intermediate file lifecycle, including status reporting via `x5f2` and writing-done notifications via `wrdn` +- Logging, metrics, and other required supportability machinery + +#### NeXus file format + +- [Neutron event data](https://github.com/isisComputingGroup/datastreaming/issues/83) + * Implemented by a custom writer-module listening to `ev44` and `pu00` from `_events` topic. Localised to a single writer module. +- [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) + * Implemented by a custom writer-module listening to `ev44` and `pu00` from `_events` topic, and `vc00` from `_vetoConfig` topic. Localised to a single writer module. + * Implement this **inside the filewriter**, rather than as a separate post-processing step, because we are likely to need all the permutations of: + - Event mode only + - Histogram mode only + - Both events and histograms, describing the same data, in the same file + - Separate event and histogram files describing the same data + - Detectors in event mode but monitors histogrammed +- [Static metadata](https://github.com/isisComputingGroup/datastreaming/issues/87) + * Statically inserted into `nexus_structure`; this is trivial work per-dataset, once the core infrastructure work is complete. +- [Dynamic metadata](https://github.com/isisComputingGroup/datastreaming/issues/88) + * Implemented by custom writer-modules that implement the specific accumulation logic for that dataset or group. Localised to a number of new writer modules, most of which are not complicated, but there are a number of them to implement. +- [SELog / blocks](https://github.com/isisComputingGroup/datastreaming/issues/89) - ensure to include the 'going back in time' requirement + * Implemented by a writer-module, similar to writer-module in ESS filewriter which covers back-in-time requirement (alongside forwarder emitting updates for all PVs at predictable intervals even if unchanged) + * Subscribes to alarm, unit, value messages from `_sampleEnv` topic, going 'back in time' by at least the filewriter's periodic-forwarding interval at run start. +- [EPICS PutLog](https://github.com/isisComputingGroup/datastreaming/issues/91) + * Would use either a new generic writer-module for `NXtextlog`, or the generic `NXlog` writer-module as above. + * Needs extra infrastructure to forward the `PutLog` to Kafka in the first place - likely using `vs00` schema. +- [`runlog/icp_event`](https://github.com/isisComputingGroup/datastreaming/issues/92) + * Specialised writer-module; needs to subscribe to a number of different topics in order to fabricate an `icp_event` dataset that resembles what the ICP previously wrote (and what Mantid therefore expects). +- [Period-specific metadata](https://github.com/isisComputingGroup/datastreaming/issues/93) + * Specialised writer-module, subscribing to `ev44`, `pu00`, and any other necessary schemas +- [`isis_vms_compat`](https://github.com/isisComputingGroup/datastreaming/issues/94) + * Static metadata in `nexus_structure` wherever possible; these datasets are then very cheap to add support for. + * Specialised writer module(s) where runtime calculations are required. +- [Monitor data](https://github.com/isisComputingGroup/datastreaming/issues/95) + * Re-uses bulk of the support/logic from either event-data or histogram data as appropriate. +- [`instrument` group](https://github.com/isisComputingGroup/datastreaming/issues/97) + * Static metadata in `nexus_structure` wherever possible; these datasets are then very cheap to add support for. + * Specialised writer module(s) where runtime calculations are required. +- [Instrument-specific nexus extras e.g. `instrument_components.nxs`](https://github.com/ISISComputingGroup/DataStreaming/issues/106) + * Static metadata in `nexus_structure` wherever possible; these datasets are then very cheap to add support for. + +#### Downstream workflows + +- [File archiving workflows](https://github.com/ISISComputingGroup/DataStreaming/issues/85) +- [Generating journals](https://github.com/isisComputingGroup/datastreaming/issues/86) + +The filewriter would emit a `wrdn` message when it has completed writing a file, containing both the run-start metadata (e.g. run number) and metadata computed during the run (e.g. accumulated proton charge). + +The metadata computed during the run would be designed in a comparable way to the writer-modules: it 'subscribes' to messages that it wants to be notified of during a run, and accumulates those while the run +is in progress. This ensures that the emitted `wrdn` saw the exact same messages as the filewriter and describes the same data - including in the case of dropped messages or other bad-data scenarios. + +This `wrdn` message would be used to trigger journaling and archiving processes, which would live in separate processes outside the filewriter. + +#### [Intermediate and autosave files](https://github.com/isisComputingGroup/datastreaming/issues/96) + +We would add a Kafka message which commands the filewriter to write an intermediate file 'now'. At the next self-consistent time, the filewriter would: +- Close the in-progress file +- 'fork' the in-progress file (by copying bytes on disk) +- Perform any end-of-run actions and save out one of the files as the 'intermediate'. +- Re-open and continue accumulating any new messages into the original file. + +Because this filewriter operates in a pooled configuration, a fallback, reducing complexity but also potentially reducing performance, would be to emit parallel run start and run stop messages, picked up by another filewriter process to generate intermediate files. The separate filewriter process would then have to re-ingest all relevant messages from Kafka in order to write it's file. This is a low-risk option, and may be suitable for HRPD-X where data rates are not particularly high. + +#### [Deployment topology](https://github.com/isisComputingGroup/datastreaming/issues/98) + +We would adopt a pooled architecture, staying close to the ESS design: worker selection by all filewriters joining a common Kafka consumer-group for the filewriter command topic. + +As per both ESS and SuperMuSR filewriters, we would attempt to avoid adding any dependency on a specific deployment technology in the filewriter repository itself (with the exception of basic, optional, container images). + +Graceful shutdown logic comparable to the ESS filewriter's SIGHUP mechanism would allow orchestrators to gracefully perform rolling restarts/upgrades, while not tying us tightly to a specific technology. + +### Code quality + +- No code currently exists, so there is nothing to evaluate. +- Our testing strategy should take inspiration from the ESS filewriter, which has test coverage at multiple levels. +- We should also aim to have a system test which compares our output against an ICP-generated file, and asserts that the layout is identical other than known, intentional differences. + +### Supportability + +- Logging would be performed via standard rust logging infrastructure +- Metrics would be exposed via prometheus-scrape compatible endpoints, comparable to other streaming processes. This allows metrics to be consumed easily and concurrently by our existing Nagios monitoring infrastructure, DSG Grafana dashboards, and any future monitoring infrastructure chosen by the MNeuData project. +- Error handling and error recovery schemes will need to be defined as part of implementation, but should take note of the failure modes already accounted for in either the SuperMuSR or ESS filewriters. diff --git a/filewriter_evaluation/supermusr.md b/filewriter_evaluation/supermusr.md new file mode 100644 index 0000000..ce59469 --- /dev/null +++ b/filewriter_evaluation/supermusr.md @@ -0,0 +1,132 @@ +## SuperMuSR filewriter + +### Background + +The [SuperMuSR filewriter](https://github.com/ISISNeutronMuon/digital-muon-pipeline/tree/main/nexus-writer) is in development as part of the SuperMuSR streaming pipeline. + +The overall architecture is described by the [pipeline documentation](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/architecture/index.html). For background about the file-format the SuperMuSR filewriter was designed to write, see [file format ADR](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/adrs/0009-nexus-file-format.html). + +This option for our filewriter implementation proposes using the SuperMuSR filewriter as a technical foundation, either forking it or working with the SuperMuSR team to incrementally adapt the SuperMuSR filewriter for neutron needs. + +### Fit with requirements + +#### NeXus file format + +For this comparison, POLREF run POLREF00046184.nxs was written using both the ICP and the SuperMuSR filewriter, with minimal changes to support neutron data. + +Generally: the SuperMuSR filewriter **hard-codes the structure of a Muon Nexus file**. The Muon Nexus format implemented by the current SuperMuSR filewriter differs substantially from existing ICP-written files, and from Neutron files. Adopting the SuperMuSR filewriter would therefore be an almost total re-write of everything under `nexus_structure`, while retaining much of the 'plumbing'. + +A major initial decision would be whether to stick with the existing hard-coding of file structure, or to move to a dynamic `nexus_structure`, which is a **major** refactor but gives several requirements an immediate viable implementation path. + +- [Neutron event data](https://github.com/isisComputingGroup/datastreaming/issues/83) + * Event data in general is **partially supported** + * The Kafka schema used is currently SuperMuSR-specific, but [an incomplete branch to support ev44](https://github.com/ISISNeutronMuon/digital-muon-pipeline/tree/ev44_support) exists + * The key datasets are written, but with a number of differences and omissions relative to ICP files. As an example, the `total_counts` dataset is absent. + * There is support for writing a per-frame `veto_flags` dataset in `NXevent_data`, but there does not appear to be a corresponding dataset for the vetoes a user had enabled at data acquisition time. + * To support this, we would need to: + * Implement a neutron-specific `ev44` event-data writer, reusing much of the code in the existing `aev2` support, or alternatively, refactor the existing event_data support to cater for both `ev44` and `aev2` + * Add support for the small number of missing datasets (for example total_counts). +- [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) + * **Unsupported** in the filewriter itself + * The SuperMuSR pipeline generates histograms as a separate step from event-mode Nexus file writing. This is instead done by [MNeuEventLib](https://github.com/ISISMuon/MNeuEventLib) for SuperMuSR. + * To best support all combinations of files including pure-event, pure-histogram, event+histogram, detectors in event mode with monitors in histogram mode etc, we would want to add support for writing a histogrammed version of the event-mode data in the filewriter directly. This would require new code. +- [Static metadata](https://github.com/isisComputingGroup/datastreaming/issues/87) + * This is **mostly unsupported** + * The majority of the datasets listed do not exist + * For some datasets which *are* written, the meaning is different from ICP files. + * To support this, we would need to either: + * Support a dynamic nexus_structure and include this information in the run start message + * Individually add support for each of the listed datasets; the datasets are simple, static pieces of metadata, so writing each one is not a large task. +- [Dynamic metadata](https://github.com/isisComputingGroup/datastreaming/issues/88) + * **Unsupported** + * None of the listed datasets are written in a similar way to the existing ICP files. + * Some streamed `f144`s are inserted into `runlog` (e.g. `/raw_data_1/runlog/IN:POLREF:DAE:RAWFRAMES`), but these rely on calculations performed in the ICP and so will not work for a pure-streaming system without an ICP. As the calculations are done in a different place, the `runlog` may therefore be inconsistent with the event data actually written to file. + * To support this: + * New code would need to be written for each piece of dynamic metadata. Most of these pieces of code are small and localised changes, but there are a reasonable number of them to implement. +- [SELog / blocks](https://github.com/isisComputingGroup/datastreaming/issues/89) - ensure to include the 'going back in time' requirement + * This is **partially supported** + * Values and timestamps are written + * Blocks which do not update during a run are missed. + * There is some support in code for alarm datasets. + * Units are missing on all value datasets for all logs; no support for `un00` is currently present. + * To support this, we would need to: + * Ensure the filewriter can [get the list of blocks needing to be written from somewhere; whether a `nexus_structure` or some other mechanism](https://github.com/ISISNeutronMuon/digital-muon-pipeline/issues/423) + * Make [a change](https://github.com/ISISNeutronMuon/digital-muon-pipeline/issues/424) to 'go back' to the most recent block update on run start, to ensure blocks which do not change get their most recent value and timestamp inserted to cover run start. +- [EPICS PutLog](https://github.com/isisComputingGroup/datastreaming/issues/91) + * **Unsupported**. + * To support this, we would need to write new code, which would likely take in these entries as a `vs00` schema and write them as an `NXlog` or `NXtextlog`. This would reuse much of the same logic as a sample environment block writer. +- [`runlog/icp_event`](https://github.com/isisComputingGroup/datastreaming/issues/92) + * **Partially supported**. + * Some support for "internally generated events" exists, which has *some*, but not all, of the semantics of the ICP runlog. However, it is not in a format which mantid needs for Neutron data. + * To support this, we would need to refactor the existing support to more closely align/emulate the previous ICP behaviour. +- [Period-specific metadata](https://github.com/isisComputingGroup/datastreaming/issues/93) + * **Unsupported** (except for some very minimal metadata). + * To support this, we would need to add new support for it; many datasets would be comparable to other 'dynamic metadata' datasets, a few would be static metadata (covered by either `nexus_structure` or explicit support). +- [`isis_vms_compat`](https://github.com/isisComputingGroup/datastreaming/issues/94) + * **Unsupported**. + * To support this, we would need to add support for 'faking' `isis_vms_compat` data from the event-stream and writing it to file. + * Some of this data is static and known in advance; in this case, it could be included in the `nexus_structure` (if that mechanism is added). Otherwise custom logic will be needed for each dataset. +- [Monitor data](https://github.com/isisComputingGroup/datastreaming/issues/95) + * **Unsupported** (not applicable to the Muon beamlines for which this filewriter was designed) + * To support this, we would need to add specific monitor-writing logic. This would likely share significant components of event_data or histogramming logic, depending on the mode monitors are being written in (which is ultimately user-configurable). +- [`instrument` group](https://github.com/isisComputingGroup/datastreaming/issues/97) + * **Mostly unsupported**. + * Some datasets created (minimally) under `instrument/source`, but most are not populated with any meaningful data + * All datasets under `instrument/dae` unsupported + * To support this, we would need to either: + * Add support for a dynamic `nexus_structure`, rather than the existing hard-coded structure, which would be a significant refactor + * Individually add support for each of the listed datasets; the datasets are relatively simple/static pieces of metadata, so writing each one is not a large task. +- [Instrument-specific nexus extras e.g. `instrument_components.nxs`](https://github.com/ISISComputingGroup/DataStreaming/issues/106) + * **Unsupported**. + * To support this, we would need to either: + * Add support for a dynamic `nexus_structure`, rather than the existing hard-coded structure, which would be a significant refactor + * A simpler but less flexible change to always copy in datasets from a source nexus file to the written file on run start - which meets this requirement, but does not help with other similar differences in expected nexus structure. + +#### Downstream workflows + +- [File archiving workflows](https://github.com/ISISComputingGroup/DataStreaming/issues/85) + * **Partially supported** + * The SuperMuSR filewriter [does include some support for archival workflows](https://github.com/ISISNeutronMuon/digital-muon-pipeline/blob/main/nexus-writer/src/flush_to_archive.rs) in the form of copying to a network drive. + * There is no support for checksumming present. + * To support this, we would need to either: + * Add full support for archiving workflows inside the filewriter. + * Alternatively, emit suitable `wrdn` messages and implement archiving logic in a downstream process. Emitting *minimal* `wrdn` messages should be a small amount of work. +- [Generating journals](https://github.com/isisComputingGroup/datastreaming/issues/86) + * **Unsupported** + * No support for generating/updating journals is present. + * To support this, we would need to: + * Ensure suitable `wrdn` messages are emitted, for a downstream process to update the instrument journal. To avoid downstream processes needing to open a potentially large nexus file only to extract minimal metadata, it would be best if the `wrdn` messages included the relevant pieces of dynamic metadata (e.g. total counts, micro-amp hours). No similar concept currently exists. + +#### [Intermediate and autosave files](https://github.com/isisComputingGroup/datastreaming/issues/96) + +- **Unsupported** + * There does not appear to be any support for these concepts. + * The file is flushed to disk incrementally. + * SWMR support is not being used. + * This filewriter uses a filewriter-per-instrument, so we cannot use an approach of generating a parallel write-request with different start/stop times. +- To support this, we would need to: + * Introduce a change to pause file-writing activity, copy a file, 'end' one copy, and then continue writing to the other file. The file is already flushed and self-consistent between messages. + + +#### [Deployment topology](https://github.com/isisComputingGroup/datastreaming/issues/98) + +- [A SuperMuSR ADR](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/adrs/0004-core-technologies-for-in-flight-data-processing-software.html#context) states that their "components should not be tied to a specific means of deployment, orchestration or execution or execution environment". In practice, a Linux OS is assumed, but a specific deployment technology is not. +- The SuperMuSR filewriter operates in a filewriter-per-instrument configuration. + +### Code quality + +- The major dependencies are listed [here](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/adrs/0004-core-technologies-for-in-flight-data-processing-software.html#decision): Rust, Nix, OCI container images. +- A few unit tests exist, but the amount of testing is minimal compared to the amount of code. +- Reasonable inline documentation is present. +- The codebase looks organised in a reasonable way, but is part of a monorepo which would make adopting it as a standalone component more difficult. An initial task would be to extract the filewriter component from the monorepo; the coupling to the monorepo is not excessively tight. +- In places, there is tight coupling to how the SuperMuSR data pipeline operates, which may need refactoring to also be suitable for how the neutron data pipeline operates. Cross-process assumptions are likely to be the most risky area during refactoring, as neither team have expertise in the other team's data pipeline model (even though they are *architecturally* similar at a very high level). +- Collaboration with SuperMuSR is probably technically desirable, but SuperMuSR is coming online relatively soon which will likely mean the SuperMuSR team are very busy with commissioning work. Attempting to make major changes to the SuperMuSR filewriter while SuperMuSR is simultaneously commissioning might not be viable. + +### Supportability + +Supportability and error handling look to have been carefully considered: +- The SuperMuSR filewriter includes support for logging and metrics throughout. +- A number of error conditions [have been considered](https://github.com/ISISNeutronMuon/digital-muon-pipeline/blob/cbed81bb66bfd7ec53a6be64f8e3bf1dcc9c1eab/nexus-writer/src/error.rs). +- Code-level error-handling strategies are [documented](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/adrs/0010-revised-error-handling-using-miette.html). +- Some known assumptions [are documented](https://github.com/ISISNeutronMuon/digital-muon-pipeline/tree/main/nexus-writer). +- [ADRs are available](https://isisneutronmuon.github.io/digital-muon-pipeline-docs/adrs/0001-record-architecture-decisions.html) for some major decisions, many of which relate to supportability infrastructure. diff --git a/filewriter_evaluation/template.md b/filewriter_evaluation/template.md new file mode 100644 index 0000000..79ecdee --- /dev/null +++ b/filewriter_evaluation/template.md @@ -0,0 +1,55 @@ +## Option + +### Background + +Some context about where this filewriter comes from, where it is used, what this implementation option looks like at a very high level. + +### Fit with requirements + +For each of the sections below, fill out a brief high-level view of how our requirement would be met using this filewriter, and a summary of the extent of work that would need to be done to add the relevant functionality. + +#### NeXus file format + +Summarise which of the following areas of our NeXus file format are already implemented, partially implemented, or would need implementing. +Document whether any specific datasets would be particularly difficult to add, and areas that would need special architectural attention. + +- [Neutron event data](https://github.com/isisComputingGroup/datastreaming/issues/83) +- [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) +- [Static metadata](https://github.com/isisComputingGroup/datastreaming/issues/87) +- [Dynamic metadata](https://github.com/isisComputingGroup/datastreaming/issues/88) +- [SELog / blocks](https://github.com/isisComputingGroup/datastreaming/issues/89) - ensure to include the 'going back in time' requirement +- [EPICS PutLog](https://github.com/isisComputingGroup/datastreaming/issues/91) +- [`runlog/icp_event`](https://github.com/isisComputingGroup/datastreaming/issues/92) +- [Period-specific metadata](https://github.com/isisComputingGroup/datastreaming/issues/93) +- [`isis_vms_compat`](https://github.com/isisComputingGroup/datastreaming/issues/94) - for the purposes of evaluation, imagine that we will need to write a limited subset of the current `isis_vms_compat` datasets. +- [Monitor data](https://github.com/isisComputingGroup/datastreaming/issues/95) +- [`instrument` group](https://github.com/isisComputingGroup/datastreaming/issues/97) +- [Instrument-specific nexus extras e.g. `instrument_components.nxs`](https://github.com/ISISComputingGroup/DataStreaming/issues/106) + +#### Downstream workflows + +Summarise how this filewriter would support downstream workflows, which are part of the file-writing infrastructure as a whole but may not necessarily be part of the filewriter directly. + +For example: +- [File archiving workflows](https://github.com/ISISComputingGroup/DataStreaming/issues/85) +- [Generating journals](https://github.com/isisComputingGroup/datastreaming/issues/86) + +#### [Intermediate and autosave files](https://github.com/isisComputingGroup/datastreaming/issues/96) + +Summarise how we would meet requirements for autosave and intermediate files using this filewriter. + +#### [Deployment topology](https://github.com/isisComputingGroup/datastreaming/issues/98) + +Summarise how this filewriter expects to be deployed. Include scalability considerations. +For example: is it a dynamic-pool architecture, a static-pool architecture, or a filewriter-per-instrument architecture? Does it strongly assume a specific deployment technology? + +### Code quality + +Add some general notes about the quality of the codebase. +Including areas like: high-level & code-level documentation, unit & system testing, codebase layout, major dependencies. +For external repositories, include information on how feasible a genuine collaboration is, as opposed to a local forked repository. + +### Supportability + +Add some general notes about how supportable the filewriter would be in production, focusing on error scenarios. +Including areas like: logging, metrics, error handling, error recovery, hardening against invalid/unexpected data.