diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index 191ca58f05..ccecac72e3 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -17,9 +17,36 @@ The file is EBSD (Electron Backscatter Diffraction) scan data. The most importan - **Pattern-quality metrics** — values such as Band Contrast, Band Slope, Bands, and Mean Angular Deviation describe how clear and reliable each measurement is. These are commonly used to flag unreliable pixels (see *Reference Frames* below). - **Per-phase (Ensemble) data** — the crystal structure, lattice constants, and material name for each phase. An **Ensemble** here means one distinct material/crystal type. +### Reading More Than One Scan + +An H5OINA file can hold several scans. Selecting more than one stacks them into a single +3D **Image Geometry**: the X and Y extents and step sizes come from the first selected +scan, the Z extent is the number of selected scans, and the Z spacing is the **Z Spacing** +parameter. The selected scans become the slices of one 3D microstructure. Therefore, +every scan must describe the same grid and use the same phase definitions. The phase-group +names, material names, Laue groups, space groups, and lattice constants must match. All +scans share one **Ensemble Attribute Matrix**, so different phase definitions cannot be +represented correctly. Import scans with different phase definitions separately. + +The **Stacking Order** setting carried by the scan selection chooses which end of the +list lands at Z = 0. *Low To High* stacks the scans in the order they are listed, so the +first selected scan is at Z = 0. *High To Low* stacks them in the reverse of that order, +so the last selected scan is at Z = 0. + ### Limitations of the Filter -The current implementation only understands **FORMAT VERSION 2.0** of the H5OINA file. A user can still read a newer H5OINA file, but the filter will only extract the VERSION 2.0 headers and data. If additional data is needed from the file, the [Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter can be used to augment this filter. +The filter reads a fixed set of header keys and nine data columns, the set Oxford documents +for **FORMAT VERSION 2.0**. The file's `Format Version` value is not used to select what is +read, and a file without that value is read the same way, so a file imports exactly when it +carries that fixed set — the Format Version 5.0 export bundled with this filter's tests +still does. Any column outside the set — for example `Pattern Quality`, `Beam Position +X`/`Y` or the `Electron Image` tree — is ignored, and can be brought in with the +[Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter. + +**Importing diffraction patterns is not yet supported for H5OINA files.** Turning on +*Import Pattern Data* stops the filter with an error rather than producing a partial result. +A file's `Processed Patterns` or `Unprocessed Patterns` dataset can be read with the +[Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter. ![Overview of the user interface.](Images/ImportH5OinaFilter_1.png) @@ -35,11 +62,26 @@ Historical reference frame operations for Oxford data are the following: + Sample Reference Frame: 180o about the <010> Axis + Crystal Reference Frame: None -The user also may want to assign un-indexed pixels to be ignored by flagging them as "bad". The [Multi-Threshold Objects](../SimplnxCore/MultiThresholdObjectsFilter.md) filter can be used to define this *mask* by thresholding on values such as *Error* = 0. +The user also may want to assign un-indexed pixels to be ignored by flagging them as "bad". The [Multi-Threshold Objects](../SimplnxCore/MultiThresholdObjectsFilter.md) filter can be used to define this *mask*. For H5OINA data, threshold on `Phase` > 0: an un-indexed point carries phase 0, which is the reserved invalid-phase slot. + +Do not assume `Error` = 0 marks a good point in an H5OINA file. AZtec writes an enumerated status code there whose values are not the same as the `.ctf` convention: in the AZtec export bundled with this filter's tests, every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0. A mask built from `Error` = 0 would select nothing on that file. ### Radians and Degrees -All orientation data in the H5OINA file are in radians. +All orientation data in the H5OINA file are in radians, and the imported `Euler` array is in +radians as well — no conversion is applied. + +The per-phase `LatticeConstants` array is the one place where a unit does change on import. +An H5OINA file stores its three lattice angles in radians; they are imported as **degrees**, +so that the array means the same thing no matter which EBSD format the phase came from. A +cubic phase therefore reports `90, 90, 90` rather than `1.5707964, 1.5707964, 1.5707964`. +The three lattice dimensions are imported unchanged. + +**This is a breaking change to a published output, and it is not in any released +version.** DREAM3D-NX 7.0.0 through 7.4.1 reported those three slots in radians for +H5OINA imports. See the migration notes below before comparing an H5OINA import against +a `.dream3d` file, a regression baseline, or a pipeline result produced by one of those +releases. ### The Axis Alignment Issue for Hexagonal Symmetry [1] @@ -50,7 +92,7 @@ All orientation data in the H5OINA file are in radians. + Caution: it appears that the axis alignment is a choice that must be made when installing TSL software so determination of which convention is in use must be made on a case-by-case basis. It is fixed to the y-convention in the HKL software. + The main clue that something is wrong in a conversion is that either the 2110 & 1010 pole figures are transposed, or that a peak in the inverse pole figure that should be present at 2110 has shifted over to 1010. + DREAM3D-NX uses the TSL/EDAX convention. -+ __The result of this is that the filter will by default add 30 degrees to the second Euler Angle (phi2) when reading Oxford `.h5oina` files. This can be disabled by the user if necessary.__ ++ __The result of this is that the filter will by default add 30 degrees to the third Euler angle (phi2) of every point whose phase is Hexagonal-High when reading Oxford `.h5oina` files. Because the file's Euler angles are in radians, the value actually added is 30 degrees expressed in radians (pi/6, about 0.5235988). Points of any other symmetry, and un-indexed points, are never adjusted. This can be disabled by the user if necessary.__ | Figure 1 | |--------| @@ -65,13 +107,61 @@ Once the reference frames are correct, the imported Euler angles are typically c None — this filter reads directly from a `.h5oina` file on disk. +## Created Outputs + +The array names are the H5OINA dataset names, so several contain spaces. + +### Cell Attribute Matrix + +| Name | Type | Components | Notes | +|------|------|------------|-------| +| `Band Contrast` | uint8 | 1 | | +| `Band Slope` | uint8 | 1 | | +| `Bands` | uint8 | 1 | | +| `Error` | uint8 | 1 | AZtec status code; see the note on masking below — do not assume 0 means "indexed" | +| `Euler` | float32 | 3 | Radians; phi2 optionally shifted for Hexagonal-High points | +| `Mean Angular Deviation` | float32 | 1 | | +| `Phase` | int32 or uint8 | 1 | int32 by default; uint8 when *Convert Phase Data to Int32* is off. 0 marks an un-indexed point | +| `X` | float32 | 1 | | +| `Y` | float32 | 1 | | + +### Ensemble Attribute Matrix + +One tuple per phase in the file, plus tuple 0, which is reserved for the invalid phase that +un-indexed points refer to. + +| Name | Type | Components | Notes | +|------|------|------------|-------| +| `CrystalStructures` | uint32 | 1 | Mapped from the file's Laue group; 999 in tuple 0 | +| `LatticeConstants` | float32 | 6 | a, b, c then alpha, beta, gamma in **degrees**; all zero in tuple 0 | +| `MaterialName` | string | 1 | `"Invalid Phase"` in tuple 0 | + % Auto generated parameter table will be inserted here +## Migration Notes + +Documented behavioral differences from DREAM3D-NX 7.0.0 through 7.4.1 are maintained as +Deviation entries in the source tree at +`src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md`. Two of them +change values that an existing pipeline may compare against. + +- **`LatticeConstants` angle slots are degrees, not radians** (`ReadH5OinaDataFilter-D6`). + Every `.dream3d` file written by 7.0.0 through 7.4.1 carries radians in components 3, 4 + and 5 of that array, so any saved exemplar, regression baseline or pipeline comparison + that reads them changes value. Multiply a stored radian value by 180/pi to compare it + against a new import, and remove any downstream conversion that was compensating for + the radian values. Nothing else in the import changes unit: the `Euler` array is still + radians and the three lattice dimensions are still unconverted. +- **The hexagonal x-axis alignment adds 30 degrees, not 30 radians** + (`ReadH5OinaDataFilter-D1`). Orientations imported from a file with a hexagonal phase + by 7.0.0 through 7.4.1 with *Convert Hexagonal X-Axis to EDAX Standard* left on — its + default — are wrong and cannot be corrected after the fact; re-import the file. + ## Example Pipelines ## References -[1] Rollett, A.D. Lecture Slides located at [http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf](http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf) +[1] Wright, S. I. and De Graef, M., "Electron backscatter diffraction," *International Tables for Crystallography*. [EBSD reference-frame conventions](https://onlinelibrary.wiley.com/iucr/itc/Cc/wf5160/). ## DREAM3D-NX Help diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp index 9f934961f1..c19d7a95d7 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp @@ -3,44 +3,131 @@ #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "H5Support/H5Lite.h" +#include "H5Support/H5ScopedSentinel.h" +#include "H5Support/H5Utilities.h" + +#include + +#include +#include +#include +#include +#include + using namespace nx::core; namespace { +// The EDAX/TSL convention aligns the hexagonal crystal x-axis with [2-1-10] while +// Oxford Instruments aligns it with [10-10]; converting between them is a 30 degree +// rotation about [0001] applied to phi2. An H5OINA file stores its Euler angles in +// RADIANS (unlike a .ctf file, which stores degrees), so the value added here is 30 +// degrees expressed in radians -- 30 * (pi/180), i.e. pi/6 -- and not the literal 30. +// The addition runs on a double intermediate so the stored float32 is the correctly +// rounded result, matching how the .ctf importer applies the same correction. +constexpr float64 k_HexagonalAlignmentRadians = 30.0 * ebsdlib::constants::k_PiOver180D; + +// The nine datasets H5OINAReader reads out of a scan's Data group, paired with the +// number of components each one carries per scan point. +const std::vector> k_RequiredDataSets = { + {ebsdlib::H5OINA::BandContrast, 1}, {ebsdlib::H5OINA::BandSlope, 1}, {ebsdlib::H5OINA::Bands, 1}, {ebsdlib::H5OINA::Error, 1}, {ebsdlib::H5OINA::Euler, 3}, + {ebsdlib::H5OINA::MeanAngularDeviation, 1}, {ebsdlib::H5OINA::Phase, 1}, {ebsdlib::H5OINA::X, 1}, {ebsdlib::H5OINA::Y, 1}, +}; + +/** + * @brief Confirms that every Data dataset of a scan holds exactly as many elements as + * the Image Geometry expects. + * + * H5OINAReader sizes its buffers to whatever extent each dataset actually has, while + * the geometry and the destination arrays are sized from the header's X Cells and + * Y Cells. If a file's datasets are shorter than the header claims, copying + * totalPoints elements out of those buffers reads past their end. + * + * No reader API exposes the sizes the reader actually allocated, so the extents are + * re-probed from the file with H5Lite. That is the file's extents as of this call + * rather than the buffer sizes themselves; the two agree unless the file changed + * between H5OINAReader::readFile() and this call. A dataset LONGER than the header + * describes is rejected as well as a shorter one: the reader would size its buffers + * to the longer extent and the extra rows would be silently dropped, which is not a + * result a caller can distinguish from a correct import. + */ +Result<> validateDataSetExtents(const std::filesystem::path& filePath, const std::string& scanName, usize totalPoints) +{ + hid_t fileId = H5Support::H5Utilities::openFile(filePath.string(), true); + if(fileId < 0) + { + return MakeErrorResult( + -34971, fmt::format("The file '{}' could not be reopened to verify the extents of scan '{}'. The file may have been moved or changed since preflight.", filePath.string(), scanName)); + } + H5Support::H5ScopedFileSentinel sentinel(fileId, false); + + const std::string dataGroupPath = fmt::format("/{}/{}/{}", scanName, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data); + for(const auto& [dataSetName, componentCount] : k_RequiredDataSets) + { + const std::string dataSetPath = dataGroupPath + "/" + dataSetName; + std::vector dims; + H5T_class_t classType = H5T_NO_CLASS; + usize typeSize = 0; + if(H5Support::H5Lite::getDatasetInfo(fileId, dataSetPath, dims, classType, typeSize) < 0) + { + // A missing dataset is fatal inside H5OINAReader, which has already run by the + // time this is called, so there is nothing to add here. + continue; + } + usize elementCount = 1; + for(const hsize_t dim : dims) + { + elementCount *= static_cast(dim); + } + const usize expectedCount = totalPoints * componentCount; + if(elementCount != expectedCount) + { + return MakeErrorResult(-34971, fmt::format("The dataset '{}' of scan '{}' in '{}' holds {} element(s), but the {} scan point(s) described by the scan's header require {}. The file is " + "malformed or was changed after preflight.", + dataSetPath, scanName, filePath.string(), elementCount, totalPoints, expectedCount)); + } + } + return {}; +} template -void copyRawData(const ReadH5DataInputValues* m_InputValues, size_t totalPoints, DataStructure& m_DataStructure, ebsdlib::H5OINAReader& m_Reader, const std::string& name, usize offset) +void copyRawData(const ReadH5DataInputValues* inputValues, usize count, DataStructure& dataStructure, ebsdlib::H5OINAReader& reader, const std::string& name, usize offset) { using ArrayType = DataArray; - auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(name)); + auto& dataRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(name)); auto* dataStorePtr = dataRef.getDataStore(); - const nonstd::span rawDataPtr(reinterpret_cast(m_Reader.getPointerByName(name)), totalPoints); - std::copy(rawDataPtr.begin(), rawDataPtr.end(), dataStorePtr->begin() + offset); + const nonstd::span rawDataSpan(reinterpret_cast(reader.getPointerByName(name)), count); + std::copy(rawDataSpan.begin(), rawDataSpan.end(), dataStorePtr->begin() + offset); } +/** + * @brief Applies the EDAX hexagonal x-axis alignment to the phi2 of every hexagonal + * point of one scan's tuple slab. + */ template -void convertHexEulerAngle(const ReadH5DataInputValues* m_InputValues, size_t totalPoints, DataStructure& m_DataStructure) +void convertHexEulerAngle(const ReadH5DataInputValues* inputValues, usize totalPoints, usize tupleOffset, DataStructure& dataStructure) { using ArrayType = DataArray; - if(m_InputValues->EdaxHexagonalAlignment) - { - auto& crystalStructuresRef = m_DataStructure.getDataRefAs(m_InputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - auto& crystalStructuresDSRef = crystalStructuresRef.getDataStoreRef(); + const auto& crystalStructuresRef = dataStructure.getDataRefAs(inputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); + const auto& crystalStructuresDSRef = crystalStructuresRef.getDataStoreRef(); - auto& cellPhasesRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); - auto& cellPhasesDSRef = cellPhasesRef.getDataStoreRef(); + const auto& cellPhasesRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); + const auto& cellPhasesDSRef = cellPhasesRef.getDataStoreRef(); - auto& eulerRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Euler)); - auto& eulerDataStoreRef = eulerRef.getDataStoreRef(); + auto& eulerRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Euler)); + auto& eulerDataStoreRef = eulerRef.getDataStoreRef(); - for(size_t i = 0; i < totalPoints; i++) + // Only this scan's slab is visited. Looping from 0 every time would shift the first + // scan's points once per scan and never reach the later scans' points. + for(usize tupleIdx = tupleOffset; tupleIdx < tupleOffset + totalPoints; tupleIdx++) + { + if(crystalStructuresDSRef[cellPhasesDSRef[tupleIdx]] == ebsdlib::CrystalStructure::Hexagonal_High) { - if(crystalStructuresDSRef[cellPhasesDSRef[i]] == ebsdlib::CrystalStructure::Hexagonal_High) - { - eulerDataStoreRef[3 * i + 2] = eulerDataStoreRef[3 * i + 2] + 30.0F; // See the documentation for this correction factor - } + const auto phi2 = static_cast(eulerDataStoreRef[3 * tupleIdx + 2]); + eulerDataStoreRef[3 * tupleIdx + 2] = static_cast(phi2 + k_HexagonalAlignmentRadians); } } } @@ -48,8 +135,8 @@ void convertHexEulerAngle(const ReadH5DataInputValues* m_InputValues, size_t tot } // namespace // ----------------------------------------------------------------------------- -ReadH5OinaData::ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues) -: IEbsdOemReader(dataStructure, mesgHandler, shouldCancel, inputValues) +ReadH5OinaData::ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues) +: IEbsdOemReader(dataStructure, messageHandler, shouldCancel, inputValues) { } @@ -59,75 +146,125 @@ ReadH5OinaData::~ReadH5OinaData() noexcept = default; // ----------------------------------------------------------------------------- Result<> ReadH5OinaData::operator()() { - return execute(); + auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); + imageGeom.setUnits(IGeometry::LengthUnit::Micrometer); + + // The scan loop is kept here rather than in IEbsdOemReader::execute() so that the + // cancel checks, the progress messages and the stacking order below apply to this + // filter only. + // + // The stacking order chooses which end of the selection list lands in tuple slab 0: + // Low-to-High reads the scans in the order they are listed, High-to-Low reads them in + // the reverse of that order, so the last selected scan occupies slab 0. + std::vector orderedScanNames(m_InputValues->SelectedScanNames.scanNames.cbegin(), m_InputValues->SelectedScanNames.scanNames.cend()); + if(m_InputValues->SelectedScanNames.stackingOrder == RefFrameZDir::k_HightoLow) + { + std::reverse(orderedScanNames.begin(), orderedScanNames.end()); + } + + const usize scanCount = orderedScanNames.size(); + for(usize scanIdx = 0; scanIdx < scanCount; scanIdx++) + { + if(m_ShouldCancel) + { + return {}; + } + + m_CurrentScanName = orderedScanNames[scanIdx]; + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", m_CurrentScanName, scanIdx + 1, scanCount)}); + Result<> readResults = readData(m_CurrentScanName); + if(readResults.invalid()) + { + return readResults; + } + + if(m_ShouldCancel) + { + return {}; + } + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", m_CurrentScanName, scanIdx + 1, scanCount)}); + Result<> copyDataResults = copyRawEbsdData(static_cast(scanIdx)); + if(copyDataResults.invalid()) + { + return copyDataResults; + } + } + return {}; } // ----------------------------------------------------------------------------- -Result<> ReadH5OinaData::copyRawEbsdData(int index) +Result<> ReadH5OinaData::copyRawEbsdData(int scanIndex) { const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const usize totalPoints = imageGeom.getNumXCells() * imageGeom.getNumYCells(); - const usize offset = index * totalPoints; - - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandContrast, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandSlope, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Bands, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Error, offset); - copyRawData(m_InputValues, totalPoints * 3, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Euler, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::MeanAngularDeviation, offset); + // Scan `scanIndex` occupies the tuple slab [scanIndex * totalPoints, (scanIndex + 1) * totalPoints). + const usize tupleOffset = static_cast(scanIndex) * totalPoints; + + const std::string& scanName = m_CurrentScanName; + + if(Result<> extentResults = validateDataSetExtents(m_InputValues->SelectedScanNames.inputFilePath, scanName, totalPoints); extentResults.invalid()) + { + return extentResults; + } + + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandContrast, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandSlope, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Bands, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Error, tupleOffset); + // Euler carries three components per scan point, so both the element count and the + // destination offset are three times the tuple counts. + copyRawData(m_InputValues, totalPoints * 3, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Euler, tupleOffset * 3); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::MeanAngularDeviation, tupleOffset); + + // The phase value of every point indexes the ensemble arrays, both in the alignment + // loop below and in every downstream filter, so it is range checked before it is + // stored. The valid range is [0, phase count]: 0 is the reserved Invalid Phase slot. + { + const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); + const usize ensembleTupleCount = crystalStructures.getNumberOfTuples(); + const nonstd::span rawPhaseSpan(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); + for(usize tupleIdx = 0; tupleIdx < totalPoints; tupleIdx++) + { + if(static_cast(rawPhaseSpan[tupleIdx]) >= ensembleTupleCount) + { + return MakeErrorResult(-34972, fmt::format("Scan point {} of scan '{}' in '{}' carries phase value {}, which is outside the valid range [0, {}] established by the file's phase definitions.", + tupleIdx, scanName, m_InputValues->SelectedScanNames.inputFilePath.string(), rawPhaseSpan[tupleIdx], ensembleTupleCount - 1)); + } + } + } + if(m_InputValues->ConvertPhaseToInt32) { - const nonstd::span rawDataPtr(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); - using ArrayType = DataArray; - auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); + const nonstd::span rawDataSpan(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); + auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); auto* dataStorePtr = dataRef.getDataStore(); - for(size_t i = 0; i < totalPoints; i++) + for(usize tupleIdx = 0; tupleIdx < totalPoints; tupleIdx++) { - dataStorePtr->setValue(i + offset, static_cast(rawDataPtr[i])); + dataStorePtr->setValue(tupleIdx + tupleOffset, static_cast(rawDataSpan[tupleIdx])); } } else { - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Phase, offset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Phase, tupleOffset); } - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::X, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Y, offset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::X, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Y, tupleOffset); - if(m_InputValues->EdaxHexagonalAlignment) + if(m_ShouldCancel) { - if(m_InputValues->ConvertPhaseToInt32) - { - convertHexEulerAngle(m_InputValues, totalPoints, m_DataStructure); - } - else - { - convertHexEulerAngle(m_InputValues, totalPoints, m_DataStructure); - } + return {}; } - if(m_InputValues->ReadPatternData) + if(m_InputValues->EdaxHexagonalAlignment) { - const uint16* patternDataPtr = m_Reader->getPatternData(); - if(patternDataPtr == nullptr) + if(m_InputValues->ConvertPhaseToInt32) { - return MakeErrorResult(-34970, "Pattern data was requested but no pattern data was found in the data file"); + convertHexEulerAngle(m_InputValues, totalPoints, tupleOffset, m_DataStructure); } - std::array pDims = {{0, 0}}; - m_Reader->getPatternDims(pDims); - if(pDims[0] != 0 && pDims[1] != 0) + else { - std::vector pDimsV(2); - pDimsV[0] = pDims[0]; - pDimsV[1] = pDims[1]; - auto& patternData = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::UnprocessedPatterns)); - const usize numComponents = patternData.getNumberOfComponents(); - for(usize i = 0; i < totalPoints; i++) - { - for(usize j = 0; j < numComponents; ++j) - { - patternData[offset + numComponents * i + j] = patternDataPtr[numComponents * i + j]; - } - } + convertHexEulerAngle(m_InputValues, totalPoints, tupleOffset, m_DataStructure); } } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp index f0dbb9cf5d..efeb9d46cb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp @@ -3,19 +3,20 @@ #include "OrientationAnalysis/OrientationAnalysis_export.hpp" #include "OrientationAnalysis/utilities/IEbsdOemReader.hpp" +#include + namespace nx::core { /** * @class ReadH5OinaData - * @brief This filter will read a single .h5 file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the - * intermediate .h5ebsd file. + * @brief Reads one or more scans from an H5OINA file into one Image Geometry. */ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaData : public IEbsdOemReader { public: - ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues); + ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues); ~ReadH5OinaData() noexcept override; ReadH5OinaData(const ReadH5OinaData&) = delete; @@ -25,7 +26,16 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaData : public IEbsdOemReader operator()(); - Result<> copyRawEbsdData(int index) override; + Result<> copyRawEbsdData(int scanIndex) override; + +private: + /** + * @brief The scan name whose data is currently in the reader's buffers. The shared + * copyRawEbsdData(int) signature carries only the destination slab index, and under a + * High-to-Low stacking order that index no longer matches the scan's position in the + * selection list, so the name is carried here instead of being re-derived from it. + */ + std::string m_CurrentScanName; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp index d204390fe4..30394bf172 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp @@ -22,8 +22,12 @@ #include #include -#include -namespace fs = std::filesystem; +#include +#include +#include +#include + +#include using namespace nx::core; @@ -74,7 +78,10 @@ Parameters ReadH5OinaDataFilter::parameters() const params.insert(std::make_unique(k_ConvertPhaseToInt32_Key, "Convert Phase Data to Int32", "Native Phases data value is uint8. Convert to Int32 for better filter compatibility", true)); params.insert(std::make_unique(k_Origin_Key, "Origin", "The origin of the volume", std::vector{0.0F, 0.0F, 0.0F}, std::vector{"x", "y", "z"})); params.insert(std::make_unique(k_ZSpacing_Key, "Z Spacing (Microns)", "The spacing in microns between each layer.", 1.0f)); - params.insert(std::make_unique(k_ReadPatternData_Key, "Import Pattern Data", "Whether or not to import the pattern data", false)); + params.insert(std::make_unique(k_ReadPatternData_Key, "Import Pattern Data", + "Whether or not to import the diffraction pattern data. Pattern import is not yet supported for H5OINA files, so turning this on stops the filter " + "with an error.", + false)); params.insertSeparator(Parameters::Separator{"Output Image Geometry"}); params.insert(std::make_unique(k_CreatedImageGeometryPath_Key, "Image Geometry", "The path to the created Image Geometry", DataPath({ImageGeom::k_TypeName}))); params.insertSeparator(Parameters::Separator{"Output Cell Attribute Matrix"}); @@ -90,7 +97,9 @@ Parameters ReadH5OinaDataFilter::parameters() const //------------------------------------------------------------------------------ IFilter::VersionType ReadH5OinaDataFilter::parametersVersion() const { - return 1; + // Version 2: Pattern import is explicitly unsupported, and multi-scan inputs + // must have compatible geometry and identical phase definitions. + return 2; } //------------------------------------------------------------------------------ @@ -115,30 +124,200 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure DataPath cellEnsembleAMPath = pImageGeometryNameValue.createChildPath(pCellEnsembleAttributeMatrixNameValue); DataPath cellAMPath = pImageGeometryNameValue.createChildPath(pCellAttributeMatrixNameValue); - PreflightResult preflightResult; nx::core::Result resultOutputActions; std::vector preflightUpdatedValues; - if(pZSpacingValue <= 0) + const std::string inputFilePath = pSelectedScanNamesValue.inputFilePath.string(); + + if(!std::isfinite(pZSpacingValue) || pZSpacingValue <= 0) { - return MakePreflightErrorResult(-9580, fmt::format("The Z Spacing field contains a value ({}) that is non-positive. The Z Spacing field must be set to a positive value.", pZSpacingValue)); + return MakePreflightErrorResult(-9580, fmt::format("The Z Spacing value ({}) must be finite and positive.", pZSpacingValue)); } if(pSelectedScanNamesValue.scanNames.empty()) { - return MakePreflightErrorResult(-9581, "At least one scan must be chosen. Please select a scan from the list."); + return MakePreflightErrorResult(-9581, "At least one scan must be chosen. Please select a scan from the list."); + } + if(pReadPatternDataValue) + { + return MakePreflightErrorResult(-9583, fmt::format("Pattern import is not yet supported for H5OINA files, so 'Import Pattern Data' must be turned off to read '{}'. The diffraction " + "patterns a file does contain can be read with the 'Read HDF5 Dataset' filter.", + inputFilePath)); } // read in the necessary info from the input h5 file ebsdlib::H5OINAReader reader; - reader.setFileName(pSelectedScanNamesValue.inputFilePath.string()); - reader.setReadPatternData(pReadPatternDataValue); - reader.setHDF5Path(pSelectedScanNamesValue.scanNames.front()); + reader.setFileName(inputFilePath); + reader.setReadPatternData(false); + + // Every selected scan must be present in the file. Checking only the first one + // leaves a bad later name to fail part way through execute, with the scans that + // were already imported left in the output arrays. + std::list availableScanNames; + if(const int err = reader.readScanNames(availableScanNames); err < 0) + { + return MakePreflightErrorResult(-9582, fmt::format("An error occurred while listing the scans in '{}'.\n Error Code: {}\n Message: {}", inputFilePath, err, reader.getErrorMessage())); + } + for(const std::string& scanName : pSelectedScanNamesValue.scanNames) + { + if(std::find(availableScanNames.cbegin(), availableScanNames.cend(), scanName) == availableScanNames.cend()) + { + std::string availableList; + for(const std::string& availableScanName : availableScanNames) + { + availableList += (availableList.empty() ? "" : ", ") + availableScanName; + } + return MakePreflightErrorResult(-9586, fmt::format("The selected scan '{}' is not present in '{}'. The scans available in this file are: {}", scanName, inputFilePath, + availableList.empty() ? std::string("") : availableList)); + } + } + + const std::string& firstScanName = pSelectedScanNamesValue.scanNames.front(); + reader.setHDF5Path(firstScanName); if(const int err = reader.readHeaderOnly(); err < 0) { - return MakePreflightErrorResult(-9582, fmt::format("An error occurred while reading the header data\n{} : {}", err, reader.getErrorMessage())); + return MakePreflightErrorResult( + -9582, fmt::format("An error occurred while reading the header of scan '{}' in '{}'.\n Error Code: {}\n Message: {}", firstScanName, inputFilePath, err, reader.getErrorMessage())); + } + + // The geometry is sized from these two values, so a count below 1 has to be + // rejected here rather than producing an empty or absurdly large geometry. + if(reader.getXDimension() < 1 || reader.getYDimension() < 1) + { + return MakePreflightErrorResult(-9584, fmt::format("The header of scan '{}' in '{}' reports X Cells = {} and Y Cells = {}. Both must be at least 1. The file may be malformed or may not be an " + "H5OINA file.", + firstScanName, inputFilePath, reader.getXDimension(), reader.getYDimension())); + } + + const auto validateSpacing = [&](const std::string& scanName, ebsdlib::H5OINAReader& scanReader) -> Result<> { + const float32 xStep = scanReader.getXStep(); + const float32 yStep = scanReader.getYStep(); + if(!std::isfinite(xStep) || !std::isfinite(yStep) || xStep <= 0.0F || yStep <= 0.0F) + { + return MakeErrorResult(-9591, fmt::format("Scan '{}' in '{}' reports X Step = {} and Y Step = {}. Both values must be finite and positive.", scanName, inputFilePath, xStep, yStep)); + } + return {}; + }; + + if(Result<> spacingCheck = validateSpacing(firstScanName, reader); spacingCheck.invalid()) + { + return MakePreflightErrorResult(spacingCheck.errors().front().code, spacingCheck.errors().front().message); } - // create the Image Geometry and it's attribute matrices + // The Ensemble Attribute Matrix is sized from the number of phase groups in the + // FIRST selected scan, but the shared ensemble fill in IEbsdOemReader::readData runs + // once per selected scan and places each phase at the index carried by its HDF5 group + // name. Both properties below therefore have to hold for EVERY selected scan, not just + // the first: a later scan with a group named outside 1..N, or with more phase groups + // than the first scan, writes past the end of the ensemble arrays at execute. + const auto phases = reader.getPhaseVector(); + const usize ensemblePhaseCount = phases.size(); + const auto validatePhaseGroups = [&](const std::string& scanName, const auto& scanPhases, bool comparePhaseDefinitions) -> Result<> { + if(scanPhases.size() != ensemblePhaseCount) + { + return MakeErrorResult(-9589, fmt::format("Scan '{}' in '{}' declares {} phase group(s), but scan '{}' declares {}. Every selected scan must declare the same phase groups, because the single " + "Ensemble Attribute Matrix that all of the stacked scans share is sized and filled from those groups. Import scans with differing phase lists " + "separately.", + scanName, inputFilePath, scanPhases.size(), firstScanName, ensemblePhaseCount)); + } + for(const auto& phase : scanPhases) + { + const int32 phaseIndex = phase->getPhaseIndex(); + if(phaseIndex < 1 || static_cast(phaseIndex) > ensemblePhaseCount) + { + return MakeErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares {} phase(s), but one of them carries index {}. The phase groups of an H5OINA file must be named 1 through {}.", scanName, + inputFilePath, ensemblePhaseCount, phaseIndex, ensemblePhaseCount)); + } + + if(comparePhaseDefinitions) + { + const auto referencePhaseIter = std::find_if(phases.cbegin(), phases.cend(), [phaseIndex](const auto& referencePhase) { return referencePhase->getPhaseIndex() == phaseIndex; }); + if(referencePhaseIter == phases.cend()) + { + return MakeErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares phase index {}, but scan '{}' does not. Every selected scan must use the same phase group names.", scanName, + inputFilePath, phaseIndex, firstScanName)); + } + + const auto& referencePhase = *referencePhaseIter; + if(phase->getPhaseName() != referencePhase->getPhaseName()) + { + return MakeErrorResult(-9590, + fmt::format("Phase group {} of scan '{}' in '{}' has material name '{}', but the same group of scan '{}' has material name '{}'. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, phase->getPhaseName(), firstScanName, referencePhase->getPhaseName())); + } + if(phase->getLaueGroup() != referencePhase->getLaueGroup()) + { + return MakeErrorResult(-9590, fmt::format("Phase group {} of scan '{}' in '{}' has Laue group {}, but the same group of scan '{}' has Laue group {}. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, static_cast(phase->getLaueGroup()), firstScanName, static_cast(referencePhase->getLaueGroup()))); + } + if(phase->getSpaceGroup() != referencePhase->getSpaceGroup()) + { + return MakeErrorResult(-9590, fmt::format("Phase group {} of scan '{}' in '{}' has space group {}, but the same group of scan '{}' has space group {}. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, phase->getSpaceGroup(), firstScanName, referencePhase->getSpaceGroup())); + } + + const std::vector scanLatticeConstants = phase->getLatticeConstants(); + const std::vector referenceLatticeConstants = referencePhase->getLatticeConstants(); + if(scanLatticeConstants != referenceLatticeConstants) + { + return MakeErrorResult(-9590, + fmt::format("Phase group {} of scan '{}' in '{}' has lattice constants [{}], but the same group of scan '{}' has lattice constants [{}]. The selected scans form " + "one 3D microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, fmt::join(scanLatticeConstants, ", "), firstScanName, fmt::join(referenceLatticeConstants, ", "))); + } + } + } + return {}; + }; + + if(Result<> phaseCheck = validatePhaseGroups(firstScanName, phases, false); phaseCheck.invalid()) + { + return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); + } + + // Every other selected scan has to describe the same grid, because the geometry + // and every cell array are sized from the first scan's header alone. A second + // reader is used so the checks below do not disturb the header state that the + // preflight-updated values and the output actions are built from. + { + ebsdlib::H5OINAReader scanCheckReader; + scanCheckReader.setFileName(inputFilePath); + scanCheckReader.setReadPatternData(false); + for(const std::string& scanName : pSelectedScanNamesValue.scanNames) + { + if(scanName == firstScanName) + { + continue; + } + scanCheckReader.setHDF5Path(scanName); + if(const int err = scanCheckReader.readHeaderOnly(); err < 0) + { + return MakePreflightErrorResult( + -9582, fmt::format("An error occurred while reading the header of scan '{}' in '{}'.\n Error Code: {}\n Message: {}", scanName, inputFilePath, err, scanCheckReader.getErrorMessage())); + } + if(Result<> spacingCheck = validateSpacing(scanName, scanCheckReader); spacingCheck.invalid()) + { + return MakePreflightErrorResult(spacingCheck.errors().front().code, spacingCheck.errors().front().message); + } + if(scanCheckReader.getXDimension() != reader.getXDimension() || scanCheckReader.getYDimension() != reader.getYDimension() || scanCheckReader.getXStep() != reader.getXStep() || + scanCheckReader.getYStep() != reader.getYStep()) + { + return MakePreflightErrorResult( + -9585, fmt::format("Scan '{}' in '{}' describes a {} x {} grid with steps ({}, {}), but scan '{}' describes a {} x {} grid with steps ({}, {}). Every selected scan must describe the same " + "grid, because they are stacked into a single Image Geometry.", + scanName, inputFilePath, scanCheckReader.getXDimension(), scanCheckReader.getYDimension(), scanCheckReader.getXStep(), scanCheckReader.getYStep(), firstScanName, + reader.getXDimension(), reader.getYDimension(), reader.getXStep(), reader.getYStep())); + } + if(Result<> phaseCheck = validatePhaseGroups(scanName, scanCheckReader.getPhaseVector(), true); phaseCheck.invalid()) + { + return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); + } + } + } + + // Create the Image Geometry and its attribute matrices. const CreateImageGeometryAction::DimensionType dims = {static_cast(reader.getXDimension()), static_cast(reader.getYDimension()), pSelectedScanNamesValue.scanNames.size()}; const ShapeType tupleDims = {dims[2], dims[1], dims[0]}; { @@ -151,7 +330,6 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure EbsdReaderUtilities::GeneratePreflightScanInformation(reader, preflightUpdatedValues); EbsdReaderUtilities::GeneratePreflightPhaseInformation(reader, preflightUpdatedValues); - const auto phases = reader.getPhaseVector(); std::vector ensembleTupleDims{phases.size() + 1}; { auto createAttributeMatrixAction = std::make_unique(cellEnsembleAMPath, ensembleTupleDims); @@ -190,19 +368,6 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure resultOutputActions.value().appendAction(std::make_unique(DataType::float32, tupleDims, std::vector{1}, cellAMPath.createChildPath(ebsdlib::H5OINA::X))); resultOutputActions.value().appendAction(std::make_unique(DataType::float32, tupleDims, std::vector{1}, cellAMPath.createChildPath(ebsdlib::H5OINA::Y))); - if(pReadPatternDataValue) - { - std::array patternDims = {{0, 0}}; - reader.getPatternDims(patternDims); - if(patternDims[0] == 0 || patternDims[1] == 0) - { - return MakePreflightErrorResult(-9583, fmt::format("The parameter 'Read Pattern Data' has been enabled but there does not seem to be any pattern data in the file for the scan name selected")); - } - auto createArrayAction = std::make_unique(DataType::uint16, tupleDims, std::vector{static_cast(patternDims[0]), static_cast(patternDims[1])}, - cellAMPath.createChildPath(ebsdlib::H5OINA::UnprocessedPatterns)); - resultOutputActions.value().appendAction(std::move(createArrayAction)); - } - return {std::move(resultOutputActions), std::move(preflightUpdatedValues)}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp index 103700b0f0..690dc5a2b5 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp @@ -116,4 +116,3 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaDataFilter : public IFilter } // namespace nx::core SIMPLNX_DEF_FILTER_TRAITS(nx::core, ReadH5OinaDataFilter, "fad3d47f-f1e1-4429-bc65-5e021be62ba0"); -/* LEGACY UUID FOR THIS FILTER 3ff4701b-3a0c-52e3-910a-fa927aa6584c */ diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 1ccd5c77da..7c6aa4503d 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -1,134 +1,1373 @@ -#include +/* ============================================================================ + * ReadH5OinaData V&V test suite. + * + * Verification is established INDEPENDENTLY of any DREAM3D output, per the V&V + * policy (src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md). There is + * no DREAM3D 6.5.171 H5OINA importer to compare against, so the oracle carries + * the whole burden: + * + * - .h5oina parsing : Part of the system under test. EbsdLib's H5OINAReader + * owns HDF5 traversal, header and phase parsing, required- + * dataset enforcement, and its own error codes. The tests + * verify its user-visible results against analytical fixture + * values and the independent h5py readback. + * - SIMPLNX value-add : Class 1 (analytical) + Class 4 (invariant). The filter's + * value-add is deterministic plumbing on top of the reader: + * geometry construction from the first scan's header, array + * creation and typing, per-scan slab offsets, the verbatim + * column copies, the uint8 -> int32 Phase widening, the + * EDAX hexagonal alignment applied to phi2, ensemble slot-0 + * defaults and per-phase fill, and the malformed-input + * guards. The toy .h5oina fixtures below are written by + * this file with H5Lite and every expected value is derived + * from the fixture spec, never from observed output. + * - Real AZtec file : Class 2 independent readback. The archived production + * H5Oina_Test_Data.h5oina is compared against a readback of + * its own Data group datasets performed with H5Lite, i.e. the + * raw file bytes, bypassing H5OINAReader entirely. An + * equivalent h5py readback in a second language is recorded + * in the V&V working folder, described in + * vv/provenance/ReadH5OinaDataFilter.md. + * + * Precision pinning (the ReadCtfData lesson): every floating-point value written + * into a toy fixture below is a float32 literal stored as float32, so the file + * round-trips it bit-for-bit and the verbatim copies are asserted with exact + * equality. (The cell counts and the Laue and space group numbers are int32 + * scalars and five of the nine data columns are uint8, all of which round-trip + * exactly as well. The production file's values are whatever AZtec wrote; that + * test asserts exact equality too, but against an independent readback of the + * file rather than against literals.) + * + * The hexagonal alignment adds 30 degrees expressed in radians, computed with a + * double-precision intermediate and stored back as float32. Five fixture phi2 + * values -- 0.1F and 0.34F, used in Fixture A and in Fixture C's first scan, and + * 0.09F, 0.22F and 0.35F in Fixture C's second scan -- are chosen because their + * correctly-rounded float32 results DIFFER between a double-precision + * intermediate and a float32 intermediate, so the expectations pin the shape of + * the arithmetic and not merely its magnitude. Note that none of those five is an + * exactly representable decimal, and that is deliberate: a dyadic value has + * trailing zero mantissa bits, so adding the constant rounds to the same float32 + * either way and cannot separate the two paths. 0.25F, 0.5F and 0.75F are exactly + * representable and are carried alongside on hexagonal points as controls whose + * results are identical under either intermediate. The hexagonal phase's third + * lattice angle, 2.0943952F -> 120.0F, is a sixth such discriminator: a float32 + * intermediate in the radians-to-degrees conversion would give 120.00000762939453F. + * Every such literal was derived with IEEE-754 float32/float64 semantics in NumPy; + * the derivation script and its recorded output are described in + * vv/provenance/ReadH5OinaDataFilter.md. + * + * The archive's H5Oina_Test_Data.dream3d exemplar is no longer consulted: it was + * generated by this very filter, so it is a self-oracle that pins "the filter + * keeps doing what it did" rather than "the filter is right". + * ========================================================================== */ +#include "OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp" +#include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" +#include "OrientationAnalysis/Parameters/OEMEbsdScanSelectionParameter.h" + +#include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/DataGroupCreationParameter.hpp" #include "simplnx/Parameters/VectorParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp" -#include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" -#include "OrientationAnalysis/Parameters/OEMEbsdScanSelectionParameter.h" +#include "H5Support/H5Lite.h" +#include "H5Support/H5Utilities.h" + +#include +#include +#include + +#include -#include +#include +#include #include +#include +#include +#include + namespace fs = std::filesystem; using namespace nx::core; using namespace nx::core::Constants; +using namespace H5Support; namespace { -const std::string k_ScanName = "1"; +const std::string k_CellAttributeMatrixName = "Cell Data"; +const std::string k_CellEnsembleAttributeMatrixName = "Cell Ensemble Data"; +const DataPath k_ImageGeomPath({ImageGeom::k_TypeName}); +const DataPath k_CellAMPath = k_ImageGeomPath.createChildPath(k_CellAttributeMatrixName); +const DataPath k_EnsembleAMPath = k_ImageGeomPath.createChildPath(k_CellEnsembleAttributeMatrixName); + +// EbsdLib/IO/HKL/CtfConstants.h LaueGroupTable: 9 = LG_Hexagonal_High, 11 = LG_Cubic_High. +// EbsdLib/Core/EbsdLibConstants.h: Hexagonal_High = 0, Cubic_High = 1, UnknownCrystalStructure = 999. +constexpr int32 k_LaueHexagonalHigh = 9; +constexpr int32 k_LaueCubicHigh = 11; + +//------------------------------------------------------------------------------ +// Toy .h5oina fixture description. Written out in full, the dataset set below is +// exactly the minimum H5OINAReader requires: the four Header scalars, one or more +// Phases/ groups carrying Phase Name / Lattice Dimensions / Lattice Angles / +// Laue Group / Space Group, and the nine Data datasets it reads unconditionally. +// The root Manufacturer / Software Version / Index datasets are written for +// realism only -- the reader never reads them. +// +// The guard fixtures deliberately depart from that minimum: `omitBandsDataset` +// and `omitLatticeAngles` drop a required dataset outright, and `groupName` +// overrides the phase index that the group name carries. The remaining guard +// fixtures keep the full dataset set but give it values a well formed file would +// not have. +//------------------------------------------------------------------------------ +struct PhaseSpec +{ + std::string name; + int32 laueGroup = k_LaueCubicHigh; + int32 spaceGroup = 225; + std::array latticeDimensions = {1.0F, 1.0F, 1.0F}; + // Lattice angles as an H5OINA file stores them: RADIANS. + std::array latticeAngles = {1.5707964F, 1.5707964F, 1.5707964F}; + // Group name override; empty means "use the phase's 1-based position". + std::string groupName; + bool omitLatticeAngles = false; +}; + +struct ScanSpec +{ + std::string name = "1"; + int32 xCells = 1; + int32 yCells = 1; + float32 xStep = 1.0F; + float32 yStep = 1.0F; + std::vector phases; + std::vector phase; + std::vector bandContrast; + std::vector bandSlope; + std::vector bands; + std::vector error; + std::vector euler; // 3 components per point, interleaved + std::vector mad; + std::vector x; + std::vector y; + // Guard-fixture switches. + bool omitBandsDataset = false; +}; + +//------------------------------------------------------------------------------ +template +void WriteVector(hid_t gid, const std::string& name, const std::vector& dims, const std::vector& data) +{ + const herr_t err = H5Lite::writeVectorDataset(gid, name, dims, data); + REQUIRE(err >= 0); } -TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Valid Filter Execution", "[OrientationAnalysis][ReadH5OinaDataFilter]") +//------------------------------------------------------------------------------ +hid_t OpenOrCreateGroup(hid_t fileId, const std::string& path) +{ + const hid_t err = H5Utilities::createGroupsFromPath(path, fileId); + REQUIRE(err >= 0); + const hid_t gid = H5Gopen(fileId, path.c_str(), H5P_DEFAULT); + REQUIRE(gid >= 0); + return gid; +} + +//------------------------------------------------------------------------------ +// Writes the fixture to / and returns the path. +//------------------------------------------------------------------------------ +fs::path WriteH5OinaFixture(const std::string& fileName, const std::vector& scans, bool writeFormatVersion = true, const std::string& formatVersion = "5.0") +{ + const fs::path filePath = fs::path(unit_test::k_BinaryTestOutputDir.view()) / fileName; + const hid_t fileId = H5Fcreate(filePath.string().c_str(), H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + REQUIRE(fileId >= 0); + + if(writeFormatVersion) + { + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::FormatVersion, formatVersion) >= 0); + } + // Inert realism -- the reader never reads these three. + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::Manufacturer, "Oxford Instruments") >= 0); + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::SoftwareVersion, "V&V toy fixture") >= 0); + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::Index, "1") >= 0); + + for(const ScanSpec& scan : scans) + { + const std::string headerPath = fmt::format("{}/{}/{}", scan.name, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Header); + const hid_t headerGid = OpenOrCreateGroup(fileId, headerPath); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::XCells, scan.xCells) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::YCells, scan.yCells) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::XStep, scan.xStep) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::YStep, scan.yStep) >= 0); + + for(usize i = 0; i < scan.phases.size(); i++) + { + const PhaseSpec& phaseSpec = scan.phases[i]; + const std::string groupName = phaseSpec.groupName.empty() ? std::to_string(i + 1) : phaseSpec.groupName; + const hid_t phaseGid = OpenOrCreateGroup(fileId, fmt::format("{}/{}/{}", headerPath, ebsdlib::H5OINA::Phases, groupName)); + REQUIRE(H5Lite::writeStringDataset(phaseGid, ebsdlib::H5OINA::PhaseName, phaseSpec.name) >= 0); + WriteVector(phaseGid, ebsdlib::H5OINA::LatticeDimensions, {1, 3}, {phaseSpec.latticeDimensions[0], phaseSpec.latticeDimensions[1], phaseSpec.latticeDimensions[2]}); + if(!phaseSpec.omitLatticeAngles) + { + WriteVector(phaseGid, ebsdlib::H5OINA::LatticeAngles, {1, 3}, {phaseSpec.latticeAngles[0], phaseSpec.latticeAngles[1], phaseSpec.latticeAngles[2]}); + } + REQUIRE(H5Lite::writeScalarDataset(phaseGid, ebsdlib::H5OINA::LaueGroup, phaseSpec.laueGroup) >= 0); + REQUIRE(H5Lite::writeScalarDataset(phaseGid, ebsdlib::H5OINA::SpaceGroup, phaseSpec.spaceGroup) >= 0); + REQUIRE(H5Gclose(phaseGid) >= 0); + } + REQUIRE(H5Gclose(headerGid) >= 0); + + const hid_t dataGid = OpenOrCreateGroup(fileId, fmt::format("{}/{}/{}", scan.name, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data)); + const auto pointCount = static_cast(scan.phase.size()); + WriteVector(dataGid, ebsdlib::H5OINA::BandContrast, {pointCount}, scan.bandContrast); + WriteVector(dataGid, ebsdlib::H5OINA::BandSlope, {pointCount}, scan.bandSlope); + if(!scan.omitBandsDataset) + { + WriteVector(dataGid, ebsdlib::H5OINA::Bands, {pointCount}, scan.bands); + } + WriteVector(dataGid, ebsdlib::H5OINA::Error, {pointCount}, scan.error); + WriteVector(dataGid, ebsdlib::H5OINA::Euler, {pointCount, 3}, scan.euler); + WriteVector(dataGid, ebsdlib::H5OINA::MeanAngularDeviation, {pointCount}, scan.mad); + WriteVector(dataGid, ebsdlib::H5OINA::Phase, {pointCount}, scan.phase); + WriteVector(dataGid, ebsdlib::H5OINA::X, {pointCount}, scan.x); + WriteVector(dataGid, ebsdlib::H5OINA::Y, {pointCount}, scan.y); + REQUIRE(H5Gclose(dataGid) >= 0); + } + + REQUIRE(H5Fclose(fileId) >= 0); + return filePath; +} + +//------------------------------------------------------------------------------ +// Fixture phases. +//------------------------------------------------------------------------------ +// Hexagonal: Laue group 9 -> CrystalStructure Hexagonal_High (0). Its lattice +// angles are 90/90/120 degrees written as radians, which is what an H5OINA file +// carries; the third angle differs from the second so a gamma slot that echoed +// beta would be visible. +const PhaseSpec k_HexPhase{"Hex Phase A", k_LaueHexagonalHigh, 194, {2.5F, 2.5F, 4.0F}, {1.5707964F, 1.5707964F, 2.0943952F}}; +// Cubic: Laue group 11 -> CrystalStructure Cubic_High (1). +const PhaseSpec k_CubicPhase{"Cubic Phase B", k_LaueCubicHigh, 225, {3.5F, 3.5F, 3.5F}, {1.5707964F, 1.5707964F, 1.5707964F}}; + +//------------------------------------------------------------------------------ +// Fixture A: one scan "1", 3 x 2 cells, XStep 0.25 / YStep 0.5, two phases. +// +// Points 0, 1 and 2 all carry phi2 = 0.1F on a hexagonal, a cubic and an +// unindexed (phase 0) point respectively, so the three expectations differ only +// through the alignment branch. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureAScan() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 3; + scan.yCells = 2; + scan.xStep = 0.25F; + scan.yStep = 0.5F; + scan.phases = {k_HexPhase, k_CubicPhase}; + scan.phase = {1, 2, 0, 2, 1, 1}; + scan.bandContrast = {10, 20, 30, 40, 50, 60}; + scan.bandSlope = {11, 21, 31, 41, 51, 61}; + scan.bands = {1, 2, 3, 4, 5, 6}; + scan.error = {0, 1, 0, 2, 0, 3}; + scan.euler = { + 0.25F, 0.125F, 0.1F, // pt 0 hexagonal -> phi2 shifted + 0.5F, 0.375F, 0.1F, // pt 1 cubic -> phi2 untouched + 0.75F, 0.625F, 0.1F, // pt 2 unindexed -> phi2 untouched + 1.0F, 0.875F, 0.75F, // pt 3 cubic -> phi2 untouched + 1.25F, 1.125F, 0.34F, // pt 4 hexagonal -> phi2 shifted + 1.5F, 1.375F, 0.25F, // pt 5 hexagonal -> phi2 shifted (dyadic control) + }; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}; + scan.x = {0.0F, 0.25F, 0.5F, 0.0F, 0.25F, 0.5F}; + scan.y = {0.0F, 0.0F, 0.0F, 0.5F, 0.5F, 0.5F}; + return scan; +} + +// Fixture A expected Euler with the alignment ON. The three shifted values are the +// correctly-rounded float32 results of double(phi2) + 30*(pi/180): +// 0.1F -> 0.6235987544059753F (a float32 intermediate would give 0.6235988140106201F) +// 0.34F -> 0.8635987639427185F (a float32 intermediate would give 0.8635988235473633F) +// 0.25F -> 0.7735987901687622F (identical under either intermediate -- the control) +// Adding the literal 30.0F, as a degree-valued importer would, gives 30.1F / 30.34F / 30.25F. +const std::vector k_FixtureAEulerAligned = { + 0.25F, 0.125F, 0.6235987544059753F, 0.5F, 0.375F, 0.1F, 0.75F, 0.625F, 0.1F, 1.0F, 0.875F, 0.75F, 1.25F, 1.125F, 0.8635987639427185F, 1.5F, 1.375F, 0.7735987901687622F, +}; +// The same fixture with the alignment OFF: every value verbatim from the file. +const std::vector k_FixtureAEulerVerbatim = { + 0.25F, 0.125F, 0.1F, 0.5F, 0.375F, 0.1F, 0.75F, 0.625F, 0.1F, 1.0F, 0.875F, 0.75F, 1.25F, 1.125F, 0.34F, 1.5F, 1.375F, 0.25F, +}; + +//------------------------------------------------------------------------------ +// Fixture B: two scans, 2 x 2 each, CUBIC only. With no hexagonal points the +// Euler array must be a pure verbatim copy, so any slab-placement error is +// visible directly. Scan "2"'s values are disjoint from scan "1"'s. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureBScan1() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 2; + scan.yCells = 2; + scan.xStep = 0.25F; + scan.yStep = 0.5F; + scan.phases = {k_CubicPhase}; + scan.phase = {1, 1, 1, 1}; + scan.bandContrast = {10, 11, 12, 13}; + scan.bandSlope = {20, 21, 22, 23}; + scan.bands = {1, 2, 3, 4}; + scan.error = {0, 1, 2, 3}; + scan.euler = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F}; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F}; + scan.x = {0.0F, 0.25F, 0.0F, 0.25F}; + scan.y = {0.0F, 0.0F, 0.5F, 0.5F}; + return scan; +} + +ScanSpec MakeFixtureBScan2() +{ + ScanSpec scan = MakeFixtureBScan1(); + scan.name = "2"; + scan.bandContrast = {110, 111, 112, 113}; + scan.bandSlope = {120, 121, 122, 123}; + scan.bands = {5, 6, 7, 8}; + scan.error = {4, 5, 6, 7}; + scan.euler = {2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F}; + scan.mad = {1.125F, 1.25F, 1.375F, 1.5F}; + // Every column has to differ from scan 1's, or a slab-placement error in that + // column cannot be detected across the two slabs. The phase group set is kept + // identical because the ensemble arrays are shared by every stacked scan; only + // the per-point phase VALUES differ, using the reserved unindexed value 0. + scan.phase = {1, 0, 1, 0}; + scan.x = {1.0F, 1.25F, 1.0F, 1.25F}; + scan.y = {1.0F, 1.0F, 1.5F, 1.5F}; + return scan; +} + +//------------------------------------------------------------------------------ +// Fixture C: two scans, 2 x 2 each, every point HEXAGONAL. A shift applied to +// the wrong slab, or applied more than once to slab 0, changes values that are +// pinned exactly here. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureCScan1() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 2; + scan.yCells = 2; + scan.xStep = 0.5F; + scan.yStep = 0.25F; + scan.phases = {k_HexPhase}; + scan.phase = {1, 1, 1, 1}; + scan.bandContrast = {30, 31, 32, 33}; + scan.bandSlope = {40, 41, 42, 43}; + scan.bands = {1, 2, 3, 4}; + scan.error = {0, 0, 0, 0}; + scan.euler = {0.25F, 0.5F, 0.1F, 0.75F, 1.0F, 0.34F, 1.25F, 1.5F, 0.25F, 1.75F, 2.0F, 0.5F}; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F}; + scan.x = {0.0F, 0.5F, 0.0F, 0.5F}; + scan.y = {0.0F, 0.0F, 0.25F, 0.25F}; + return scan; +} + +ScanSpec MakeFixtureCScan2() +{ + ScanSpec scan = MakeFixtureCScan1(); + scan.name = "2"; + scan.bandContrast = {130, 131, 132, 133}; + scan.bandSlope = {140, 141, 142, 143}; + scan.bands = {5, 6, 7, 8}; + scan.error = {1, 1, 1, 1}; + scan.euler = {2.25F, 2.5F, 0.09F, 2.75F, 3.0F, 0.22F, 3.25F, 3.5F, 0.75F, 3.75F, 4.0F, 0.35F}; + scan.mad = {1.125F, 1.25F, 1.375F, 1.5F}; + return scan; +} + +// Fixture C expected Euler across both slabs with the alignment ON. Every phi2 is +// float32(double(phi2) + 30*(pi/180)); 0.1F/0.34F in slab 0 and 0.09F/0.22F/0.35F +// in slab 1 separate a double intermediate from a float32 one, while 0.25F, 0.5F +// and 0.75F round identically under either. +const std::vector k_FixtureCEulerAligned = { + // slab 0 (scan "1") + 0.25F, + 0.5F, + 0.6235987544059753F, + 0.75F, + 1.0F, + 0.8635987639427185F, + 1.25F, + 1.5F, + 0.7735987901687622F, + 1.75F, + 2.0F, + 1.0235987901687622F, + // slab 1 (scan "2") + 2.25F, + 2.5F, + 0.6135987639427185F, + 2.75F, + 3.0F, + 0.7435987591743469F, + 3.25F, + 3.5F, + 1.2735987901687622F, + 3.75F, + 4.0F, + 0.8735987544059753F, +}; + +//------------------------------------------------------------------------------ +Arguments MakeArgs(const fs::path& inputFile, const std::list& scanNames, float32 zSpacing = 1.0F, bool hexAlignment = true, bool convertPhaseToInt32 = true, bool readPatternData = false, + uint32 stackingOrder = RefFrameZDir::k_LowtoHigh) +{ + Arguments args; + const OEMEbsdScanSelectionParameter::ValueType scanSelections = {inputFile, stackingOrder, scanNames}; + args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); + args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(zSpacing)); + args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0F))); + args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(readPatternData)); + args.insertOrAssign(ReadH5OinaDataFilter::k_EdaxHexagonalAlignment_Key, std::make_any(hexAlignment)); + args.insertOrAssign(ReadH5OinaDataFilter::k_ConvertPhaseToInt32_Key, std::make_any(convertPhaseToInt32)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(k_ImageGeomPath)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellAttributeMatrixName)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleAttributeMatrixName)); + return args; +} + +//------------------------------------------------------------------------------ +template +void CompareArrayValues(const DataStructure& dataStructure, const DataPath& arrayPath, const std::vector& expected) +{ + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(arrayPath)); + const auto& dataArrayRef = dataStructure.getDataRefAs>(arrayPath); + REQUIRE(dataArrayRef.getSize() == expected.size()); + for(usize i = 0; i < expected.size(); i++) + { + INFO(fmt::format("Array '{}' index {}", arrayPath.toString(), i)); + REQUIRE(dataArrayRef[i] == expected[i]); + } +} + +//------------------------------------------------------------------------------ +void CompareStringArrayValues(const DataStructure& dataStructure, const DataPath& arrayPath, const std::vector& expected) +{ + REQUIRE_NOTHROW(dataStructure.getDataRefAs(arrayPath)); + const auto& stringArrayRef = dataStructure.getDataRefAs(arrayPath); + REQUIRE(stringArrayRef.getNumberOfTuples() == expected.size()); + for(usize i = 0; i < expected.size(); i++) + { + INFO(fmt::format("Array '{}' index {}", arrayPath.toString(), i)); + REQUIRE(stringArrayRef[i] == expected[i]); + } +} + +//------------------------------------------------------------------------------ +// Reads a dataset straight out of a .h5oina with H5Lite, i.e. from the file +// bytes, without involving H5OINAReader. Used by the real-file Class 2 readback. +//------------------------------------------------------------------------------ +template +std::vector ReadRawDataset(const fs::path& filePath, const std::string& datasetPath) +{ + // H5Utilities::closeFile() takes its argument by non-const reference and assigns -1 to + // it, so this identifier cannot be const. + hid_t fileId = H5Utilities::openFile(filePath.string(), true); + REQUIRE(fileId >= 0); + std::vector data; + const herr_t err = H5Lite::readVectorDataset(fileId, datasetPath, data); + REQUIRE(err >= 0); + REQUIRE(H5Utilities::closeFile(fileId) >= 0); + return data; +} +} // namespace + +//------------------------------------------------------------------------------ +// Class 1 (analytical) + Class 4 (invariant) oracle over Fixture A with the +// shipped parameter defaults (hexagonal alignment ON, Phase widened to int32). +// Every expected value below is derived from the fixture spec above. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "H5Oina_Test_Data.tar.gz", "H5Oina_Test_Data"); + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_oracle.h5oina", {MakeFixtureAScan()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}, 0.75F); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // --- Geometry: dims and spacing come from the header; the z extent is the + // --- number of selected scans, the z spacing is the parameter, the origin is + // --- the parameter, and the units are the filter's hard-coded Micrometer. --- + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(3, 2, 1)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(0.25F, 0.5F, 0.75F)); + REQUIRE(imageGeom.getOrigin() == FloatVec3(0.0F, 0.0F, 0.0F)); + REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); + + // --- Cell arrays: eight verbatim copies plus the aligned Euler array. ------- + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), {10, 20, 30, 40, 50, 60}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), {11, 21, 31, 41, 51, 61}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), {0, 1, 0, 2, 0, 3}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), {0.0F, 0.25F, 0.5F, 0.0F, 0.25F, 0.5F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), {0.0F, 0.0F, 0.0F, 0.5F, 0.5F, 0.5F}); + // Phase is widened from the file's uint8 to int32 by default, values unchanged + // (including the unindexed 0 at point 2). + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureAEulerAligned); + + // --- Ensemble arrays: slot 0 is the reserved Invalid Phase, then one slot per + // --- phase group in the file. Laue 9 -> Hexagonal_High (0), Laue 11 -> + // --- Cubic_High (1). The file's radian lattice angles are stored as degrees, + // --- matching the .ang and .ctf importers: 90, 90, 120 for the hexagonal + // --- phase and 90, 90, 90 for the cubic one. ------------------------------- + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures), + {ebsdlib::CrystalStructure::UnknownCrystalStructure, ebsdlib::CrystalStructure::Hexagonal_High, ebsdlib::CrystalStructure::Cubic_High}); + CompareStringArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::MaterialName), {"Invalid Phase", "Hex Phase A", "Cubic Phase B"}); + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::LatticeConstants), + { + 0.0F, 0.0F, 0.0F, 0.0F, 0.0F, 0.0F, // slot 0: Invalid Phase + 2.5F, 2.5F, 4.0F, 90.0F, 90.0F, 120.0F, // slot 1: Hex Phase A + 3.5F, 3.5F, 3.5F, 90.0F, 90.0F, 90.0F, // slot 2: Cubic Phase B + }); + + // Class 4 invariant: the ensemble matrix always carries exactly one more tuple + // than the file has phase groups. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures))); + REQUIRE(dataStructure.getDataRefAs(k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures)).getNumberOfTuples() == 3); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The two value-transform options across their full 2 x 2 grid. Class 4 +// invariants: the Phase values never change with either option, and the +// hexagonal shift never reaches a cubic or unindexed point. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Conversion Option Combinations", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_options.h5oina", {MakeFixtureAScan()}); + + const auto hexAlignment = GENERATE(true, false); + const auto convertPhaseToInt32 = GENERATE(true, false); - // Read Exemplar DREAM3D File - auto exemplarFilePath = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.dream3d", unit_test::k_TestFilesDir)); - DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + DYNAMIC_SECTION("hexAlignment=" << hexAlignment << " convertPhaseToInt32=" << convertPhaseToInt32) + { + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}, 1.0F, hexAlignment, convertPhaseToInt32); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), hexAlignment ? k_FixtureAEulerAligned : k_FixtureAEulerVerbatim); + + if(convertPhaseToInt32) + { + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + } + else + { + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + } + + // The remaining columns are verbatim regardless of either option. + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +//------------------------------------------------------------------------------ +// Two scans stacked into one geometry. Fixture B is cubic-only, so every cell +// array -- Euler included -- must be a verbatim copy landing in its own tuple +// slab: scan "1" in tuples [0, 4) and scan "2" in tuples [4, 8). +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Multi-Scan Slab Placement", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_multiscan.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); - // Instantiate the filter, a DataStructure object and an Arguments Object ReadH5OinaDataFilter filter; DataStructure dataStructure; - Arguments args; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 0.5F); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - auto h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.h5oina", unit_test::k_TestFilesDir)); - OEMEbsdScanSelectionParameter::ValueType scanSelections = {h5TestFile, ebsdlib::RefFrameZDir::LowtoHigh, {k_ScanName}}; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(2, 2, 2)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(0.25F, 0.5F, 0.5F)); - // Create default Parameters for the filter. - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0f))); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(DataPath({ImageGeom::k_TypeName}))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellData)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleData)); + // 24 Euler values: scan "1"'s 12 then scan "2"'s 12, each verbatim. + CompareArrayValues( + dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), + { + 0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F, 2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F, + }); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), {10, 11, 12, 13, 110, 111, 112, 113}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), {20, 21, 22, 23, 120, 121, 122, 123}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6, 7, 8}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), {0, 1, 2, 3, 4, 5, 6, 7}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 1.125F, 1.25F, 1.375F, 1.5F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 1, 1, 1, 1, 0, 1, 0}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), {0.0F, 0.25F, 0.0F, 0.25F, 1.0F, 1.25F, 1.0F, 1.25F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), {0.0F, 0.0F, 0.5F, 0.5F, 1.0F, 1.0F, 1.5F, 1.5F}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Two hexagonal scans. Every point takes the alignment, so the pinned values +// establish that the shift reaches scan "2"'s slab and is applied to scan "1"'s +// slab exactly once. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Multi-Scan Hexagonal Alignment", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_multiscan_hex.h5oina", {MakeFixtureCScan1(), MakeFixtureCScan2()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 2.0F); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(2, 2, 2)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getSpacing() == FloatVec3(0.5F, 0.25F, 2.0F)); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureCEulerAligned); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The pinned reader ignores the Format Version entirely -- its four version +// branches are empty and the dataset itself is optional. All three variants must +// produce the same output. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Format Version Variants", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const auto variant = GENERATE(as{}, "5.0", "2.0", "absent"); + + DYNAMIC_SECTION("Format Version " << variant) + { + const fs::path inputFile = WriteH5OinaFixture(fmt::format("read_h5oina_vv_version_{}.h5oina", variant), {MakeFixtureAScan()}, variant != "absent", variant); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(3, 2, 1)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureAEulerAligned); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +//------------------------------------------------------------------------------ +// Value-add preflight rejections that need no file at all. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Parameter Rejections", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_params.h5oina", {MakeFixtureAScan()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeArgs(inputFile, {"1"}); + int32 expectedCode = 0; + + SECTION("Non-positive Z Spacing (-9580)") + { + args = MakeArgs(inputFile, {"1"}, 0.0F); + expectedCode = -9580; + } + SECTION("No Scan Names Selected (-9581)") + { + args = MakeArgs(inputFile, {}); + expectedCode = -9581; + } + SECTION("Pattern Import Not Supported (-9583)") + { + args = MakeArgs(inputFile, {"1"}, 1.0F, true, true, true); + expectedCode = -9583; + } + SECTION("Non-finite Z Spacing (-9580)") + { + args = MakeArgs(inputFile, {"1"}, std::numeric_limits::quiet_NaN()); + expectedCode = -9580; + } - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == expectedCode); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); - const auto& imageGeom = dataStructure.getDataRefAs(DataPath({ImageGeom::k_TypeName})); - const auto& exemplarImageGeom = exemplarDataStructure.getDataRefAs(DataPath({k_ExemplarDataContainer})); - REQUIRE(imageGeom.getDimensions() == exemplarImageGeom.getDimensions()); - REQUIRE(imageGeom.getSpacing() == exemplarImageGeom.getSpacing()); - REQUIRE(imageGeom.getOrigin() == exemplarImageGeom.getOrigin()); - REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: a header whose cell counts are not usable is rejected before +// a zero-sized (or absurdly-sized) geometry is created. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Cell Counts rejected (-9584)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + std::string fileName; + + SECTION("Zero X Cells") + { + scan.xCells = 0; + fileName = "read_h5oina_vv_zero_xcells.h5oina"; + } + SECTION("Zero Y Cells") + { + scan.yCells = 0; + fileName = "read_h5oina_vv_zero_ycells.h5oina"; + } + SECTION("Negative X Cells") + { + scan.xCells = -3; + fileName = "read_h5oina_vv_negative_xcells.h5oina"; + } + + const fs::path inputFile = WriteH5OinaFixture(fileName, {scan}); - UnitTest::CompareExemplarToGeneratedData(dataStructure, exemplarDataStructure, DataPath({ImageGeom::k_TypeName, k_CellData}), k_ExemplarDataContainer); + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); - const DataPath cellEnsemblePath({ImageGeom::k_TypeName, k_CellEnsembleData}); - const DataPath exemplarCellEnsemblePath({k_ExemplarDataContainer, k_CellEnsembleData}); - const auto& crystalStructures = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - const auto& crystalStructuresExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - UnitTest::CompareDataArrays(crystalStructures, crystalStructuresExemplar); - const auto& latticeConstants = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::LatticeConstants)); - const auto& latticeConstantsExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::LatticeConstants)); - UnitTest::CompareDataArrays(latticeConstants, latticeConstantsExemplar); - const auto& materialName = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::MaterialName)); - const auto& materialNameExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::MaterialName)); - UnitTest::CompareStringArrays(materialNameExemplar, materialName); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9584); UnitTest::CheckArraysInheritTupleDims(dataStructure); } -TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: InValid Filter Execution", "[OrientationAnalysis][ReadH5OinaDataFilter]") +//------------------------------------------------------------------------------ +// Value-add guard: the geometry is built from the FIRST selected scan's header, +// so every other selected scan has to agree with it. A second scan with a +// different grid is rejected at preflight rather than driving an out-of-bounds +// read of the reader's buffers at execute. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Header Mismatch rejected (-9585)", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_ImportH5Data.tar.gz", "6_6_ImportH5Data"); + ScanSpec scan2 = MakeFixtureBScan2(); + + SECTION("Differing cell counts") + { + scan2.xCells = 3; + scan2.yCells = 3; + scan2.phase.assign(9, 1); + scan2.bandContrast.assign(9, 7); + scan2.bandSlope.assign(9, 7); + scan2.bands.assign(9, 7); + scan2.error.assign(9, 0); + scan2.euler.assign(27, 0.5F); + scan2.mad.assign(9, 0.25F); + scan2.x.assign(9, 0.0F); + scan2.y.assign(9, 0.0F); + } + SECTION("Differing step size") + { + scan2.xStep = 0.75F; + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_scan_mismatch.h5oina", {MakeFixtureBScan1(), scan2}); - // Instantiate the filter, a DataStructure object and an Arguments Object ReadH5OinaDataFilter filter; DataStructure dataStructure; - Arguments args; - args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0f))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(DataPath({ImageGeom::k_TypeName}))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellData)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleData)); + const Arguments args = MakeArgs(inputFile, {"1", "2"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9585); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: every selected scan name is checked against the file at +// preflight, not just the first one. Without this a bad second name fails +// part-way through execute with half the arrays already populated. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Missing Scan Name rejected (-9586)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_scan.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); + + std::list scanNames; + SECTION("Missing first scan") + { + scanNames = {"9"}; + } + SECTION("Missing second scan") + { + scanNames = {"1", "9"}; + } - auto h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/FirstLook AB Site 1 Map Data 4.h5oina", unit_test::k_TestFilesDir)); - OEMEbsdScanSelectionParameter::ValueType scanSelections = {h5TestFile, ebsdlib::RefFrameZDir::LowtoHigh, {k_ScanName}}; + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, scanNames); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9586); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} - SECTION("Invalid Z Spacing") +//------------------------------------------------------------------------------ +// Value-add guard: phase indices come from the HDF5 phase GROUP NAMES, while the +// ensemble matrix is sized from the phase COUNT. A file whose groups are not +// numbered 1..N is rejected instead of writing past the end of the ensemble +// arrays. +// +// The ensemble fill in the shared IEbsdOemReader::readData runs once per SELECTED +// scan, so the check has to run for every selected scan and not only the first. +// The "Later selected scan" section is the same defect shape moved into scan "2": +// with the check applied to the first scan only, that file passes preflight and +// crashes the process at execute. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Phase Index Out Of Range rejected (-9587)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + std::vector scans; + std::list scanNames; + std::string fileName; + + SECTION("First selected scan") { - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(0.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + ScanSpec scan = MakeFixtureBScan1(); + scan.phases[0].groupName = "7"; // one phase, but it claims index 7 + scans = {scan}; + scanNames = {"1"}; + fileName = "read_h5oina_vv_phase_index.h5oina"; } - SECTION("No Scan Names Selected") + SECTION("Later selected scan") { - scanSelections.scanNames.clear(); - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + ScanSpec scan2 = MakeFixtureBScan2(); + scan2.phases[0].groupName = "7"; + scans = {MakeFixtureBScan1(), scan2}; + scanNames = {"1", "2"}; + fileName = "read_h5oina_vv_phase_index_scan2.h5oina"; } - SECTION("Invalid h5 file type (incompatible manufacturer)") + + const fs::path inputFile = WriteH5OinaFixture(fileName, scans); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, scanNames); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9587); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// EbsdLib guard: phase group names must be strict positive integers. The +// pre-correction reader passed the name directly to std::stoi(), which threw an +// exception for this file instead of returning a user-facing reader error. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Phase Group Name rejected (-9582)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureBScan1(); + scan.phases[0].groupName = "phase-one"; + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_invalid_phase_group_name.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9582); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find("phase-one") != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: the ensemble arrays are sized from the FIRST selected scan's +// phase count, and every selected scan fills them from its own phase groups. A +// later scan that declares more phase groups than the first therefore writes past +// the end of all three ensemble arrays, and a later scan that declares fewer +// silently leaves the first scan's phases in place under different definitions. +// Neither is a malformed file: different scans of a real multi-scan AZtec export +// may legitimately declare different phase lists. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Phase Count Mismatch rejected (-9589)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan1 = MakeFixtureBScan1(); + ScanSpec scan2 = MakeFixtureBScan2(); + std::string fileName; + + SECTION("Later scan declares more phases") { - h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/FirstLook AB Site 1 Map Data 4.h5oina", unit_test::k_TestFilesDir)); - scanSelections.inputFilePath = h5TestFile; - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + // Groups "1", "2" and "3" against the first scan's single group "1". + scan2.phases = {k_CubicPhase, k_HexPhase, k_CubicPhase}; + fileName = "read_h5oina_vv_phase_count_more.h5oina"; } + SECTION("Later scan declares fewer phases") + { + scan1.phases = {k_CubicPhase, k_HexPhase}; + fileName = "read_h5oina_vv_phase_count_fewer.h5oina"; + } + + const fs::path inputFile = WriteH5OinaFixture(fileName, {scan1, scan2}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9589); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// A multi-scan import creates one 3D microstructure with one shared Ensemble +// Attribute Matrix. Therefore, the phase definition at each index must be the +// same in every selected scan. A mismatch cannot be represented correctly. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Phase Definition Mismatch rejected (-9590)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan2 = MakeFixtureBScan2(); + std::string mismatchField; + + SECTION("Material name") + { + scan2.phases[0].name = "Different cubic phase"; + mismatchField = "material name"; + } + SECTION("Laue group") + { + scan2.phases[0].laueGroup = k_LaueHexagonalHigh; + mismatchField = "Laue group"; + } + SECTION("Space group") + { + scan2.phases[0].spaceGroup = 194; + mismatchField = "space group"; + } + SECTION("Lattice dimensions") + { + scan2.phases[0].latticeDimensions[0] = 4.25F; + mismatchField = "lattice constants"; + } + SECTION("Lattice angles") + { + scan2.phases[0].latticeAngles[2] = 2.0943952F; + mismatchField = "lattice constants"; + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_definition_mismatch.h5oina", {MakeFixtureBScan1(), scan2}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9590); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find("'2'") != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(mismatchField) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The X and Y step values become the Image Geometry spacing. They must be finite +// and positive before the geometry action is created. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Scan Spacing rejected (-9591)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + + SECTION("Zero X Step") + { + scan.xStep = 0.0F; + } + SECTION("Negative Y Step") + { + scan.yStep = -0.5F; + } + SECTION("Non-finite X Step") + { + scan.xStep = std::numeric_limits::quiet_NaN(); + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_invalid_spacing.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9591); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: EbsdLib sizes its data buffers to the ACTUAL dataset extent +// while the geometry is sized from the header's cell counts. A file whose Data +// datasets are shorter than X Cells * Y Cells is rejected before the copy spans +// past the end of those buffers. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Dataset Extent Mismatch rejected (-34971)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + // Header declares a 4 x 4 = 16 point scan; the Data datasets hold 8 points. + ScanSpec scan = MakeFixtureBScan1(); + scan.xCells = 4; + scan.yCells = 4; + scan.phase.assign(8, 1); + scan.bandContrast.assign(8, 5); + scan.bandSlope.assign(8, 6); + scan.bands.assign(8, 7); + scan.error.assign(8, 0); + scan.euler.assign(24, 0.25F); + scan.mad.assign(8, 0.125F); + scan.x.assign(8, 0.0F); + scan.y.assign(8, 0.0F); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_short_data.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -34971); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: a Phase byte larger than the file's phase count would index +// past the end of the CrystalStructures array in the alignment loop. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Out-of-Range Phase Value rejected (-34972)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + // One phase in the header, so the valid Phase range is [0, 1]; point 2 carries 5. + ScanSpec scan = MakeFixtureBScan1(); + scan.phase = {1, 1, 5, 1}; + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_range.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -34972); + REQUIRE(executeResult.result.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Error propagation from the EbsdLib boundary. A Data group missing one +// of the nine required datasets is fatal inside H5OINAReader::readData; the +// filter surfaces it as -8970 with the reader's own code and message attached. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Data Column (-8970)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + scan.omitBandsDataset = true; + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_bands.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + // The Data datasets are only read at execute, so preflight (header only) succeeds. + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -8970); + // The message names the scan and the file so the user can find the problem. + REQUIRE(executeResult.result.errors()[0].message.find("'1'") != std::string::npos); + REQUIRE(executeResult.result.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Error propagation from the EbsdLib boundary at header-read time: a +// phase group missing its Lattice Angles dataset is reported through -9582, and +// the message carries the file path and scan name. +// +// The header of every selected scan is read at preflight, from two different +// return sites: the first selected scan's read and the per-scan loop's read. Both +// are pinned here, and each section asserts the scan name its own site injects. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + std::vector scans; + std::list scanNames; + std::string fileName; + std::string expectedScanName; + + SECTION("First selected scan") + { + ScanSpec scan = MakeFixtureAScan(); + scan.phases[0].omitLatticeAngles = true; + scans = {scan}; + scanNames = {"1"}; + fileName = "read_h5oina_vv_missing_angles.h5oina"; + expectedScanName = "'1'"; + } + SECTION("Later selected scan") + { + ScanSpec scan2 = MakeFixtureBScan2(); + scan2.phases[0].omitLatticeAngles = true; + scans = {MakeFixtureBScan1(), scan2}; + scanNames = {"1", "2"}; + fileName = "read_h5oina_vv_missing_angles_scan2.h5oina"; + expectedScanName = "'2'"; + } + + const fs::path inputFile = WriteH5OinaFixture(fileName, scans); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, scanNames); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9582); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(expectedScanName) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The stacking order carried by the scan-selection parameter chooses which end of +// the scan list lands in tuple slab 0. Low-to-High stacks the scans in the order +// they are listed; High-to-Low stacks them in the reverse of that order, so the +// LAST selected scan occupies slab 0. +// +// Fixture B's two scans are disjoint in every column, so each order is pinned by +// values that the other order cannot produce. The geometry is identical under both +// orders -- only the slab assignment changes. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Stacking Order", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_stacking.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); + + const std::vector k_Scan1Euler = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F}; + const std::vector k_Scan2Euler = {2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F}; + + uint32 stackingOrder = RefFrameZDir::k_LowtoHigh; + std::vector expectedEuler; + std::vector expectedBandContrast; + std::vector expectedPhase; + + SECTION("Low to High stacks the scans in list order") + { + stackingOrder = RefFrameZDir::k_LowtoHigh; + expectedEuler = k_Scan1Euler; + expectedEuler.insert(expectedEuler.end(), k_Scan2Euler.cbegin(), k_Scan2Euler.cend()); + expectedBandContrast = {10, 11, 12, 13, 110, 111, 112, 113}; + expectedPhase = {1, 1, 1, 1, 1, 0, 1, 0}; + } + SECTION("High to Low stacks the scans in reverse list order") + { + stackingOrder = RefFrameZDir::k_HightoLow; + expectedEuler = k_Scan2Euler; + expectedEuler.insert(expectedEuler.end(), k_Scan1Euler.cbegin(), k_Scan1Euler.cend()); + expectedBandContrast = {110, 111, 112, 113, 10, 11, 12, 13}; + expectedPhase = {1, 0, 1, 0, 1, 1, 1, 1}; + } + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, stackingOrder); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.warnings().empty()); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(2, 2, 2)); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), expectedEuler); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), expectedBandContrast); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), expectedPhase); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Class 2 independent readback against the archived production AZtec file. +// +// The expected cell values are read straight out of the .h5oina's own Data +// datasets with H5Lite, bypassing H5OINAReader, and the ensemble expectations are +// derived from the file's phase group (Laue 11 -> Cubic_High; lattice angles +// pi/2 radians -> 90 degrees). The archive's H5Oina_Test_Data.dream3d exemplar is +// deliberately NOT consulted -- it was written by this filter and would only pin +// the filter against itself. +// +// This file is a single 25 x 25 cubic scan, so it exercises the single-scan copy +// plumbing, the Phase widening (its Phase column contains both 0 and 1) and the +// ensemble fill on production data. It cannot exercise the hexagonal alignment or +// the multi-scan slab offsets; the toy fixtures above carry those. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Real AZtec File Readback", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "H5Oina_Test_Data.tar.gz", "H5Oina_Test_Data"); + + const fs::path inputFile = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.h5oina", unit_test::k_TestFilesDir)); + REQUIRE(fs::exists(inputFile)); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // Geometry: X Cells = Y Cells = 25, X Step = Y Step = 4.0, one scan selected. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(25, 25, 1)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(4.0F, 4.0F, 1.0F)); + REQUIRE(imageGeom.getOrigin() == FloatVec3(0.0F, 0.0F, 0.0F)); + REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); + + // Every cell array is a verbatim copy of the corresponding Data dataset: this + // scan's only phase is Laue 11 (Cubic_High), so no point takes the alignment. + const std::string dataPrefix = fmt::format("/1/{}/{}/", ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::BandContrast)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::BandSlope)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Bands)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Error)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Euler)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), + ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::MeanAngularDeviation)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::X)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Y)); + + // Phase is the file's uint8 column widened to int32. + { + const std::vector rawPhase = ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Phase); + std::vector expectedPhase(rawPhase.size()); + for(usize i = 0; i < rawPhase.size(); i++) + { + expectedPhase[i] = static_cast(rawPhase[i]); + } + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), expectedPhase); + } + + // Ensemble: slot 0 is the reserved Invalid Phase; slot 1 is the file's single + // "Titanium cubic" phase, Laue group 11 -> Cubic_High, lattice dimensions + // 3.192 and lattice angles pi/2 radians stored as 90 degrees. + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures), + {ebsdlib::CrystalStructure::UnknownCrystalStructure, ebsdlib::CrystalStructure::Cubic_High}); + CompareStringArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::MaterialName), {"Invalid Phase", "Titanium cubic"}); + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::LatticeConstants), + { + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 3.192F, + 3.192F, + 3.192F, + 90.0F, + 90.0F, + 90.0F, + }); UnitTest::CheckArraysInheritTupleDims(dataStructure); } diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..dc73f17233 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -0,0 +1,186 @@ +# V&V Report: ReadH5OinaDataFilter + +| | | +|---|---| +| Plugin | OrientationAnalysis | +| SIMPLNX UUID | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| DREAM.3D 6.5.171 equivalent | **None.** | +| Verified commit | ** | +| Status | READY FOR REVIEW | +| Sign-off | Pending second-engineer PR review. | + +## At a glance + +| Aspect | Current state | +|---|---| +| Algorithm Relationship | **New filter, no legacy equivalent.** `ReadH5OinaDataFilter` was added in SIMPLNX and first shipped in DREAM3D-NX 7.0.0. | +| Oracle (confirmed) | **Class 1 analytical + Class 4 invariant, with Class 2 independent h5py readback.** The tests create small H5OINA inputs and compare all imported values with independently derived values. | +| Code paths enumerated | **26 of 33 paths exercised.** The uncovered paths require file-permission changes, file replacement during execution, cancel injection, or HDF5 objects that the fixture writer cannot create. | +| Tests today | **20 test cases and 9,943 assertions through ctest.** The tests cover data conversion, multi-scan stacking, phase definitions, malformed inputs, error propagation, and a real AZtec file. | +| Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained as input only.** The vendor `.h5oina` file is used. The generated `.dream3d` output was retired as a circular oracle. | +| Legacy comparison | **Not applicable.** DREAM.3D 6.5.171 has no H5OINA importer. The independent h5py readback replaces the legacy comparison. | +| Bug flags | **Thirteen bugs resolved:** `ReadH5OinaDataFilter-D1` through `ReadH5OinaDataFilter-D13`. The affected released versions are DREAM3D-NX 7.0.0 through 7.4.1. | +| V&V phase | **COMPLETE** | + +## Summary + +`ReadH5OinaDataFilter` imports one or more Oxford Instruments AZtec H5OINA scans into one Image Geometry. Verification uses inline Class 1 analytical data, Class 4 invariants, and an independent Class 2 h5py readback of a vendor file. The V&V found and fixed thirteen defects in the filter and EbsdLib H5OINA reader. + +## Dependency state + +- EbsdLib version 3.1.2 +- Intended DREAM3D-NX release: 7.5.0 + +## Algorithm Relationship + +**New filter, no legacy equivalent.** + +*Evidence:* DREAM.3D 6.5.171 contains no H5OINA importer. The filter was added to SIMPLNX by PR #700 in commit `a51dd5f3d` and has no SIMPL conversion function or legacy UUID mapping. + +## Oracle + +*Class:* **1 (Analytical) + 4 (Invariant)**, with **2 (Independent readback)** for the vendor file. + +*Applied:* The tests write small H5OINA files from explicit fixture specifications. Expected geometry, cell arrays, ensemble arrays, unit conversions, stacking order, and rejection results are derived from those specifications. A separate h5py script reads the vendor file and derives its expected imported values without EbsdLib or SIMPLNX. + +*Encoded:* `test/ReadH5OinaDataTest.cpp` contains 20 test cases. The archived `h5oina_oracle.py` script reproduces its Class 1 values and the vendor-file readback. + +*Second-engineer review:* Pending PR review. + +## Bugs found and fixed + +This branch fixes all defects in this table. The fixes are intended for DREAM3D-NX 7.5.0. EbsdLib defects require EbsdLib 3.1.2. + +| Deviation | Defect | Affected released versions | Resolution in this branch | +|---|---|---|---| +| `ReadH5OinaDataFilter-D1` | The hexagonal alignment added 30 radians instead of 30 degrees to `phi2`. | DREAM3D-NX 7.0.0 through 7.4.1. | The filter adds 30 degrees expressed in radians with a double-precision intermediate. | +| `ReadH5OinaDataFilter-D2` | The Euler destination offset used tuples instead of elements. Later scans overwrote part of an earlier scan. | DREAM3D-NX 7.0.0 through 7.4.1. | The Euler copy multiplies the tuple offset by three components. | +| `ReadH5OinaDataFilter-D3` | Hexagonal alignment repeatedly modified the first scan and did not modify later scans. | DREAM3D-NX 7.0.0 through 7.4.1. | The alignment loop uses the current scan slab. | +| `ReadH5OinaDataFilter-D4` | Pattern import was exposed but could not succeed. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight reports that H5OINA pattern import is not supported and gives an alternative. | +| `ReadH5OinaDataFilter-D5` | The gamma lattice angle copied the beta angle. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib copies the third lattice angle into the gamma slot. | +| `ReadH5OinaDataFilter-D6` | H5OINA lattice angles were imported in radians while other EBSD readers use degrees. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib converts the lattice angles to degrees. | +| `ReadH5OinaDataFilter-D7` | A phase without a required lattice dataset caused invalid access and a process crash. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates the required phase datasets before it reads their values. | +| `ReadH5OinaDataFilter-D8` | Multi-scan phase groups could exceed the ensemble bounds or describe different phases at the same index. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight validates every phase index, phase count, and phase definition in every selected scan. | +| `ReadH5OinaDataFilter-D9` | The Stacking Order setting was accepted but ignored. | DREAM3D-NX 7.0.0 through 7.4.1. | The scan order now determines which scan is placed at Z = 0. | +| `ReadH5OinaDataFilter-D10` | H5OINA reader error messages were constructed but stored as empty strings. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib stores and forwards the complete error message. | +| `ReadH5OinaDataFilter-D11` | The reader did not validate both cell counts and returned an error code that differed from its stored code. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates rows and columns and returns the stored error code. | +| `ReadH5OinaDataFilter-D12` | A non-numeric phase-group name caused `std::stoi()` to throw and left an HDF5 group open. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates the complete group name before it opens the group. | +| `ReadH5OinaDataFilter-D13` | Non-positive or non-finite scan spacing could create an invalid Image Geometry. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight requires finite, positive X, Y, and Z spacing. | + +## Code path coverage + +26 of 33 paths are exercised. + +Source: `Filters/ReadH5OinaDataFilter.cpp`, `Filters/Algorithms/ReadH5OinaData.cpp`, and the shared `utilities/IEbsdOemReader.hpp` read and ensemble-fill path. + +| # | Phase | Path | Test case | +|---|---|---|---| +| 1 | Preflight | Reject non-positive or non-finite Z spacing (`-9580`) | `Parameter Rejections` | +| 2 | Preflight | Reject an empty scan selection (`-9581`) | `Parameter Rejections` | +| 3 | Preflight | Reject pattern import (`-9583`) | `Parameter Rejections` | +| 4 | Preflight | Fail while listing scans (`-9582`) | *Not directly tested. Requires a present file that cannot be read.* | +| 5 | Preflight | Reject a selected scan that is not in the file (`-9586`) | `Missing Scan Name rejected` | +| 6 | Preflight | Fail while reading the first scan header (`-9582`) | `Missing Lattice Angles`; `Invalid Phase Group Name` | +| 7 | Preflight | Reject cell counts below one (`-9584`) | `Invalid Cell Counts rejected` | +| 8 | Preflight | Reject non-positive or non-finite X or Y spacing (`-9591`) | `Invalid Scan Spacing rejected` | +| 9 | Preflight | Reject a phase index outside the ensemble bounds (`-9587`) | `Phase Index Out Of Range rejected` | +| 10 | Preflight | Fail while reading a later scan header (`-9582`) | `Missing Lattice Angles` | +| 11 | Preflight | Reject a later scan with a different grid (`-9585`) | `Scan Header Mismatch rejected` | +| 12 | Preflight | Reject a later scan with a different phase count (`-9589`) | `Scan Phase Count Mismatch rejected` | +| 13 | Preflight | Reject a later scan with different phase definitions (`-9590`) | `Scan Phase Definition Mismatch rejected` | +| 14 | Preflight | Create geometry dimensions, spacing, origin, and cell matrix | Class 1 oracle; multi-scan tests | +| 15 | Preflight | Create the ensemble matrix and three ensemble arrays | Class 1 oracle | +| 16 | Preflight | Create nine cell arrays and select the Phase type | `Conversion Option Combinations` | +| 17 | Preflight | Generate display-only scan and phase information | *Not directly tested. The values are shown only in preflight.* | +| 18 | Execute | Propagate an EbsdLib data-read failure (`-8970`) | `Missing Data Column` | +| 19 | Execute | Reject an empty phase vector (`-8971`) | *Not directly tested. H5OINAReader rejects this file during preflight first.* | +| 20 | Execute | Initialize ensemble tuple 0 and fill phase tuples | Class 1 oracle; real-file readback | +| 21 | Execute | Apply Low-to-High or High-to-Low stacking | `Stacking Order` | +| 22 | Copy | Fail when the file cannot be reopened (`-34971`) | *Not directly tested. Requires file replacement during execution.* | +| 23 | Copy | Skip an extent probe that cannot inspect a dataset | *Not directly tested. The reader rejects a missing required dataset first.* | +| 24 | Copy | Reject a dataset extent that differs from the header (`-34971`) | `Dataset Extent Mismatch rejected` | +| 25 | Copy | Copy the four `uint8` arrays into the current scan slab | Class 1 oracle; multi-scan placement | +| 26 | Copy | Copy Euler values with a three-component offset | `Multi-Scan Slab Placement` | +| 27 | Copy | Reject a Phase value outside the ensemble bounds (`-34972`) | `Out-of-Range Phase Value rejected` | +| 28 | Copy | Widen Phase to `int32` or retain `uint8` | `Conversion Option Combinations` | +| 29 | Copy | Copy MAD, X, and Y into the current scan slab | Class 1 oracle; multi-scan placement | +| 30 | Copy | Apply hexagonal alignment only to Hexagonal-High points in the current slab | Class 1 oracle; option sweep; multi-scan alignment | +| 31 | Execute/Copy | Return at the three cancellation checks | *Not directly tested. Requires cancel-signal injection.* | +| 32 | EbsdLib | Reject a noncanonical phase-group name (`-90034`, surfaced as `-9582`) | `Invalid Phase Group Name rejected` | +| 33 | EbsdLib | Reject an unreadable phase group, lattice dimensions, or Laue group | *Not directly tested. The fixture writer cannot create the unreadable-group case; the other required-dataset cases use the same checked path as lattice angles.* | + +## Test inventory + +| Test case | Status | Notes | +|---|---|---| +| `Class 1 Analytical Oracle` | new-for-V&V | Verifies geometry, all cell arrays, all ensemble arrays, alignment, and invariants. | +| `Conversion Option Combinations` | new-for-V&V | Covers the 2 x 2 alignment and Phase-type option matrix. | +| `Multi-Scan Slab Placement` | new-for-V&V | Verifies all scan slabs with disjoint values. | +| `Multi-Scan Hexagonal Alignment` | new-for-V&V | Verifies alignment on each scan slab. | +| `Format Version Variants` | new-for-V&V | Covers version 5.0, version 2.0, and no version dataset. | +| `Parameter Rejections` | new-for-V&V | Covers Z spacing, empty selection, and unsupported pattern import. | +| `Invalid Cell Counts rejected` | new-for-V&V | Covers zero and negative cell counts. | +| `Scan Header Mismatch rejected` | new-for-V&V | Covers grid dimensions and step-size differences. | +| `Missing Scan Name rejected` | new-for-V&V | Covers missing first and later scan names. | +| `Phase Index Out Of Range rejected` | new-for-V&V | Covers invalid phase indices in the first and later scans. | +| `Invalid Phase Group Name rejected` | new-for-V&V | Verifies that EbsdLib reports a non-numeric group name without throwing. | +| `Scan Phase Count Mismatch rejected` | new-for-V&V | Covers later scans with more or fewer phase groups. | +| `Scan Phase Definition Mismatch rejected` | new-for-V&V | Covers material, Laue, space-group, lattice-dimension, and lattice-angle differences. | +| `Invalid Scan Spacing rejected` | new-for-V&V | Covers zero, negative, and non-finite scan spacing. | +| `Dataset Extent Mismatch rejected` | new-for-V&V | Verifies header and dataset extent agreement. | +| `Out-of-Range Phase Value rejected` | new-for-V&V | Verifies Phase values before ensemble indexing. | +| `Missing Data Column` | new-for-V&V | Verifies EbsdLib execute error propagation. | +| `Missing Lattice Angles` | new-for-V&V | Verifies EbsdLib preflight error propagation for first and later scans. | +| `Stacking Order` | new-for-V&V | Verifies both scan orders. | +| `Real AZtec File Readback` | kept, modified | Uses the vendor H5OINA input and independent readback. The generated DREAM3D exemplar is not used. | +| `InValid Filter Execution` | retired | The old test used a missing file and did not verify the intended error. | + +All 20 test cases pass. The ctest run reports 9,943 assertions. + +## Test sensitivity verification + +Test sensitivity verification introduces one temporary defect at a time and confirms that the applicable test fails. + +Eleven temporary defects were evaluated. Each defect caused the expected V&V test to fail. + +| Temporary defect | Test that detected the defect | Result | +|---|---|---| +| Restore the 30-radian alignment value. | Class 1 oracle; option sweep; multi-scan alignment; format variants | Detected | +| Use a float32 intermediate for alignment. | Class 1 oracle; option sweep; multi-scan alignment; format variants | Detected | +| Use the tuple offset for the Euler element offset. | Multi-scan slab placement; multi-scan alignment | Detected | +| Apply alignment from tuple 0 for every scan. | Multi-scan alignment | Detected | +| Copy beta into the gamma lattice slot. | Class 1 oracle | Detected | +| Leave lattice angles in radians. | Class 1 oracle; real-file readback | Detected | +| Remove the dataset-extent validation. | Dataset extent mismatch | Detected | +| Pass an invalid phase-group name to `std::stoi()`. | Invalid phase group name | Detected | +| Disable phase-definition validation. | Scan phase definition mismatch | Detected | +| Disable spacing validation. | Parameter rejections; invalid scan spacing | Detected | +| Remove the file path from the Phase-range error. | Out-of-range Phase value | Detected | + +## Exemplar archive + +`H5Oina_Test_Data.tar.gz` is retained because it contains a vendor H5OINA input. The generated DREAM3D output was a circular oracle and was retired for this filter. The replacement uses inline Class 1 analytical data, Class 4 invariants, and the Class 2 h5py readback. + +SHA512: `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03`. + +## Deviations from DREAM.3D 6.5.171 + +DREAM.3D 6.5.171 has no H5OINA importer. Therefore, no legacy comparison is possible. + +The deviations document compares this verified version with `ReadH5OinaDataFilter` as shipped in DREAM3D-NX 7.0.0 through 7.4.1. See `vv/deviations/ReadH5OinaDataFilter.md` for the root cause, affected users, and recommendation for each defect. + +| Deviation | Observed difference | +|---|---| +| `ReadH5OinaDataFilter-D1` | Hexagonal `phi2` alignment uses 30 degrees instead of 30 radians. | +| `ReadH5OinaDataFilter-D2` | Every scan writes its Euler values to the correct tuple slab. | +| `ReadH5OinaDataFilter-D3` | Hexagonal alignment modifies each scan exactly once. | +| `ReadH5OinaDataFilter-D4` | Unsupported pattern import reports a clear preflight error. | +| `ReadH5OinaDataFilter-D5` | The gamma lattice slot contains gamma instead of beta. | +| `ReadH5OinaDataFilter-D6` | Lattice angles use degrees instead of radians. | +| `ReadH5OinaDataFilter-D7` | Missing phase datasets produce an error instead of invalid access. | +| `ReadH5OinaDataFilter-D8` | Multi-scan inputs must have safe phase indices and identical phase definitions. | +| `ReadH5OinaDataFilter-D9` | Stacking Order controls the scan order. | +| `ReadH5OinaDataFilter-D10` | Reader error messages reach the user. | +| `ReadH5OinaDataFilter-D11` | Both cell counts are validated and the returned error code is consistent. | +| `ReadH5OinaDataFilter-D12` | Invalid phase-group names produce an error instead of an exception. | +| `ReadH5OinaDataFilter-D13` | Geometry spacing must be finite and positive. | diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..261898ce13 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -0,0 +1,303 @@ +# Deviations: ReadH5OinaDataFilter + +**There is no DREAM3D 6.5.171 equivalent of this filter**, so there is no legacy comparison and no legacy-versus-SIMPLNX deviation to record. The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and has no `FromSIMPLJson`, no legacy-UUID mapping entry and no SIMPL conversion fixtures. + +This file therefore records, in the same structured form, every difference between the behavior DREAM3D-NX shipped and the correct behavior. Entries are referenced by stable ID (`ReadH5OinaDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. + +The filter first shipped in DREAM3D-NX 7.0.0. D1 through D13 affect every released version from 7.0.0 through 7.4.1. D5, D6, D7, D10, D11, and D12 are corrections to EbsdLib's `H5OINAReader` and require EbsdLib 3.1.2. + +--- + +## ReadH5OinaDataFilter-D1 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D1` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** With "Convert Hexagonal X-Axis to EDAX Standard" on — its shipped default — every scan point whose phase maps to `Hexagonal_High` had 30.0 added to its φ2. The file's Euler angles are radians, so this added **thirty radians**, not thirty degrees. Thirty radians is 4.867 radians modulo 2π, so the resulting orientation bears no relation to either convention: the correction is 57.3 times the intended one (the ratio is exactly 180/π) and lands at an arbitrary angle. Every downstream product of those orientations — pole figures, IPF colors, misorientations, grain segmentation — was wrong for every hexagonal point of every H5OINA file imported at default settings. Cubic and unindexed points were unaffected. + +**Root cause:** Bug. The correction is a 30 degree rotation about [0001] applied to φ2, and the `.ctf` importer, whose files store degrees, correctly adds the literal `30.0`. The H5OINA path reused that literal even though H5OINA stores radians and the filter performs no degrees-to-radians conversion anywhere. + +**Affected users:** Anyone importing an `.h5oina` file containing a hexagonal phase — titanium, magnesium, zirconium, zinc and hcp alloys generally — without turning the option off. Because the option defaults to on, this was the default outcome. Files with no hexagonal phase, including the archived 25 × 25 titanium-cubic test file, were never affected, which is why the pre-existing test suite could not see it. + +**Correct behavior:** The value added is 30 degrees expressed in radians, `30 × (π/180)`, computed on a double-precision intermediate so the stored float32 is the correctly-rounded result — the same arithmetic shape the `.ctf` importer uses. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, `…::"Conversion Option Combinations"` and `…::"Multi-Scan Hexagonal Alignment"`, whose φ2 expectations distinguish the correct value both from the literal-30 result and from a float32-intermediate result. + +**Recommendation:** Trust the corrected behavior. Results produced by 7.0.0 through 7.4.1 from `.h5oina` files with a hexagonal phase and the option left on must be regenerated; there is no post-hoc correction, because the shift was applied before any downstream analysis. Users who cannot upgrade should turn the option off and apply the 30 degree rotation with [Rotate Euler Reference Frame](../../docs/RotateEulerRefFrameFilter.md). + +--- + +## ReadH5OinaDataFilter-D2 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D2` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** When two or more scans were selected and stacked into one Image Geometry, the `Euler` array of every scan after the first landed at the wrong place. Scan *k* occupies the tuple slab starting at *k*·X·Y, so its Euler block belongs at element 3·*k*·X·Y; it was written at element *k*·X·Y instead — one third of the correct offset, because the offset was counted in tuples rather than in elements. Scan 2's Euler block therefore overwrote the last two thirds of scan 1's Euler data, and the last two thirds of scan 2's own slab were left at their zero-initialized values. Every other cell array was placed correctly, so the corruption was confined to orientations and was silent: no error, no warning, and an output whose array sizes and geometry were all correct. Single-scan imports were unaffected, because the offset is zero. + +**Root cause:** Bug. The `Euler` copy correctly passes an element count of `totalPoints × 3` but passed the destination offset in tuples. The sibling `ReadH5OimData` writes the same interleave correctly as `(sliceTupleStart + i) * 3`. + +**Affected users:** Anyone importing more than one scan from an `.h5oina` file in a single filter invocation. Single-scan imports — the common case, and the only case the pre-existing test covered — were never affected. + +**Correct behavior:** The destination offset is three times the tuple offset. Pinned by `test/ReadH5OinaDataTest.cpp::"Multi-Scan Slab Placement"`, which uses a cubic-only two-scan fixture so no alignment transform can mask a placement error, and asserts all 24 Euler values across both slabs. + +**Recommendation:** Trust the corrected behavior. Multi-scan H5OINA imports produced by 7.0.0 through 7.4.1 must be regenerated. + +--- + +## ReadH5OinaDataFilter-D3 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D3` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** In a multi-scan import with the hexagonal alignment on, the alignment loop always walked scan-point indices 0 through X·Y — the first scan's tuples — regardless of which scan was being copied. In an *S*-scan stack, the first scan's hexagonal points therefore received the shift *S* times over, and no point of any later scan received it at all. Single-scan imports were unaffected. + +**Root cause:** Bug. `copyRawEbsdData` computed the slab offset for the copies but the alignment helper it called neither took nor applied that offset. + +**Affected users:** Anyone importing more than one scan from an `.h5oina` file containing a hexagonal phase. Compounds with D1 and D2 on the same imports. + +**Correct behavior:** The alignment loop iterates the scan's own slab, so each point is visited exactly once. Pinned by `test/ReadH5OinaDataTest.cpp::"Multi-Scan Hexagonal Alignment"`, in which every point of both scans is hexagonal, so a shift applied to the wrong slab or applied twice changes a pinned value. + +**Recommendation:** Trust the corrected behavior. + +--- + +## ReadH5OinaDataFilter-D4 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D4` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved — as a limitation, honestly reported | + +**Symptom:** Turning on "Import Pattern Data" always failed, on every file, with "The parameter 'Read Pattern Data' has been enabled but there does not seem to be any pattern data in the file for the scan name selected" — including for files that plainly do contain pattern data. The archived production AZtec file, for instance, carries a 625 × 512 × 622 `Processed Patterns` dataset and still produced that message. + +**Root cause:** Bug in the reported reason, over a missing feature. `H5OINAReader::getPatternData()` returns `nullptr` unconditionally, `getPatternDims(std::array)` takes its argument **by value** with an empty body, and the reader's pattern-reading block is commented out. Preflight's pattern dimensions therefore stayed `{0, 0}` and the failure fired regardless of file content, attributing a missing library feature to the user's file. Behind that failure the execute-side plumbing was itself inconsistent: preflight created `Unprocessed Patterns` as `uint16` while execute fetched it as `UInt8Array`, which would have thrown had it ever been reached, and its copy loop used the tuple offset where it needed the element offset. The filter also only ever targeted `Unprocessed Patterns`, while real AZtec exports may carry only `Processed Patterns`. + +**Affected users:** Anyone who turned the parameter on. No user ever obtained pattern data from an `.h5oina` file through this filter. + +**Correct behavior:** The parameter is retained for pipeline compatibility, and preflight now reports the true reason — pattern import is not yet supported for H5OINA files — naming the file and pointing at the SimplnxCore **Read HDF5 Dataset** filter as the way to read the patterns a file does contain. The unreachable, inconsistent execute block and its error code `-34970` are removed. The parameter's help text and the filter documentation state the limitation. Pinned by `test/ReadH5OinaDataTest.cpp::"Parameter Rejections"`, section "Pattern Import Not Supported (-9583)". + +**Recommendation:** Trust the corrected behavior. Implementing H5OINA pattern import is a feature request against EbsdLib, not a fix. + +--- + +## ReadH5OinaDataFilter-D5 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D5` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** The `LatticeConstants` ensemble array reported each phase's γ angle as a copy of its β angle. The file's third lattice angle was never read. For a cubic phase, where α = β = γ, the error is invisible; for a hexagonal phase, whose angles are 90/90/120, the reported γ was 90 instead of 120, and the same applies to any monoclinic, triclinic, trigonal or rhombohedral cell whose γ differs from its β. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readHeader()` assembled the six lattice constants as `{a, b, c, angles[0], angles[1], angles[1]}`. + +**Affected users:** Anyone importing an `.h5oina` file whose phases are not cubic or tetragonal, and who reads `LatticeConstants` downstream. It does not affect orientations, phase indices or crystal-structure symmetry, which come from the Laue group and not from the lattice angles. + +**Correct behavior:** The gamma slot receives the third angle. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, whose hexagonal fixture phase has γ ≠ β specifically so that a gamma slot echoing beta is visible. + +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. + +--- + +## ReadH5OinaDataFilter-D6 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D6` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | +| **Breaking change** | **Yes.** The value of a published output array changes for every H5OINA import. | + +**Symptom:** The three angle slots of `LatticeConstants` were reported in radians for H5OINA imports and in degrees for every other EBSD importer. A cubic phase imported from an `.h5oina` file reported `1.5707964, 1.5707964, 1.5707964`, while the same phase imported from a `.ctf` or `.ang` file reported `90, 90, 90`. The array's meaning therefore depended on which file format the phase happened to come from, with nothing in the data to say which. + +**Root cause:** Library inconsistency, not a misreading. An H5OINA file stores its lattice angles in radians, and that is correct for the format; `H5OINAReader` copied them through unchanged, while the `.ang` and `.ctf` importers populate the same slots from degree-valued file fields. **The H5OINA files themselves are correct**; what differed was the unit contract at the importer boundary. + +**Affected users:** Anyone comparing or combining phase information across file formats, and anyone reading `LatticeConstants` from an H5OINA import while assuming the degrees convention that the rest of DREAM3D-NX uses. + +**Correct behavior:** `H5OINAReader` converts the angles from radians to degrees on import, on a double-precision intermediate, so the array carries the same unit no matter which format the phase came from. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"` (90, 90, 120 for the hexagonal fixture phase) and `…::"Real AZtec File Readback"` (90, 90, 90 for the production file's titanium-cubic phase). + +**Release note and migration:** This is a **breaking change to a published output**. The first release that carries it is DREAM3D-NX 7.5.0 with EbsdLib 3.1.2. Every `.dream3d` file written by 7.0.0 through 7.4.1 has radians in components 3, 4 and 5 of `LatticeConstants`, so any saved exemplar, regression baseline or pipeline comparison that reads those components changes value on upgrade. To compare a stored radian value against a new import, multiply it by 180/π. Consumers that compensated by converting H5OINA lattice angles themselves must stop doing so. Nothing else in the import changes unit: `Euler` is still radians and the three lattice dimensions are still unconverted. The user-facing migration note is in `docs/ReadH5OinaDataFilter.md` under "Migration Notes". The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). + +**Recommendation:** Trust the corrected behavior — the degrees convention is the one the rest of the toolkit uses and the one the other importers already produced. + +--- + +## ReadH5OinaDataFilter-D7 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D7` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** An `.h5oina` file whose phase group was missing its `Lattice Dimensions` or `Lattice Angles` dataset **crashed the process**. This is empirically demonstrated, not inferred: running the fixture against a build with the corrected filter sources but the pre-correction `H5OINAReader` reports the regression test as `OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582) (SEGFAULT)` rather than as a failure. The suite log recording that run is listed in `vv/provenance/ReadH5OinaDataFilter.md`. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readHeader()` read the two vector datasets while discarding their error codes, then indexed elements 0 through 2 of the resulting vectors, which are empty when the dataset is absent. The `Laue Group` and `Space Group` reads discarded their error codes as well, and a failed phase-group open was not checked at all. + +**Affected users:** Anyone opening a truncated, partially written or otherwise malformed `.h5oina` file. The crash occurs during preflight, so it takes down the application as the file is selected, before any pipeline runs. + +**Correct behavior:** The four required phase reads are checked and reported with EbsdLib error codes `-90030` through `-90033`, each naming the phase and the dataset; `Space Group` remains optional because the reader only passes it through. The filter surfaces the failure as `-9582` with the file path and the scan name. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"EbsdLib Error Passthrough - Missing Lattice Angles (-9582)"`, which is the fixture that used to crash. + +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. + +--- + +## ReadH5OinaDataFilter-D8 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D8` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** A later scan with a phase index above the first scan's phase count wrote past the ensemble arrays and could crash the process. Scans with the same indices but different phase definitions silently used one shared definition for all scan slices. + +**Root cause:** Bug. The single Ensemble Attribute Matrix that all of the stacked scans share is sized from the first selected scan's phase count, `phases.size() + 1`. The shared ensemble fill in `utilities/IEbsdOemReader.hpp` then runs once per selected scan and writes `crystalStructures[phaseId]`, `materialNames[phaseId]` and `latticeConstants` component `phaseId`, where `phaseId` is the integer in that scan's HDF5 phase group name. Any later scan whose phase index exceeds the first scan's phase count writes past the end of all three arrays. Preflight range-checked those indices for the first selected scan only, and the execute-side `-34972` check inspects the phase *column values*, not the phase *group names*, and runs after the ensemble fill has already happened. + +**Affected users:** Anyone selecting more than one scan from an `.h5oina` file whose scans differ in their phase lists. Single-scan imports and multi-scan imports of files with one uniform phase list — which is every file in the shipped test data — were never affected, which is why the pre-existing suite could not see it. + +**Correct behavior:** Preflight validates the phase indices (`-9587`), phase count (`-9589`), and phase definitions (`-9590`) of every selected scan. The definitions must match because the scans become slices of one 3D microstructure and share one Ensemble Attribute Matrix. + +**Recommendation:** Trust the corrected behavior. Re-import any multi-scan file whose phase definitions differ between scans. Import those scans separately if they describe different microstructures. + +--- + +## ReadH5OinaDataFilter-D9 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D9` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** The **Stacking Order** setting carried by the scan selection had no effect. Choosing *High To Low* produced exactly the same output as *Low To High*, with no error and no warning, so the Z order of a multi-scan stack could not be changed from the parameter that appears to control it. + +**Root cause:** Bug. The scan loop iterated `SelectedScanNames.scanNames` in list order unconditionally and never read `SelectedScanNames.stackingOrder`. The setting reached the algorithm — it is a member of `OEMEbsdScanSelectionParameter::ValueType` — and was simply not consulted. + +**Affected users:** Anyone importing more than one scan who set the stacking order to *High To Low*. Single-scan imports are unaffected, because the two orders coincide. + +**Correct behavior:** *Low To High* stacks the scans in the order they are listed; *High To Low* stacks them in the reverse of that order, so the last selected scan occupies tuple slab 0. Pinned by `test/ReadH5OinaDataTest.cpp::"Stacking Order"`, whose two sections assert cell values that only the corresponding order can produce. + +**Recommendation:** Trust the corrected behavior. A pipeline saved under 7.0.0 through 7.4.1 with *High To Low* selected now produces a Z-reversed stack relative to what it used to produce, which is what it always asked for; check any such pipeline before re-running it. The two sibling filters `ReadH5OimData` and `ReadH5EspritData` still ignore the same setting — see the V&V report's Follow-ups. + +--- + +## ReadH5OinaDataFilter-D10 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D10` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** Every failure reported out of `H5OINAReader` reached the user with a blank `Message:` field. A malformed `.h5oina` produced an error code and an empty explanation, so nothing in the message said which dataset, phase or scan was at fault. + +**Root cause:** Bug in the trusted parsing boundary. Ten failure paths in `H5OINAReader` were written as `std::string str; std::stringstream ss(str); ss << …; setErrorMessage(str);`. `std::stringstream(str)` copies `str` into the stream's own buffer, so everything composed went into that copy and the still-empty original was handed to `setErrorMessage()`. A related defect sat one level up: `readFile()` replaced whatever `readHeader()` or `readData()` had set with a generic "could not read header" / "could not read data", discarding the specific reason even where one had been composed correctly. + +**Affected users:** Anyone who hit any `H5OINAReader` error — a malformed file, a missing dataset, an absent scan. The filter's own `-8970` and `-9582` messages name the file and the scan, so the file was identifiable; the reason was not. + +**Correct behavior:** Each of the ten sites composes into a stream of its own and reports that stream's contents, and `readFile()`'s wrappers name the scan and append the inner message rather than replacing it. Corrected in EbsdLib 3.1.2. + +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. + +--- + +## ReadH5OinaDataFilter-D11 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D11` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** Two defects in one code path. A scan whose header declared zero rows was rejected with error code `-90301` recorded on the reader but `-301` handed back as the return value, so a caller reporting the return value and a caller reading `getErrorCode()` disagreed about what had happened. A scan whose header declared a **negative** column count was not rejected at all: the count was widened to `size_t`, producing an enormous allocation request. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readData()` validated the row count only, and its rejection returned a literal that did not match the code it had just set. + +**Affected users:** Anyone opening a truncated or otherwise malformed `.h5oina` file. DREAM3D-NX reports the code it receives, so the mismatched value was the one the user saw. + +**Correct behavior:** Both the row and the column count are validated and the rejection returns the code it sets. Corrected in EbsdLib 3.1.2. The filter rejects the same shape earlier and independently: `-9584` rejects `X Cells` or `Y Cells` below 1 at preflight, which is pinned by `test/ReadH5OinaDataTest.cpp::"Invalid Cell Counts rejected (-9584)"`, including the negative case. + +**Recommendation:** Trust the corrected behavior. The filter's `-9584` guard fires first for a file selected through DREAM3D-NX, so this entry matters to other `H5OINAReader` callers. + +--- + +## ReadH5OinaDataFilter-D12 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D12` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** A non-numeric or noncanonical phase-group name caused `std::stoi()` to throw during preflight. The exception stopped the import and left the opened HDF5 phase group unclosed. + +**Root cause:** Bug. `H5OINAReader::readHeader()` passed each HDF5 group name directly to `std::stoi()` after it opened the group. The code did not validate that the complete name was a positive integer. + +**Affected users:** Users with a malformed or manually edited H5OINA file whose phase group is not named with a canonical positive integer. + +**Correct behavior:** EbsdLib parses the complete group name before it opens the group. It returns `-90034` for a noncanonical name. The filter reports the error through `-9582` and identifies the file, scan, and group name. + +**Recommendation:** Trust the corrected behavior. Correct the phase-group names or export the file again from AZtec. + +--- + +## ReadH5OinaDataFilter-D13 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D13` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** The filter accepted non-positive or non-finite X, Y, or Z spacing and created an invalid Image Geometry. + +**Root cause:** Bug. Preflight checked only whether Z spacing was less than or equal to zero. It did not reject NaN, and it did not validate the X and Y step values from the file. + +**Affected users:** Users with malformed H5OINA step values or a non-finite Z Spacing parameter. + +**Correct behavior:** Preflight requires finite, positive X, Y, and Z spacing. It reports `-9580` for Z spacing and `-9591` for scan spacing, with the actual values and file context. + +**Recommendation:** Trust the corrected behavior. Correct the spacing values or export the file again from AZtec. + +--- + +## Malformed-input rejections + +These are not behavioral deviations on well-formed files — every one of them is a rejection path that stands where 7.0.0 through 7.4.1 performed an out-of-range read, an out-of-range write, or produced a silently useless output. They are listed here so a reader auditing the error-code series has one place to find them. The multi-scan phase-group cases are the exception and are recorded as D8, because they are reachable from a well-formed file. + +| Code | Rejects | Previously | +|---|---|---| +| `-9584` | `X Cells` or `Y Cells` below 1 in the first selected scan | The counts were cast to `usize` unchecked, producing a zero-sized geometry, or an enormous one from a negative count | +| `-9585` | A selected scan whose grid or step sizes differ from the first selected scan's | Only the first scan's header was ever read; the copy then spanned the later scan's reader buffers using the first scan's point count | +| `-9586` | A selected scan name that is not in the file | Only the first name was checked, at preflight; a bad later name failed part-way through execute with the earlier scans already written into the output arrays and no rollback | +| `-9587` | A phase group of **any** selected scan named outside 1 through N, where N is the first selected scan's phase count | The check covered the first selected scan only, so a group named `7` in a later scan wrote past the end of the ensemble arrays — see D8 | +| `-9589` | A selected scan whose phase-group count differs from the first selected scan's | The ensemble arrays were sized from the first scan alone and filled from every scan — see D8 | +| `-9590` | A selected scan whose phase definitions differ from the first selected scan's | One shared Ensemble Attribute Matrix silently kept one scan's phase definitions for every scan slice — see D8 | +| `-9591` | Non-positive or non-finite X or Y spacing | The invalid values were assigned to the Image Geometry — see D13 | +| `-34971` | A `Data` dataset whose extent disagrees with the header's cell counts, in either direction | The reader sizes its buffers to the actual extent while the copy spans the header's point count, reading past the end of those buffers when the dataset is short and silently dropping the surplus rows when it is long | +| `-34972` | A phase value outside `[0, phase count]` | The value indexed `CrystalStructures` unchecked in the alignment loop, reading past the end of the ensemble array | + +Error codes `-34970` (null pattern data) and `-9588` (stacking order not applied) are retired: the first with the unreachable execute-side pattern block described in D4, the second with the stacking-order implementation described in D9. diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..fabbb108d3 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md @@ -0,0 +1,218 @@ +# Exemplar Archive Provenance: ReadH5OinaDataFilter + +This sidecar documents the test data behind `ReadH5OinaDataFilter` and the disposition +of each part of it. + +**Working folder.** Every file named in this document under "Recorded output", "Supporting +evidence" or "Suite logs" lives in the V&V working folder `ww_work/ReadH5OinaData/`, which +is **not committed to this repository**; it is archived to the V&V working-folder remote. +The committed deliverables are this sidecar, `vv/ReadH5OinaDataFilter.md` and +`vv/deviations/ReadH5OinaDataFilter.md`. + +--- + +## Archive identity + +| Field | Value | +|---|---| +| **Archive** | `H5Oina_Test_Data.tar.gz` | +| **SHA512** | `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03` | +| **`download_test_data()` entry** | `src/Plugins/OrientationAnalysis/test/CMakeLists.txt:146` | +| **Contents** | `H5Oina_Test_Data.h5oina`, `H5Oina_Test_Data.dream3d`, `H5Oina_Test_Data.xdmf`, `Acknowledgements.md` | +| **Used by** | `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` (the `.h5oina` only) | +| **Generated by** | **Not generated for this V&V.** The archive predates it and is unchanged, so there is no generating pipeline, script or commit to record on this side. The `.h5oina` is a vendor export, not a generated file; the `.dream3d` and `.xdmf` were produced by running this filter over it, which is why they are no longer consulted (below). | +| **Generated on / at commit** | Unknown — the archive was uploaded before the provenance convention existed. Its identity is pinned by the SHA512 above, which is byte-for-byte the value in `CMakeLists.txt`. | +| **Canonical oracle output** | **None from this archive.** The oracle for this filter is analytical: expected values are derived from the fixture specification in the test source, and the production file is compared against a readback of its own bytes. See "Replacement oracle" below. | +| **Changed?** | **No.** The archive is unchanged and was not re-uploaded. | + +## The input file: retained + +`H5Oina_Test_Data.h5oina` is a genuine Oxford Instruments AZtec export and is retained as +an irreplaceable piece of production realism. Its properties, read directly with h5py: + +| Property | Value | +|---|---| +| `Format Version` | `5.0` | +| Scans | one, named `1` | +| Grid | `X Cells` 25, `Y Cells` 25 (625 points), `X Step` = `Y Step` = 4.0 | +| Phases | one group, named `1`: "Titanium cubic", Laue group 11 (Cubic-High), space group 229 | +| Lattice | dimensions 3.192, 3.192, 3.192; angles 1.5707964 rad each (90 degrees) | +| Phase column | values {0, 1} — the file contains both indexed and unindexed points | +| Euler | float32 (625, 3) in radians, observed range [0, 6.2773] | +| Extra columns | `Beam Position X`/`Y`, `Detector Distance`, `Pattern Center X`/`Y`, `Pattern Quality`, and a 625 × 512 × 622 `Processed Patterns` dataset — none of which this filter reads | + +Recorded probe output: `probe_real_file.txt`. + +Note that the file carries `Processed Patterns` and **no** `Unprocessed Patterns`, while the +filter only ever targeted the latter — one of the observations behind deviation D4. + +## The exemplar file: no longer consulted + +`H5Oina_Test_Data.dream3d` is **no longer used as a comparison target**. It remains inside +the archive because the archive is unchanged, but no test reads it. + +**Why.** The file was produced by running this filter on the sibling `.h5oina`, so it is a +self-oracle. Its `Euler` array is bit-identical to the raw file data — all 1,875 values, +compared as `uint32` bit patterns — which is what a verbatim copy of that column should +produce; the remaining eight cell arrays were not measured for bit-identity, because the +file's disposition does not turn on it. It is *not* a forbidden legacy oracle — no legacy +H5OINA importer has ever existed, so no version of DREAM3D could have generated it — but it +pins "the filter keeps doing what it did", not "the filter is right", which is the same +failure mode. + +It also had the pre-correction behavior baked in: its `LatticeConstants` reads +`[3.192, 3.192, 3.192, 1.5707964, 1.5707964, 1.5707964]`, the radian angles of deviation +D6. A test comparing against it did not merely fail to detect D6 — it actively enforced it. +Recorded measurement: `exemplar_selforacle_check.txt`. + +Its discriminating power against the thirteen deviations is limited: the file has +a single cubic phase, so the hexagonal alignment (D1, D3) never executes; it holds one scan, +so the slab offsets (D2), the stacking order (D9) and the multi-scan ensemble write (D8) are +never reached; pattern import was off, so D4 and its latent type mismatch were never +reached; and its angles are all equal, so the gamma slot (D5) is invisible. What it did pin +— single-scan copy plumbing, geometry wiring, the phase widening and the ensemble slot-0 +defaults against a real-world file — is preserved and strengthened by its replacement. + +## Replacement oracle + +**Toy fixtures, written by the test at run time.** `test/ReadH5OinaDataTest.cpp` declares a +fixture specification as C++ structs and writes `.h5oina` files with `H5Support::H5Lite` +into the binary test-output directory. The test cases create valid inputs and focused guard +inputs at run time. Nothing is committed as binary test data and no archive upload was +needed. The valid files carry the complete dataset set that `H5OINAReader` requires. The +guard files omit a required dataset or contain a value that the filter must reject. + +Every expected value is derived from the fixture specification. The hexagonal-alignment +expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, +derived independently with IEEE-754 semantics in NumPy and embedded as literals with +derivation comments beside them. + +**Class 2 independent readback for the production file.** +`…::"Real AZtec File Readback"` compares all nine cell arrays element-wise against a readback +of the `.h5oina`'s own `Data` datasets performed with `H5Lite` — the file bytes, bypassing +`H5OINAReader` entirely — for 6,966 assertions. The geometry and the three ensemble arrays +are pinned as literals derived from the h5py readback. + +## Independent-derivation script + +| Field | Value | +|---|---| +| **Script** | `h5oina_oracle.py` | +| **Interpreter** | `/opt/local/anaconda3/envs/dream3d/bin/python`, h5py 3.16.0, NumPy 2.5.2 | +| **Author** | Michael Jackson | +| **Date** | 2026-08-24 | +| **`spec` mode** | Prints the fixture specification and every derived expectation as C++-ready float32 literals. Recorded output: `oracle_spec.txt`, reproduced bit-for-bit as `oracle_spec_rerun.txt`. This is the source of the pinned hexagonal-alignment constants. | +| **`readback` mode** | Re-reads an `.h5oina` with h5py and re-derives the expected NX arrays from the documented rules, without EbsdLib or simplnx. Recorded output for the production file: `readback_real_file.txt`. | + +The script encodes the derivation rules read out of the filter and algorithm source — geometry +from the header, slab placement, verbatim copies, the phase widening, the Laue-group mapping, +the ensemble slot-0 defaults, the radians-to-degrees lattice-angle conversion and the +hexagonal φ2 alignment — and applies them to the fixture specification or to a file's bytes. +It never runs the filter. + +## Oracle and dependency versions + +| Boundary | Version | +|---|---| +| **EbsdLib** | 3.1.2. EbsdLib is part of the system under test. Six of the thirteen deviations are corrections in `H5OINAReader`. | +| h5py / NumPy | 3.16.0 / 2.5.2, under `/opt/local/anaconda3/envs/dream3d/bin/python` | +| HDF5 | as vendored by the `NX-Com-Qt69-Vtk96-Rel-EbsdLib` preset's vcpkg manifest | + +## Supporting evidence + +| File | What it records | +|---|---| +| `pi_over_6_discriminator.txt` | The exhaustive sweep of float32 values in `[0.25, 6.5)` that identified which φ2 values separate a double-precision intermediate from a float32 one (8,289,627 of 38,797,312 do), and confirmation that `30 × k_PiOver180D` is bit-identical to the nearest double to π/6 | +| `probe_real_file.txt` | The h5py structure dump of the production `.h5oina` | +| `error_column_check.txt` | The `Error` column census of the production file — 587 points at 1, 38 at 2, none at 0 — behind the documentation's masking correction | +| `exemplar_selforacle_check.txt` | The measurement behind the exemplar's retirement: `Euler` bit-identity and the radian `LatticeConstants` | +| `mutation_table.md` | Test-sensitivity evidence for the original seven temporary defects. Internal review added four detected defects for phase-group parsing, phase-definition validation, spacing validation, and error context. | + +## Suite logs + +| File | What it records | +|---|---| +| `red_baseline.log` | The RED baseline: the full suite run against the batch-base sources with the oracle suite in place — 14 of 16 failing, including the SEGFAULT that is deviation D7 | +| `fixround1_red_baseline.log` | The RED baseline for the multi-scan phase-group guards and the stacking-order implementation: three of 17 failing before those changes | +| `fixround1_green_readh5oina.log` | The green `ctest -V` run of the filter's suite, 17 of 17 | +| `fixround1_oa_suite.log` | The full `OrientationAnalysis::` suite at the verified tree state, after a full build of the preset directory: 308 tests, one failure | +| `fixround1_simplnxcore_suite.log` | The full `SimplnxCore::` suite at the same state | +| `gate_oa_head.log`, `gate_oa_ebsdlib539.log`, `gate_head_fail_names.txt`, `gate_539_fail_names.txt` | The controlled EbsdLib rollback described below | +| `ebsdlib_suite.log`, `simplnxcore_suite.log` | Earlier EbsdLib and SimplnxCore suite runs | + +### The pre-existing OrientationAnalysis failure + +**After a full build of the preset directory (`cmake --build . --target all`), the +`OrientationAnalysis::` suite is 308 tests with exactly one failure: +`OrientationAnalysis::ComputeSchmidsFilter`, on `Schmid_Lambdas`.** That is the known +EbsdLib-staging numerical drift from the `CubicOps` precision correction; it reads no +`.h5oina` file and shares no code with this filter. Log: `fixround1_oa_suite.log`. + +Two earlier runs of the same suite in the same directory recorded **twenty-nine** failures: +`ComputeSchmidsFilter` plus twenty `PIPELINE::OrientationAnalysis::` entries and eight +`PY::OrientationAnalysis::` entries. Those twenty-eight are **artifacts of a partially built +directory, not defects.** `PIPELINE::` tests invoke `Bin/nxrunner` and `PY::` tests invoke +the Python bindings, and neither target is built by +`cmake --build . --target OrientationAnalysisUnitTest`, which is what those runs had built. +Once `--target all` rebuilt `nxrunner` against the current libraries, all twenty-eight +passed with no source change. Any claim about this build directory's baseline has to be +measured after a full build. + +The one genuine failure is shown to be independent of the H5OINA corrections by a +**controlled rollback of the EbsdLib side alone**, which is the only variable in them that +could reach an unrelated test: + +| Configuration | Result | Non-H5OINA failure set | +|---|---|---| +| simplnx and EbsdLib both at the batch head | 307 tests, 29 fail | 29 names -- `gate_head_fail_names.txt` | +| simplnx at the batch head, EbsdLib rolled back to `539ddfc` (the batch's EbsdLib base) | 307 tests, 32 fail | **the same 29 names -- `diff` of the two sorted lists is empty** | + +Logs: `gate_oa_head.log` and `gate_oa_ebsdlib539.log`, with the sorted name lists in +`gate_head_fail_names.txt` and `gate_539_fail_names.txt`. Both runs were made in the +partially built state described above, so their absolute counts are inflated; what the +comparison establishes is that rolling EbsdLib back changes the non-H5OINA failure set not +at all. The three extra failures at `539ddfc` are the three ReadH5Oina tests that the +EbsdLib corrections are required for -- `Class 1 Analytical Oracle`, `Real AZtec File +Readback` and `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)`, the last as a +SEGFAULT -- which is evidence for the EbsdLib dependency. The +structural argument agrees: `git diff --stat 539ddfc..HEAD` on EbsdLib is one file, +`Source/EbsdLib/IO/HKL/H5OINAReader.cpp`, with no header and no API change, so nothing +outside the H5OINA read path can be reached. + +The same applies to `SimplnxCore::`, which is **985 of 985 passing** after the full build +(`fixround1_simplnxcore_suite.log`). An earlier partially built run of it reported +`PIPELINE::SimplnxCore::002_ApplyTransformation_Image` failing — another `nxrunner` +pipeline test of exactly the shape described above. + +`Real AZtec File Readback` is intermittently flaky in this build dir on the test-data +sentinel extraction, not on any assertion; a failure of that shape clears on an immediate +re-run with no source change. + +The final internal-review run uses EbsdLib 3.1.2. The results are 20 of 20 H5OINA tests, +408 of 408 EbsdLib tests, and 996 of 996 SimplnxCore tests. The OrientationAnalysis suite +passes 281 of 282 tests. The only failure is the unrelated ComputeSchmids archive +comparison against values from before its precision correction. + +## Oracle-before-comparison ordering + +There was no comparison to order against: DREAM3D 6.5.171 has no H5OINA importer. The oracle +was nonetheless established before any filter output was observed — the fixture +specification, the derivation script and every pinned literal were written and recorded +(`oracle_spec.txt`, timestamped before the first build) ahead of the first test run, and the +first run of the suite against the batch-base code is the RED baseline in `red_baseline.log`. +No expected value in the test file was taken from observed output. + +## Second-engineer oracle review + +Outstanding — to be recorded at PR review. The oracle design is fully auditable from the +committed test source: the fixture specification, the derivation of every pinned literal and +the reason each discriminating φ2 value was chosen are all stated in comments beside the +values. + +## Archive disposition + +`H5Oina_Test_Data.tar.gz` stays in the GitHub +[Data_Archive release](https://github.com/BlueQuartzSoftware/simplnx/releases/tag/Data_Archive) +and on the www.dream3d.io mirror unchanged, and its `download_test_data()` entry stays in +`test/CMakeLists.txt` because the `.h5oina` input is still needed. Should the archive ever be +regenerated, the `.dream3d` and `.xdmf` members can be dropped: nothing reads them.