From 20181125e1de808c4fa41f78bef6c949e0aba84b Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 11:28:46 -0400 Subject: [PATCH 01/10] VV: Correct and guard the H5OINA importer DREAM3D-NX 7.0.0 through 7.4.1 import Oxford AZtec .h5oina files with four defects in the value-add that sits on top of EbsdLib's H5OINAReader. All four are corrected here and pinned by a Class 1 analytical oracle suite built on hand-authored .h5oina fixtures. Hexagonal alignment. The EDAX/TSL x-axis convention differs from Oxford's by a 30 degree rotation about [0001] applied to phi2. An .h5oina file stores its Euler angles in radians, but the correction added the literal 30 -- thirty radians -- to every Hexagonal_High point, so with the option at its shipped default of ON every hexagonal orientation was wrong. The correction is now 30 degrees expressed in radians, applied on a double intermediate so the stored float32 is the correctly rounded result, which is how the .ctf importer applies the same correction. Multi-scan stacking. When several scans are stacked into one Image Geometry, scan k occupies the tuple slab starting at k * X * Y. The Euler array carries three components per point, but its destination offset was the tuple offset rather than three times it, so from the second scan on the Euler block landed a third of the way into its slab, overwriting the tail of the previous scan and leaving the rest of its own slab zeroed. The hexagonal alignment separately ignored the slab offset and always walked the first scan's points, shifting them once per scan while never reaching the later scans. Both now address the scan's own slab. Pattern import. Pattern data cannot be read from an .h5oina file: the reader's pattern accessors are stubs. Turning on "Import Pattern Data" produced an error claiming the file contained no pattern data, and the execute-side plumbing behind it was internally inconsistent -- preflight created a uint16 array that execute fetched as uint8. The parameter now reports honestly that pattern import is not yet supported for this format, and the unreachable copy block is removed. Malformed-file guards. Six inputs that previously produced an out-of-range read, an out-of-range write or a zero-sized geometry are now rejected with a message naming the offending value, the scan and the file: cell counts below 1 (-9584), a selected scan whose grid differs from the first selected scan's (-9585), a selected scan that is not in the file (-9586), phase groups not numbered 1 through N (-9587), a Data dataset whose extent disagrees with the header's cell counts (-34971), and a phase value outside the range the file's phase definitions establish (-34972). Only the first selected scan used to be checked at all, so a bad later name failed part way through execute with the earlier scans already written. The scan-selection parameter's stacking order is not applied by this filter -- scans are always stacked in list order -- so selecting High To Low now warns (-9588) instead of being silently ignored. The scan loop also reports progress per scan and honours cancellation. The 16 test cases replace an exemplar comparison against a .dream3d file that this filter had itself written. Expected values are derived from the fixture specification; the hexagonal expectations are correctly-rounded IEEE-754 results that distinguish a double intermediate from a float32 one and from the literal 30. The production AZtec file is now checked against a readback of its own datasets rather than against that exemplar. Signed-off-by: Michael Jackson --- .../Filters/Algorithms/ReadH5OinaData.cpp | 243 +++- .../Filters/ReadH5OinaDataFilter.cpp | 130 +- .../test/ReadH5OinaDataTest.cpp | 1113 +++++++++++++++-- 3 files changed, 1330 insertions(+), 156 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp index 9f934961f1..3e84b965b8 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp @@ -3,44 +3,123 @@ #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "H5Support/H5Lite.h" +#include "H5Support/H5ScopedSentinel.h" +#include "H5Support/H5Utilities.h" + +#include + +#include +#include +#include +#include + using namespace nx::core; namespace { +// The EDAX/TSL convention aligns the hexagonal crystal x-axis with [2-1-10] while +// Oxford Instruments aligns it with [10-10]; converting between them is a 30 degree +// rotation about [0001] applied to phi2. An H5OINA file stores its Euler angles in +// RADIANS (unlike a .ctf file, which stores degrees), so the value added here is 30 +// degrees expressed in radians -- 30 * (pi/180), i.e. pi/6 -- and not the literal 30. +// The addition runs on a double intermediate so the stored float32 is the correctly +// rounded result, matching how the .ctf importer applies the same correction. +constexpr float64 k_HexagonalAlignmentRadians = 30.0 * ebsdlib::constants::k_PiOver180D; + +// The nine datasets H5OINAReader reads out of a scan's Data group, paired with the +// number of components each one carries per scan point. +const std::vector> k_RequiredDataSets = { + {ebsdlib::H5OINA::BandContrast, 1}, {ebsdlib::H5OINA::BandSlope, 1}, {ebsdlib::H5OINA::Bands, 1}, {ebsdlib::H5OINA::Error, 1}, {ebsdlib::H5OINA::Euler, 3}, + {ebsdlib::H5OINA::MeanAngularDeviation, 1}, {ebsdlib::H5OINA::Phase, 1}, {ebsdlib::H5OINA::X, 1}, {ebsdlib::H5OINA::Y, 1}, +}; + +/** + * @brief Confirms that every Data dataset of a scan holds exactly as many elements as + * the Image Geometry expects. + * + * H5OINAReader sizes its buffers to whatever extent each dataset actually has, while + * the geometry and the destination arrays are sized from the header's X Cells and + * Y Cells. If a file's datasets are shorter than the header claims, copying + * totalPoints elements out of those buffers reads past their end. The extents are + * read straight from the file with H5Lite so no reader API is needed to expose them. + */ +Result<> validateDataSetExtents(const std::filesystem::path& filePath, const std::string& scanName, usize totalPoints) +{ + const hid_t fileId = H5Support::H5Utilities::openFile(filePath.string(), true); + if(fileId < 0) + { + return MakeErrorResult( + -34971, fmt::format("The file '{}' could not be reopened to verify the extents of scan '{}'. The file may have been moved or changed since preflight.", filePath.string(), scanName)); + } + H5Support::H5ScopedFileSentinel sentinel(const_cast(fileId), false); + + const std::string dataGroupPath = fmt::format("/{}/{}/{}", scanName, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data); + for(const auto& [dataSetName, componentCount] : k_RequiredDataSets) + { + const std::string dataSetPath = dataGroupPath + "/" + dataSetName; + std::vector dims; + H5T_class_t classType = H5T_NO_CLASS; + usize typeSize = 0; + if(H5Support::H5Lite::getDatasetInfo(fileId, dataSetPath, dims, classType, typeSize) < 0) + { + // A missing dataset is fatal inside H5OINAReader, which has already run by the + // time this is called, so there is nothing to add here. + continue; + } + usize elementCount = 1; + for(const hsize_t dim : dims) + { + elementCount *= static_cast(dim); + } + const usize expectedCount = totalPoints * componentCount; + if(elementCount != expectedCount) + { + return MakeErrorResult(-34971, fmt::format("The dataset '{}' of scan '{}' in '{}' holds {} element(s), but the {} scan point(s) described by the scan's header require {}. The file is " + "malformed or was changed after preflight.", + dataSetPath, scanName, filePath.string(), elementCount, totalPoints, expectedCount)); + } + } + return {}; +} template -void copyRawData(const ReadH5DataInputValues* m_InputValues, size_t totalPoints, DataStructure& m_DataStructure, ebsdlib::H5OINAReader& m_Reader, const std::string& name, usize offset) +void copyRawData(const ReadH5DataInputValues* inputValues, usize count, DataStructure& dataStructure, ebsdlib::H5OINAReader& reader, const std::string& name, usize offset) { using ArrayType = DataArray; - auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(name)); + auto& dataRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(name)); auto* dataStorePtr = dataRef.getDataStore(); - const nonstd::span rawDataPtr(reinterpret_cast(m_Reader.getPointerByName(name)), totalPoints); + const nonstd::span rawDataPtr(reinterpret_cast(reader.getPointerByName(name)), count); std::copy(rawDataPtr.begin(), rawDataPtr.end(), dataStorePtr->begin() + offset); } +/** + * @brief Applies the EDAX hexagonal x-axis alignment to the phi2 of every hexagonal + * point of one scan's tuple slab. + */ template -void convertHexEulerAngle(const ReadH5DataInputValues* m_InputValues, size_t totalPoints, DataStructure& m_DataStructure) +void convertHexEulerAngle(const ReadH5DataInputValues* inputValues, usize totalPoints, usize tupleOffset, DataStructure& dataStructure) { using ArrayType = DataArray; - if(m_InputValues->EdaxHexagonalAlignment) - { - auto& crystalStructuresRef = m_DataStructure.getDataRefAs(m_InputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - auto& crystalStructuresDSRef = crystalStructuresRef.getDataStoreRef(); + const auto& crystalStructuresRef = dataStructure.getDataRefAs(inputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); + const auto& crystalStructuresDSRef = crystalStructuresRef.getDataStoreRef(); - auto& cellPhasesRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); - auto& cellPhasesDSRef = cellPhasesRef.getDataStoreRef(); + const auto& cellPhasesRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); + const auto& cellPhasesDSRef = cellPhasesRef.getDataStoreRef(); - auto& eulerRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Euler)); - auto& eulerDataStoreRef = eulerRef.getDataStoreRef(); + auto& eulerRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Euler)); + auto& eulerDataStoreRef = eulerRef.getDataStoreRef(); - for(size_t i = 0; i < totalPoints; i++) + // Only this scan's slab is visited. Looping from 0 every time would shift the first + // scan's points once per scan and never reach the later scans' points. + for(usize i = tupleOffset; i < tupleOffset + totalPoints; i++) + { + if(crystalStructuresDSRef[cellPhasesDSRef[i]] == ebsdlib::CrystalStructure::Hexagonal_High) { - if(crystalStructuresDSRef[cellPhasesDSRef[i]] == ebsdlib::CrystalStructure::Hexagonal_High) - { - eulerDataStoreRef[3 * i + 2] = eulerDataStoreRef[3 * i + 2] + 30.0F; // See the documentation for this correction factor - } + const auto phi2 = static_cast(eulerDataStoreRef[3 * i + 2]); + eulerDataStoreRef[3 * i + 2] = static_cast(phi2 + k_HexagonalAlignmentRadians); } } } @@ -59,7 +138,42 @@ ReadH5OinaData::~ReadH5OinaData() noexcept = default; // ----------------------------------------------------------------------------- Result<> ReadH5OinaData::operator()() { - return execute(); + auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); + imageGeom.setUnits(IGeometry::LengthUnit::Micrometer); + + // The scan loop is kept here rather than in IEbsdOemReader::execute() so that the + // cancel checks and the progress messages below apply to this filter only. + const usize scanCount = m_InputValues->SelectedScanNames.scanNames.size(); + int index = 0; + for(const auto& currentScanName : m_InputValues->SelectedScanNames.scanNames) + { + if(m_ShouldCancel) + { + return {}; + } + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", currentScanName, index + 1, scanCount)}); + Result<> readResults = readData(currentScanName); + if(readResults.invalid()) + { + return readResults; + } + + if(m_ShouldCancel) + { + return {}; + } + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", currentScanName, index + 1, scanCount)}); + Result<> copyDataResults = copyRawEbsdData(index); + if(copyDataResults.invalid()) + { + return copyDataResults; + } + + ++index; + } + return {}; } // ----------------------------------------------------------------------------- @@ -67,67 +181,74 @@ Result<> ReadH5OinaData::copyRawEbsdData(int index) { const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const usize totalPoints = imageGeom.getNumXCells() * imageGeom.getNumYCells(); - const usize offset = index * totalPoints; - - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandContrast, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandSlope, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Bands, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Error, offset); - copyRawData(m_InputValues, totalPoints * 3, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Euler, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::MeanAngularDeviation, offset); + // Scan `index` occupies the tuple slab [index * totalPoints, (index + 1) * totalPoints). + const usize tupleOffset = static_cast(index) * totalPoints; + + const auto scanNameIterator = std::next(m_InputValues->SelectedScanNames.scanNames.cbegin(), index); + const std::string& scanName = *scanNameIterator; + + if(Result<> extentResults = validateDataSetExtents(m_InputValues->SelectedScanNames.inputFilePath, scanName, totalPoints); extentResults.invalid()) + { + return extentResults; + } + + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandContrast, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::BandSlope, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Bands, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Error, tupleOffset); + // Euler carries three components per scan point, so both the element count and the + // destination offset are three times the tuple counts. + copyRawData(m_InputValues, totalPoints * 3, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Euler, tupleOffset * 3); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::MeanAngularDeviation, tupleOffset); + + // The phase value of every point indexes the ensemble arrays, both in the alignment + // loop below and in every downstream filter, so it is range checked before it is + // stored. The valid range is [0, phase count]: 0 is the reserved Invalid Phase slot. + { + const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); + const usize ensembleTupleCount = crystalStructures.getNumberOfTuples(); + const nonstd::span rawPhasePtr(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); + for(usize i = 0; i < totalPoints; i++) + { + if(static_cast(rawPhasePtr[i]) >= ensembleTupleCount) + { + return MakeErrorResult(-34972, fmt::format("Scan point {} of scan '{}' carries phase value {}, which is outside the valid range [0, {}] established by the file's phase definitions.", i, + scanName, rawPhasePtr[i], ensembleTupleCount - 1)); + } + } + } + if(m_InputValues->ConvertPhaseToInt32) { const nonstd::span rawDataPtr(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); - using ArrayType = DataArray; - auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); + auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); auto* dataStorePtr = dataRef.getDataStore(); - for(size_t i = 0; i < totalPoints; i++) + for(usize i = 0; i < totalPoints; i++) { - dataStorePtr->setValue(i + offset, static_cast(rawDataPtr[i])); + dataStorePtr->setValue(i + tupleOffset, static_cast(rawDataPtr[i])); } } else { - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Phase, offset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Phase, tupleOffset); } - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::X, offset); - copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Y, offset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::X, tupleOffset); + copyRawData(m_InputValues, totalPoints, m_DataStructure, *m_Reader, ebsdlib::H5OINA::Y, tupleOffset); - if(m_InputValues->EdaxHexagonalAlignment) + if(m_ShouldCancel) { - if(m_InputValues->ConvertPhaseToInt32) - { - convertHexEulerAngle(m_InputValues, totalPoints, m_DataStructure); - } - else - { - convertHexEulerAngle(m_InputValues, totalPoints, m_DataStructure); - } + return {}; } - if(m_InputValues->ReadPatternData) + if(m_InputValues->EdaxHexagonalAlignment) { - const uint16* patternDataPtr = m_Reader->getPatternData(); - if(patternDataPtr == nullptr) + if(m_InputValues->ConvertPhaseToInt32) { - return MakeErrorResult(-34970, "Pattern data was requested but no pattern data was found in the data file"); + convertHexEulerAngle(m_InputValues, totalPoints, tupleOffset, m_DataStructure); } - std::array pDims = {{0, 0}}; - m_Reader->getPatternDims(pDims); - if(pDims[0] != 0 && pDims[1] != 0) + else { - std::vector pDimsV(2); - pDimsV[0] = pDims[0]; - pDimsV[1] = pDims[1]; - auto& patternData = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::UnprocessedPatterns)); - const usize numComponents = patternData.getNumberOfComponents(); - for(usize i = 0; i < totalPoints; i++) - { - for(usize j = 0; j < numComponents; ++j) - { - patternData[offset + numComponents * i + j] = patternDataPtr[numComponents * i + j]; - } - } + convertHexEulerAngle(m_InputValues, totalPoints, tupleOffset, m_DataStructure); } } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp index d204390fe4..b02517fced 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp @@ -22,7 +22,10 @@ #include #include +#include #include +#include +#include namespace fs = std::filesystem; using namespace nx::core; @@ -74,7 +77,10 @@ Parameters ReadH5OinaDataFilter::parameters() const params.insert(std::make_unique(k_ConvertPhaseToInt32_Key, "Convert Phase Data to Int32", "Native Phases data value is uint8. Convert to Int32 for better filter compatibility", true)); params.insert(std::make_unique(k_Origin_Key, "Origin", "The origin of the volume", std::vector{0.0F, 0.0F, 0.0F}, std::vector{"x", "y", "z"})); params.insert(std::make_unique(k_ZSpacing_Key, "Z Spacing (Microns)", "The spacing in microns between each layer.", 1.0f)); - params.insert(std::make_unique(k_ReadPatternData_Key, "Import Pattern Data", "Whether or not to import the pattern data", false)); + params.insert(std::make_unique(k_ReadPatternData_Key, "Import Pattern Data", + "Whether or not to import the diffraction pattern data. Pattern import is not yet supported for H5OINA files, so turning this on stops the filter " + "with an error.", + false)); params.insertSeparator(Parameters::Separator{"Output Image Geometry"}); params.insert(std::make_unique(k_CreatedImageGeometryPath_Key, "Image Geometry", "The path to the created Image Geometry", DataPath({ImageGeom::k_TypeName}))); params.insertSeparator(Parameters::Separator{"Output Cell Attribute Matrix"}); @@ -119,6 +125,8 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure nx::core::Result resultOutputActions; std::vector preflightUpdatedValues; + const std::string inputFilePath = pSelectedScanNamesValue.inputFilePath.string(); + if(pZSpacingValue <= 0) { return MakePreflightErrorResult(-9580, fmt::format("The Z Spacing field contains a value ({}) that is non-positive. The Z Spacing field must be set to a positive value.", pZSpacingValue)); @@ -127,15 +135,113 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure { return MakePreflightErrorResult(-9581, "At least one scan must be chosen. Please select a scan from the list."); } + if(pReadPatternDataValue) + { + return MakePreflightErrorResult(-9583, fmt::format("Pattern import is not yet supported for H5OINA files, so 'Import Pattern Data' must be turned off to read '{}'. The diffraction " + "patterns a file does contain can be read with the 'Read HDF5 Dataset' filter.", + inputFilePath)); + } // read in the necessary info from the input h5 file ebsdlib::H5OINAReader reader; - reader.setFileName(pSelectedScanNamesValue.inputFilePath.string()); - reader.setReadPatternData(pReadPatternDataValue); - reader.setHDF5Path(pSelectedScanNamesValue.scanNames.front()); + reader.setFileName(inputFilePath); + reader.setReadPatternData(false); + + // Every selected scan must be present in the file. Checking only the first one + // leaves a bad later name to fail part way through execute, with the scans that + // were already imported left in the output arrays. + std::list availableScanNames; + if(const int err = reader.readScanNames(availableScanNames); err < 0) + { + return MakePreflightErrorResult(-9582, fmt::format("An error occurred while listing the scans in '{}'.\n Error Code: {}\n Message: {}", inputFilePath, err, reader.getErrorMessage())); + } + for(const std::string& scanName : pSelectedScanNamesValue.scanNames) + { + if(std::find(availableScanNames.cbegin(), availableScanNames.cend(), scanName) == availableScanNames.cend()) + { + std::string availableList; + for(const std::string& availableScanName : availableScanNames) + { + availableList += (availableList.empty() ? "" : ", ") + availableScanName; + } + return MakePreflightErrorResult(-9586, fmt::format("The selected scan '{}' is not present in '{}'. The scans available in this file are: {}", scanName, inputFilePath, + availableList.empty() ? std::string("") : availableList)); + } + } + + const std::string& firstScanName = pSelectedScanNamesValue.scanNames.front(); + reader.setHDF5Path(firstScanName); if(const int err = reader.readHeaderOnly(); err < 0) { - return MakePreflightErrorResult(-9582, fmt::format("An error occurred while reading the header data\n{} : {}", err, reader.getErrorMessage())); + return MakePreflightErrorResult( + -9582, fmt::format("An error occurred while reading the header of scan '{}' in '{}'.\n Error Code: {}\n Message: {}", firstScanName, inputFilePath, err, reader.getErrorMessage())); + } + + // The geometry is sized from these two values, so a count below 1 has to be + // rejected here rather than producing an empty or absurdly large geometry. + if(reader.getXDimension() < 1 || reader.getYDimension() < 1) + { + return MakePreflightErrorResult(-9584, fmt::format("The header of scan '{}' in '{}' reports X Cells = {} and Y Cells = {}. Both must be at least 1. The file may be malformed or may not be an " + "H5OINA file.", + firstScanName, inputFilePath, reader.getXDimension(), reader.getYDimension())); + } + + // The Ensemble Attribute Matrix is sized from the number of phase groups in the + // file, but each phase is placed at the index carried by its group name. A file + // whose phase groups are not numbered 1..N would place a phase past the end of + // the ensemble arrays. + const auto phases = reader.getPhaseVector(); + for(const auto& phase : phases) + { + const int32 phaseIndex = phase->getPhaseIndex(); + if(phaseIndex < 1 || static_cast(phaseIndex) > phases.size()) + { + return MakePreflightErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares {} phase(s), but one of them carries index {}. The phase groups of an H5OINA file must be named 1 through {}.", + firstScanName, inputFilePath, phases.size(), phaseIndex, phases.size())); + } + } + + // Every other selected scan has to describe the same grid, because the geometry + // and every cell array are sized from the first scan's header alone. A second + // reader is used so the checks below do not disturb the header state that the + // preflight-updated values and the output actions are built from. + { + ebsdlib::H5OINAReader scanCheckReader; + scanCheckReader.setFileName(inputFilePath); + scanCheckReader.setReadPatternData(false); + for(const std::string& scanName : pSelectedScanNamesValue.scanNames) + { + if(scanName == firstScanName) + { + continue; + } + scanCheckReader.setHDF5Path(scanName); + if(const int err = scanCheckReader.readHeaderOnly(); err < 0) + { + return MakePreflightErrorResult( + -9582, fmt::format("An error occurred while reading the header of scan '{}' in '{}'.\n Error Code: {}\n Message: {}", scanName, inputFilePath, err, scanCheckReader.getErrorMessage())); + } + if(scanCheckReader.getXDimension() != reader.getXDimension() || scanCheckReader.getYDimension() != reader.getYDimension() || scanCheckReader.getXStep() != reader.getXStep() || + scanCheckReader.getYStep() != reader.getYStep()) + { + return MakePreflightErrorResult( + -9585, fmt::format("Scan '{}' in '{}' describes a {} x {} grid with steps ({}, {}), but scan '{}' describes a {} x {} grid with steps ({}, {}). Every selected scan must describe the same " + "grid, because they are stacked into a single Image Geometry.", + scanName, inputFilePath, scanCheckReader.getXDimension(), scanCheckReader.getYDimension(), scanCheckReader.getXStep(), scanCheckReader.getYStep(), firstScanName, + reader.getXDimension(), reader.getYDimension(), reader.getXStep(), reader.getYStep())); + } + } + } + + // The stacking order is carried by the scan-selection parameter and shared with + // the sibling OEM readers, but this filter always stacks the scans in the order + // they appear in the list. + if(pSelectedScanNamesValue.stackingOrder != RefFrameZDir::k_LowtoHigh) + { + resultOutputActions.warnings().push_back( + {-9588, fmt::format("The stacking order is set to High To Low, which this filter does not apply: the {} selected scans are always stacked in the order they are listed. Reverse the scan " + "selection itself to change the stacking.", + pSelectedScanNamesValue.scanNames.size())}); } // create the Image Geometry and it's attribute matrices @@ -151,7 +257,6 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure EbsdReaderUtilities::GeneratePreflightScanInformation(reader, preflightUpdatedValues); EbsdReaderUtilities::GeneratePreflightPhaseInformation(reader, preflightUpdatedValues); - const auto phases = reader.getPhaseVector(); std::vector ensembleTupleDims{phases.size() + 1}; { auto createAttributeMatrixAction = std::make_unique(cellEnsembleAMPath, ensembleTupleDims); @@ -190,19 +295,6 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure resultOutputActions.value().appendAction(std::make_unique(DataType::float32, tupleDims, std::vector{1}, cellAMPath.createChildPath(ebsdlib::H5OINA::X))); resultOutputActions.value().appendAction(std::make_unique(DataType::float32, tupleDims, std::vector{1}, cellAMPath.createChildPath(ebsdlib::H5OINA::Y))); - if(pReadPatternDataValue) - { - std::array patternDims = {{0, 0}}; - reader.getPatternDims(patternDims); - if(patternDims[0] == 0 || patternDims[1] == 0) - { - return MakePreflightErrorResult(-9583, fmt::format("The parameter 'Read Pattern Data' has been enabled but there does not seem to be any pattern data in the file for the scan name selected")); - } - auto createArrayAction = std::make_unique(DataType::uint16, tupleDims, std::vector{static_cast(patternDims[0]), static_cast(patternDims[1])}, - cellAMPath.createChildPath(ebsdlib::H5OINA::UnprocessedPatterns)); - resultOutputActions.value().appendAction(std::move(createArrayAction)); - } - return {std::move(resultOutputActions), std::move(preflightUpdatedValues)}; } diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 1ccd5c77da..733973765d 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -1,134 +1,1095 @@ -#include +/* ============================================================================ + * ReadH5OinaData V&V test suite. + * + * Verification is established INDEPENDENTLY of any DREAM3D output, per the V&V + * policy (src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md). There is + * no DREAM3D 6.5.171 H5OINA importer to compare against, so the oracle carries + * the whole burden: + * + * - .h5oina parsing : Class 2 (EbsdLib reference, trusted & NOT re-tested). + * EbsdLib's H5OINAReader owns HDF5 traversal, header and + * phase parsing, required-dataset enforcement and its own + * error codes. We do not re-test any of that; we do pin + * the codes it hands back through the filter. + * - SIMPLNX value-add : Class 1 (analytical) + Class 4 (invariant). The filter's + * value-add is deterministic plumbing on top of the reader: + * geometry construction from the first scan's header, array + * creation and typing, per-scan slab offsets, the verbatim + * column copies, the uint8 -> int32 Phase widening, the + * EDAX hexagonal alignment applied to phi2, ensemble slot-0 + * defaults and per-phase fill, and the malformed-input + * guards. The toy .h5oina fixtures below are written by + * this file with H5Lite and every expected value is derived + * from the fixture spec, never from observed output. + * - Real AZtec file : Class 2 independent readback. The archived production + * H5Oina_Test_Data.h5oina is compared against a readback of + * its own Data group datasets performed with H5Lite, i.e. the + * raw file bytes, bypassing H5OINAReader entirely. The + * equivalent h5py readback is + * ww_work/ReadH5OinaData/h5oina_oracle.py (readback mode); + * its recorded output is readback_real_file.txt. + * + * Precision pinning (the ReadCtfData lesson): fixture values are float32-exact. + * The hexagonal alignment adds 30 degrees expressed in radians, computed with a + * double-precision intermediate and stored back as float32. Three fixture phi2 + * values (0.1F, 0.34F, and Fixture C's 0.09F/0.22F/0.35F) are chosen because + * their correctly-rounded float32 results DIFFER between a double-precision + * intermediate and a float32 intermediate, so the expectations pin the shape of + * the arithmetic and not merely its magnitude. 0.25F is carried alongside as a + * dyadic control whose result is identical under either intermediate. Every such + * literal was derived with IEEE-754 float32/float64 semantics in NumPy; the + * derivation lives in ww_work/ReadH5OinaData/h5oina_oracle.py and its recorded + * output oracle_spec.txt. + * + * The archive's H5Oina_Test_Data.dream3d exemplar is no longer consulted: it was + * generated by this very filter, so it is a self-oracle that pins "the filter + * keeps doing what it did" rather than "the filter is right". + * ========================================================================== */ + +#include "OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp" +#include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" +#include "OrientationAnalysis/Parameters/OEMEbsdScanSelectionParameter.h" +#include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/DataGroupCreationParameter.hpp" #include "simplnx/Parameters/VectorParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp" -#include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" -#include "OrientationAnalysis/Parameters/OEMEbsdScanSelectionParameter.h" +#include "H5Support/H5Lite.h" +#include "H5Support/H5Utilities.h" -#include +#include +#include +#include + +#include +#include + +#include #include +#include +#include + namespace fs = std::filesystem; using namespace nx::core; using namespace nx::core::Constants; +using namespace H5Support; namespace { -const std::string k_ScanName = "1"; +const std::string k_CellAttributeMatrixName = "Cell Data"; +const std::string k_CellEnsembleAttributeMatrixName = "Cell Ensemble Data"; +const DataPath k_ImageGeomPath({ImageGeom::k_TypeName}); +const DataPath k_CellAMPath = k_ImageGeomPath.createChildPath(k_CellAttributeMatrixName); +const DataPath k_EnsembleAMPath = k_ImageGeomPath.createChildPath(k_CellEnsembleAttributeMatrixName); + +// EbsdLib/IO/HKL/CtfConstants.h LaueGroupTable: 9 = LG_Hexagonal_High, 11 = LG_Cubic_High. +// EbsdLib/Core/EbsdLibConstants.h: Hexagonal_High = 0, Cubic_High = 1, UnknownCrystalStructure = 999. +constexpr int32 k_LaueHexagonalHigh = 9; +constexpr int32 k_LaueCubicHigh = 11; + +//------------------------------------------------------------------------------ +// Toy .h5oina fixture description. The dataset set below is exactly the minimum +// H5OINAReader requires: the four Header scalars, one or more Phases/ groups +// carrying Phase Name / Lattice Dimensions / Lattice Angles / Laue Group / +// Space Group, and the nine Data datasets it reads unconditionally. The root +// Manufacturer / Software Version / Index datasets are written for realism only +// -- the reader never reads them. +//------------------------------------------------------------------------------ +struct PhaseSpec +{ + std::string name; + int32 laueGroup = k_LaueCubicHigh; + int32 spaceGroup = 225; + std::array latticeDimensions = {1.0F, 1.0F, 1.0F}; + // Lattice angles as an H5OINA file stores them: RADIANS. + std::array latticeAngles = {1.5707964F, 1.5707964F, 1.5707964F}; + // Group name override; empty means "use the phase's 1-based position". + std::string groupName; + bool omitLatticeAngles = false; +}; + +struct ScanSpec +{ + std::string name = "1"; + int32 xCells = 1; + int32 yCells = 1; + float32 xStep = 1.0F; + float32 yStep = 1.0F; + std::vector phases; + std::vector phase; + std::vector bandContrast; + std::vector bandSlope; + std::vector bands; + std::vector error; + std::vector euler; // 3 components per point, interleaved + std::vector mad; + std::vector x; + std::vector y; + // Guard-fixture switches. + bool omitBandsDataset = false; +}; + +//------------------------------------------------------------------------------ +template +void WriteVector(hid_t gid, const std::string& name, const std::vector& dims, const std::vector& data) +{ + const herr_t err = H5Lite::writeVectorDataset(gid, name, dims, data); + REQUIRE(err >= 0); } -TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Valid Filter Execution", "[OrientationAnalysis][ReadH5OinaDataFilter]") +//------------------------------------------------------------------------------ +hid_t OpenOrCreateGroup(hid_t fileId, const std::string& path) { - UnitTest::LoadPlugins(); + const hid_t err = H5Utilities::createGroupsFromPath(path, fileId); + REQUIRE(err >= 0); + const hid_t gid = H5Gopen(fileId, path.c_str(), H5P_DEFAULT); + REQUIRE(gid >= 0); + return gid; +} - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "H5Oina_Test_Data.tar.gz", "H5Oina_Test_Data"); +//------------------------------------------------------------------------------ +// Writes the fixture to / and returns the path. +//------------------------------------------------------------------------------ +fs::path WriteH5OinaFixture(const std::string& fileName, const std::vector& scans, bool writeFormatVersion = true, const std::string& formatVersion = "5.0") +{ + const fs::path filePath = fs::path(unit_test::k_BinaryTestOutputDir.view()) / fileName; + const hid_t fileId = H5Fcreate(filePath.string().c_str(), H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + REQUIRE(fileId >= 0); - // Read Exemplar DREAM3D File - auto exemplarFilePath = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.dream3d", unit_test::k_TestFilesDir)); - DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + if(writeFormatVersion) + { + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::FormatVersion, formatVersion) >= 0); + } + // Inert realism -- the reader never reads these three. + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::Manufacturer, "Oxford Instruments") >= 0); + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::SoftwareVersion, "V&V toy fixture") >= 0); + REQUIRE(H5Lite::writeStringDataset(fileId, ebsdlib::H5OINA::Index, "1") >= 0); - // Instantiate the filter, a DataStructure object and an Arguments Object - ReadH5OinaDataFilter filter; - DataStructure dataStructure; - Arguments args; + for(const ScanSpec& scan : scans) + { + const std::string headerPath = fmt::format("{}/{}/{}", scan.name, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Header); + const hid_t headerGid = OpenOrCreateGroup(fileId, headerPath); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::XCells, scan.xCells) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::YCells, scan.yCells) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::XStep, scan.xStep) >= 0); + REQUIRE(H5Lite::writeScalarDataset(headerGid, ebsdlib::H5OINA::YStep, scan.yStep) >= 0); + + for(usize i = 0; i < scan.phases.size(); i++) + { + const PhaseSpec& phaseSpec = scan.phases[i]; + const std::string groupName = phaseSpec.groupName.empty() ? std::to_string(i + 1) : phaseSpec.groupName; + const hid_t phaseGid = OpenOrCreateGroup(fileId, fmt::format("{}/{}/{}", headerPath, ebsdlib::H5OINA::Phases, groupName)); + REQUIRE(H5Lite::writeStringDataset(phaseGid, ebsdlib::H5OINA::PhaseName, phaseSpec.name) >= 0); + WriteVector(phaseGid, ebsdlib::H5OINA::LatticeDimensions, {1, 3}, {phaseSpec.latticeDimensions[0], phaseSpec.latticeDimensions[1], phaseSpec.latticeDimensions[2]}); + if(!phaseSpec.omitLatticeAngles) + { + WriteVector(phaseGid, ebsdlib::H5OINA::LatticeAngles, {1, 3}, {phaseSpec.latticeAngles[0], phaseSpec.latticeAngles[1], phaseSpec.latticeAngles[2]}); + } + REQUIRE(H5Lite::writeScalarDataset(phaseGid, ebsdlib::H5OINA::LaueGroup, phaseSpec.laueGroup) >= 0); + REQUIRE(H5Lite::writeScalarDataset(phaseGid, ebsdlib::H5OINA::SpaceGroup, phaseSpec.spaceGroup) >= 0); + REQUIRE(H5Gclose(phaseGid) >= 0); + } + REQUIRE(H5Gclose(headerGid) >= 0); - auto h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.h5oina", unit_test::k_TestFilesDir)); - OEMEbsdScanSelectionParameter::ValueType scanSelections = {h5TestFile, ebsdlib::RefFrameZDir::LowtoHigh, {k_ScanName}}; + const hid_t dataGid = OpenOrCreateGroup(fileId, fmt::format("{}/{}/{}", scan.name, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data)); + const auto pointCount = static_cast(scan.phase.size()); + WriteVector(dataGid, ebsdlib::H5OINA::BandContrast, {pointCount}, scan.bandContrast); + WriteVector(dataGid, ebsdlib::H5OINA::BandSlope, {pointCount}, scan.bandSlope); + if(!scan.omitBandsDataset) + { + WriteVector(dataGid, ebsdlib::H5OINA::Bands, {pointCount}, scan.bands); + } + WriteVector(dataGid, ebsdlib::H5OINA::Error, {pointCount}, scan.error); + WriteVector(dataGid, ebsdlib::H5OINA::Euler, {pointCount, 3}, scan.euler); + WriteVector(dataGid, ebsdlib::H5OINA::MeanAngularDeviation, {pointCount}, scan.mad); + WriteVector(dataGid, ebsdlib::H5OINA::Phase, {pointCount}, scan.phase); + WriteVector(dataGid, ebsdlib::H5OINA::X, {pointCount}, scan.x); + WriteVector(dataGid, ebsdlib::H5OINA::Y, {pointCount}, scan.y); + REQUIRE(H5Gclose(dataGid) >= 0); + } + + REQUIRE(H5Fclose(fileId) >= 0); + return filePath; +} + +//------------------------------------------------------------------------------ +// Fixture phases. +//------------------------------------------------------------------------------ +// Hexagonal: Laue group 9 -> CrystalStructure Hexagonal_High (0). Its lattice +// angles are 90/90/120 degrees written as radians, which is what an H5OINA file +// carries; the third angle differs from the second so a gamma slot that echoed +// beta would be visible. +const PhaseSpec k_HexPhase{"Hex Phase A", k_LaueHexagonalHigh, 194, {2.5F, 2.5F, 4.0F}, {1.5707964F, 1.5707964F, 2.0943952F}}; +// Cubic: Laue group 11 -> CrystalStructure Cubic_High (1). +const PhaseSpec k_CubicPhase{"Cubic Phase B", k_LaueCubicHigh, 225, {3.5F, 3.5F, 3.5F}, {1.5707964F, 1.5707964F, 1.5707964F}}; + +//------------------------------------------------------------------------------ +// Fixture A: one scan "1", 3 x 2 cells, XStep 0.25 / YStep 0.5, two phases. +// +// Points 0, 1 and 2 all carry phi2 = 0.1F on a hexagonal, a cubic and an +// unindexed (phase 0) point respectively, so the three expectations differ only +// through the alignment branch. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureAScan() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 3; + scan.yCells = 2; + scan.xStep = 0.25F; + scan.yStep = 0.5F; + scan.phases = {k_HexPhase, k_CubicPhase}; + scan.phase = {1, 2, 0, 2, 1, 1}; + scan.bandContrast = {10, 20, 30, 40, 50, 60}; + scan.bandSlope = {11, 21, 31, 41, 51, 61}; + scan.bands = {1, 2, 3, 4, 5, 6}; + scan.error = {0, 1, 0, 2, 0, 3}; + scan.euler = { + 0.25F, 0.125F, 0.1F, // pt 0 hexagonal -> phi2 shifted + 0.5F, 0.375F, 0.1F, // pt 1 cubic -> phi2 untouched + 0.75F, 0.625F, 0.1F, // pt 2 unindexed -> phi2 untouched + 1.0F, 0.875F, 0.75F, // pt 3 cubic -> phi2 untouched + 1.25F, 1.125F, 0.34F, // pt 4 hexagonal -> phi2 shifted + 1.5F, 1.375F, 0.25F, // pt 5 hexagonal -> phi2 shifted (dyadic control) + }; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}; + scan.x = {0.0F, 0.25F, 0.5F, 0.0F, 0.25F, 0.5F}; + scan.y = {0.0F, 0.0F, 0.0F, 0.5F, 0.5F, 0.5F}; + return scan; +} - // Create default Parameters for the filter. +// Fixture A expected Euler with the alignment ON. The three shifted values are the +// correctly-rounded float32 results of double(phi2) + 30*(pi/180): +// 0.1F -> 0.6235987544059753F (a float32 intermediate would give 0.6235988140106201F) +// 0.34F -> 0.8635987639427185F (a float32 intermediate would give 0.8635988235473633F) +// 0.25F -> 0.7735987901687622F (identical under either intermediate -- the control) +// Adding the literal 30.0F, as a degree-valued importer would, gives 30.1F / 30.34F / 30.25F. +const std::vector k_FixtureAEulerAligned = { + 0.25F, 0.125F, 0.6235987544059753F, 0.5F, 0.375F, 0.1F, 0.75F, 0.625F, 0.1F, 1.0F, 0.875F, 0.75F, 1.25F, 1.125F, 0.8635987639427185F, 1.5F, 1.375F, 0.7735987901687622F, +}; +// The same fixture with the alignment OFF: every value verbatim from the file. +const std::vector k_FixtureAEulerVerbatim = { + 0.25F, 0.125F, 0.1F, 0.5F, 0.375F, 0.1F, 0.75F, 0.625F, 0.1F, 1.0F, 0.875F, 0.75F, 1.25F, 1.125F, 0.34F, 1.5F, 1.375F, 0.25F, +}; + +//------------------------------------------------------------------------------ +// Fixture B: two scans, 2 x 2 each, CUBIC only. With no hexagonal points the +// Euler array must be a pure verbatim copy, so any slab-placement error is +// visible directly. Scan "2"'s values are disjoint from scan "1"'s. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureBScan1() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 2; + scan.yCells = 2; + scan.xStep = 0.25F; + scan.yStep = 0.5F; + scan.phases = {k_CubicPhase}; + scan.phase = {1, 1, 1, 1}; + scan.bandContrast = {10, 11, 12, 13}; + scan.bandSlope = {20, 21, 22, 23}; + scan.bands = {1, 2, 3, 4}; + scan.error = {0, 1, 2, 3}; + scan.euler = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F}; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F}; + scan.x = {0.0F, 0.25F, 0.0F, 0.25F}; + scan.y = {0.0F, 0.0F, 0.5F, 0.5F}; + return scan; +} + +ScanSpec MakeFixtureBScan2() +{ + ScanSpec scan = MakeFixtureBScan1(); + scan.name = "2"; + scan.bandContrast = {110, 111, 112, 113}; + scan.bandSlope = {120, 121, 122, 123}; + scan.bands = {5, 6, 7, 8}; + scan.error = {4, 5, 6, 7}; + scan.euler = {2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F}; + scan.mad = {1.125F, 1.25F, 1.375F, 1.5F}; + return scan; +} + +//------------------------------------------------------------------------------ +// Fixture C: two scans, 2 x 2 each, every point HEXAGONAL. A shift applied to +// the wrong slab, or applied more than once to slab 0, changes values that are +// pinned exactly here. +//------------------------------------------------------------------------------ +ScanSpec MakeFixtureCScan1() +{ + ScanSpec scan; + scan.name = "1"; + scan.xCells = 2; + scan.yCells = 2; + scan.xStep = 0.5F; + scan.yStep = 0.25F; + scan.phases = {k_HexPhase}; + scan.phase = {1, 1, 1, 1}; + scan.bandContrast = {30, 31, 32, 33}; + scan.bandSlope = {40, 41, 42, 43}; + scan.bands = {1, 2, 3, 4}; + scan.error = {0, 0, 0, 0}; + scan.euler = {0.25F, 0.5F, 0.1F, 0.75F, 1.0F, 0.34F, 1.25F, 1.5F, 0.25F, 1.75F, 2.0F, 0.5F}; + scan.mad = {0.125F, 0.25F, 0.375F, 0.5F}; + scan.x = {0.0F, 0.5F, 0.0F, 0.5F}; + scan.y = {0.0F, 0.0F, 0.25F, 0.25F}; + return scan; +} + +ScanSpec MakeFixtureCScan2() +{ + ScanSpec scan = MakeFixtureCScan1(); + scan.name = "2"; + scan.bandContrast = {130, 131, 132, 133}; + scan.bandSlope = {140, 141, 142, 143}; + scan.bands = {5, 6, 7, 8}; + scan.error = {1, 1, 1, 1}; + scan.euler = {2.25F, 2.5F, 0.09F, 2.75F, 3.0F, 0.22F, 3.25F, 3.5F, 0.75F, 3.75F, 4.0F, 0.35F}; + scan.mad = {1.125F, 1.25F, 1.375F, 1.5F}; + return scan; +} + +// Fixture C expected Euler across both slabs with the alignment ON. Every phi2 is +// float32(double(phi2) + 30*(pi/180)); 0.1F/0.34F in slab 0 and 0.09F/0.22F/0.35F +// in slab 1 separate a double intermediate from a float32 one, while 0.25F, 0.5F +// and 0.75F round identically under either. +const std::vector k_FixtureCEulerAligned = { + // slab 0 (scan "1") + 0.25F, + 0.5F, + 0.6235987544059753F, + 0.75F, + 1.0F, + 0.8635987639427185F, + 1.25F, + 1.5F, + 0.7735987901687622F, + 1.75F, + 2.0F, + 1.0235987901687622F, + // slab 1 (scan "2") + 2.25F, + 2.5F, + 0.6135987639427185F, + 2.75F, + 3.0F, + 0.7435987591743469F, + 3.25F, + 3.5F, + 1.2735987901687622F, + 3.75F, + 4.0F, + 0.8735987544059753F, +}; + +//------------------------------------------------------------------------------ +Arguments MakeArgs(const fs::path& inputFile, const std::list& scanNames, float32 zSpacing = 1.0F, bool hexAlignment = true, bool convertPhaseToInt32 = true, bool readPatternData = false, + uint32 stackingOrder = RefFrameZDir::k_LowtoHigh) +{ + Arguments args; + const OEMEbsdScanSelectionParameter::ValueType scanSelections = {inputFile, stackingOrder, scanNames}; args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0f))); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(DataPath({ImageGeom::k_TypeName}))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellData)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleData)); - - // Preflight the filter and check result + args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(zSpacing)); + args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0F))); + args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(readPatternData)); + args.insertOrAssign(ReadH5OinaDataFilter::k_EdaxHexagonalAlignment_Key, std::make_any(hexAlignment)); + args.insertOrAssign(ReadH5OinaDataFilter::k_ConvertPhaseToInt32_Key, std::make_any(convertPhaseToInt32)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(k_ImageGeomPath)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellAttributeMatrixName)); + args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleAttributeMatrixName)); + return args; +} + +//------------------------------------------------------------------------------ +template +void CompareArrayValues(const DataStructure& dataStructure, const DataPath& arrayPath, const std::vector& expected) +{ + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(arrayPath)); + const auto& dataArrayRef = dataStructure.getDataRefAs>(arrayPath); + REQUIRE(dataArrayRef.getSize() == expected.size()); + for(usize i = 0; i < expected.size(); i++) + { + INFO(fmt::format("Array '{}' index {}", arrayPath.toString(), i)); + REQUIRE(dataArrayRef[i] == expected[i]); + } +} + +//------------------------------------------------------------------------------ +void CompareStringArrayValues(const DataStructure& dataStructure, const DataPath& arrayPath, const std::vector& expected) +{ + REQUIRE_NOTHROW(dataStructure.getDataRefAs(arrayPath)); + const auto& stringArrayRef = dataStructure.getDataRefAs(arrayPath); + REQUIRE(stringArrayRef.getNumberOfTuples() == expected.size()); + for(usize i = 0; i < expected.size(); i++) + { + INFO(fmt::format("Array '{}' index {}", arrayPath.toString(), i)); + REQUIRE(stringArrayRef[i] == expected[i]); + } +} + +//------------------------------------------------------------------------------ +// Reads a dataset straight out of a .h5oina with H5Lite, i.e. from the file +// bytes, without involving H5OINAReader. Used by the real-file Class 2 readback. +//------------------------------------------------------------------------------ +template +std::vector ReadRawDataset(const fs::path& filePath, const std::string& datasetPath) +{ + const hid_t fileId = H5Utilities::openFile(filePath.string(), true); + REQUIRE(fileId >= 0); + std::vector data; + const herr_t err = H5Lite::readVectorDataset(fileId, datasetPath, data); + REQUIRE(err >= 0); + REQUIRE(H5Utilities::closeFile(const_cast(fileId)) >= 0); + return data; +} +} // namespace + +//------------------------------------------------------------------------------ +// Class 1 (analytical) + Class 4 (invariant) oracle over Fixture A with the +// shipped parameter defaults (hexagonal alignment ON, Phase widened to int32). +// Every expected value below is derived from the fixture spec above. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_oracle.h5oina", {MakeFixtureAScan()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}, 0.75F); + auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - const auto& imageGeom = dataStructure.getDataRefAs(DataPath({ImageGeom::k_TypeName})); - const auto& exemplarImageGeom = exemplarDataStructure.getDataRefAs(DataPath({k_ExemplarDataContainer})); - REQUIRE(imageGeom.getDimensions() == exemplarImageGeom.getDimensions()); - REQUIRE(imageGeom.getSpacing() == exemplarImageGeom.getSpacing()); - REQUIRE(imageGeom.getOrigin() == exemplarImageGeom.getOrigin()); + // --- Geometry: dims and spacing come from the header; the z extent is the + // --- number of selected scans, the z spacing is the parameter, the origin is + // --- the parameter, and the units are the filter's hard-coded Micrometer. --- + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(3, 2, 1)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(0.25F, 0.5F, 0.75F)); + REQUIRE(imageGeom.getOrigin() == FloatVec3(0.0F, 0.0F, 0.0F)); REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); - UnitTest::CompareExemplarToGeneratedData(dataStructure, exemplarDataStructure, DataPath({ImageGeom::k_TypeName, k_CellData}), k_ExemplarDataContainer); + // --- Cell arrays: eight verbatim copies plus the aligned Euler array. ------- + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), {10, 20, 30, 40, 50, 60}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), {11, 21, 31, 41, 51, 61}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), {0, 1, 0, 2, 0, 3}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), {0.0F, 0.25F, 0.5F, 0.0F, 0.25F, 0.5F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), {0.0F, 0.0F, 0.0F, 0.5F, 0.5F, 0.5F}); + // Phase is widened from the file's uint8 to int32 by default, values unchanged + // (including the unindexed 0 at point 2). + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureAEulerAligned); + + // --- Ensemble arrays: slot 0 is the reserved Invalid Phase, then one slot per + // --- phase group in the file. Laue 9 -> Hexagonal_High (0), Laue 11 -> + // --- Cubic_High (1). The file's radian lattice angles are stored as degrees, + // --- matching the .ang and .ctf importers: 90, 90, 120 for the hexagonal + // --- phase and 90, 90, 90 for the cubic one. ------------------------------- + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures), + {ebsdlib::CrystalStructure::UnknownCrystalStructure, ebsdlib::CrystalStructure::Hexagonal_High, ebsdlib::CrystalStructure::Cubic_High}); + CompareStringArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::MaterialName), {"Invalid Phase", "Hex Phase A", "Cubic Phase B"}); + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::LatticeConstants), + { + 0.0F, 0.0F, 0.0F, 0.0F, 0.0F, 0.0F, // slot 0: Invalid Phase + 2.5F, 2.5F, 4.0F, 90.0F, 90.0F, 120.0F, // slot 1: Hex Phase A + 3.5F, 3.5F, 3.5F, 90.0F, 90.0F, 90.0F, // slot 2: Cubic Phase B + }); - const DataPath cellEnsemblePath({ImageGeom::k_TypeName, k_CellEnsembleData}); - const DataPath exemplarCellEnsemblePath({k_ExemplarDataContainer, k_CellEnsembleData}); - const auto& crystalStructures = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - const auto& crystalStructuresExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::CrystalStructures)); - UnitTest::CompareDataArrays(crystalStructures, crystalStructuresExemplar); - const auto& latticeConstants = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::LatticeConstants)); - const auto& latticeConstantsExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::LatticeConstants)); - UnitTest::CompareDataArrays(latticeConstants, latticeConstantsExemplar); - const auto& materialName = dataStructure.getDataRefAs(cellEnsemblePath.createChildPath(ebsdlib::AngFile::MaterialName)); - const auto& materialNameExemplar = exemplarDataStructure.getDataRefAs(exemplarCellEnsemblePath.createChildPath(ebsdlib::AngFile::MaterialName)); - UnitTest::CompareStringArrays(materialNameExemplar, materialName); + // Class 4 invariant: the ensemble matrix always carries exactly one more tuple + // than the file has phase groups. + REQUIRE(dataStructure.getDataRefAs(k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures)).getNumberOfTuples() == 3); UnitTest::CheckArraysInheritTupleDims(dataStructure); } -TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: InValid Filter Execution", "[OrientationAnalysis][ReadH5OinaDataFilter]") +//------------------------------------------------------------------------------ +// The two value-transform options across their full 2 x 2 grid. Class 4 +// invariants: the Phase values never change with either option, and the +// hexagonal shift never reaches a cubic or unindexed point. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Conversion Option Combinations", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_ImportH5Data.tar.gz", "6_6_ImportH5Data"); + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_options.h5oina", {MakeFixtureAScan()}); + + const auto hexAlignment = GENERATE(true, false); + const auto convertPhaseToInt32 = GENERATE(true, false); + + DYNAMIC_SECTION("hexAlignment=" << hexAlignment << " convertPhaseToInt32=" << convertPhaseToInt32) + { + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}, 1.0F, hexAlignment, convertPhaseToInt32); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), hexAlignment ? k_FixtureAEulerAligned : k_FixtureAEulerVerbatim); + + if(convertPhaseToInt32) + { + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + } + else + { + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 2, 0, 2, 1, 1}); + } + + // The remaining columns are verbatim regardless of either option. + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +//------------------------------------------------------------------------------ +// Two scans stacked into one geometry. Fixture B is cubic-only, so every cell +// array -- Euler included -- must be a verbatim copy landing in its own tuple +// slab: scan "1" in tuples [0, 4) and scan "2" in tuples [4, 8). +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Multi-Scan Slab Placement", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_multiscan.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); - // Instantiate the filter, a DataStructure object and an Arguments Object ReadH5OinaDataFilter filter; DataStructure dataStructure; - Arguments args; - args.insertOrAssign(ReadH5OinaDataFilter::k_Origin_Key, std::make_any(std::vector(3, 0.0f))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CreatedImageGeometryPath_Key, std::make_any(DataPath({ImageGeom::k_TypeName}))); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellAttributeMatrixName_Key, std::make_any(k_CellData)); - args.insertOrAssign(ReadH5OinaDataFilter::k_CellEnsembleAttributeMatrixName_Key, std::make_any(k_CellEnsembleData)); + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 0.5F); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(2, 2, 2)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(0.25F, 0.5F, 0.5F)); + + // 24 Euler values: scan "1"'s 12 then scan "2"'s 12, each verbatim. + CompareArrayValues( + dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), + { + 0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F, 2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F, + }); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), {10, 11, 12, 13, 110, 111, 112, 113}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), {20, 21, 22, 23, 120, 121, 122, 123}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6, 7, 8}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), {0, 1, 2, 3, 4, 5, 6, 7}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 1.125F, 1.25F, 1.375F, 1.5F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 1, 1, 1, 1, 1, 1, 1}); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Two hexagonal scans. Every point takes the alignment, so the pinned values +// establish that the shift reaches scan "2"'s slab and is applied to scan "1"'s +// slab exactly once. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Multi-Scan Hexagonal Alignment", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_multiscan_hex.h5oina", {MakeFixtureCScan1(), MakeFixtureCScan2()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 2.0F); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(2, 2, 2)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getSpacing() == FloatVec3(0.5F, 0.25F, 2.0F)); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureCEulerAligned); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The pinned reader ignores the Format Version entirely -- its four version +// branches are empty and the dataset itself is optional. All three variants must +// produce the same output. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Format Version Variants", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const auto variant = GENERATE(as{}, "5.0", "2.0", "absent"); + + DYNAMIC_SECTION("Format Version " << variant) + { + const fs::path inputFile = WriteH5OinaFixture(fmt::format("read_h5oina_vv_version_{}.h5oina", variant), {MakeFixtureAScan()}, variant != "absent", variant); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(3, 2, 1)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), k_FixtureAEulerAligned); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +//------------------------------------------------------------------------------ +// Value-add preflight rejections that need no file at all. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Parameter Rejections", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_params.h5oina", {MakeFixtureAScan()}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + Arguments args = MakeArgs(inputFile, {"1"}); + int32 expectedCode = 0; + + SECTION("Non-positive Z Spacing (-9580)") + { + args = MakeArgs(inputFile, {"1"}, 0.0F); + expectedCode = -9580; + } + SECTION("No Scan Names Selected (-9581)") + { + args = MakeArgs(inputFile, {}); + expectedCode = -9581; + } + SECTION("Pattern Import Not Supported (-9583)") + { + args = MakeArgs(inputFile, {"1"}, 1.0F, true, true, true); + expectedCode = -9583; + } + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == expectedCode); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} - auto h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/FirstLook AB Site 1 Map Data 4.h5oina", unit_test::k_TestFilesDir)); - OEMEbsdScanSelectionParameter::ValueType scanSelections = {h5TestFile, ebsdlib::RefFrameZDir::LowtoHigh, {k_ScanName}}; +//------------------------------------------------------------------------------ +// Value-add guard: a header whose cell counts are not usable is rejected before +// a zero-sized (or absurdly-sized) geometry is created. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Cell Counts rejected (-9584)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); - SECTION("Invalid Z Spacing") + ScanSpec scan = MakeFixtureAScan(); + std::string fileName; + + SECTION("Zero X Cells") { - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(0.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + scan.xCells = 0; + fileName = "read_h5oina_vv_zero_xcells.h5oina"; } - SECTION("No Scan Names Selected") + SECTION("Zero Y Cells") { - scanSelections.scanNames.clear(); - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + scan.yCells = 0; + fileName = "read_h5oina_vv_zero_ycells.h5oina"; } - SECTION("Invalid h5 file type (incompatible manufacturer)") + SECTION("Negative X Cells") { - h5TestFile = fs::path(fmt::format("{}/H5Oina_Test_Data/FirstLook AB Site 1 Map Data 4.h5oina", unit_test::k_TestFilesDir)); - scanSelections.inputFilePath = h5TestFile; - args.insertOrAssign(ReadH5OinaDataFilter::k_SelectedScanNames_Key, std::make_any(scanSelections)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ZSpacing_Key, std::make_any(1.0f)); - args.insertOrAssign(ReadH5OinaDataFilter::k_ReadPatternData_Key, std::make_any(false)); + scan.xCells = -3; + fileName = "read_h5oina_vv_negative_xcells.h5oina"; } - // Preflight the filter and check result + const fs::path inputFile = WriteH5OinaFixture(fileName, {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9584); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: the geometry is built from the FIRST selected scan's header, +// so every other selected scan has to agree with it. A second scan with a +// different grid is rejected at preflight rather than driving an out-of-bounds +// read of the reader's buffers at execute. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Header Mismatch rejected (-9585)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan2 = MakeFixtureBScan2(); + + SECTION("Differing cell counts") + { + scan2.xCells = 3; + scan2.yCells = 3; + scan2.phase.assign(9, 1); + scan2.bandContrast.assign(9, 7); + scan2.bandSlope.assign(9, 7); + scan2.bands.assign(9, 7); + scan2.error.assign(9, 0); + scan2.euler.assign(27, 0.5F); + scan2.mad.assign(9, 0.25F); + scan2.x.assign(9, 0.0F); + scan2.y.assign(9, 0.0F); + } + SECTION("Differing step size") + { + scan2.xStep = 0.75F; + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_scan_mismatch.h5oina", {MakeFixtureBScan1(), scan2}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9585); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: every selected scan name is checked against the file at +// preflight, not just the first one. Without this a bad second name fails +// part-way through execute with half the arrays already populated. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Missing Scan Name rejected (-9586)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_scan.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); + + std::list scanNames; + SECTION("Missing first scan") + { + scanNames = {"9"}; + } + SECTION("Missing second scan") + { + scanNames = {"1", "9"}; + } + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, scanNames); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9586); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: phase indices come from the HDF5 phase GROUP NAMES, while the +// ensemble matrix is sized from the phase COUNT. A file whose groups are not +// numbered 1..N is rejected instead of writing past the end of the ensemble +// arrays. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Phase Index Out Of Range rejected (-9587)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureBScan1(); + scan.phases = {k_CubicPhase}; + scan.phases[0].groupName = "7"; // one phase, but it claims index 7 + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_index.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9587); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: EbsdLib sizes its data buffers to the ACTUAL dataset extent +// while the geometry is sized from the header's cell counts. A file whose Data +// datasets are shorter than X Cells * Y Cells is rejected before the copy spans +// past the end of those buffers. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Dataset Extent Mismatch rejected (-34971)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + // Header declares a 4 x 4 = 16 point scan; the Data datasets hold 8 points. + ScanSpec scan = MakeFixtureBScan1(); + scan.xCells = 4; + scan.yCells = 4; + scan.phase.assign(8, 1); + scan.bandContrast.assign(8, 5); + scan.bandSlope.assign(8, 6); + scan.bands.assign(8, 7); + scan.error.assign(8, 0); + scan.euler.assign(24, 0.25F); + scan.mad.assign(8, 0.125F); + scan.x.assign(8, 0.0F); + scan.y.assign(8, 0.0F); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_short_data.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -34971); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Value-add guard: a Phase byte larger than the file's phase count would index +// past the end of the CrystalStructures array in the alignment loop. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Out-of-Range Phase Value rejected (-34972)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + // One phase in the header, so the valid Phase range is [0, 1]; point 2 carries 5. + ScanSpec scan = MakeFixtureBScan1(); + scan.phase = {1, 1, 5, 1}; + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_range.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -34972); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Error propagation from the trusted EbsdLib boundary. A Data group missing one +// of the nine required datasets is fatal inside H5OINAReader::readData; the +// filter surfaces it as -8970 with the reader's own code and message attached. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Data Column (-8970)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + scan.omitBandsDataset = true; + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_bands.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + // The Data datasets are only read at execute, so preflight (header only) succeeds. + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + REQUIRE(executeResult.result.errors()[0].code == -8970); + // The message names the scan and the file so the user can find the problem. + REQUIRE(executeResult.result.errors()[0].message.find("'1'") != std::string::npos); + REQUIRE(executeResult.result.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// Error propagation from the trusted EbsdLib boundary at header-read time: a +// phase group missing its Lattice Angles dataset is reported through -9582, and +// the message carries the file path and scan name. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + scan.phases[0].omitLatticeAngles = true; + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_angles.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9582); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find("'1'") != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The scan-selection parameter carries a stacking order that this filter does +// not act on: scans are always stacked in the order they appear in the list. +// Selecting High-to-Low therefore raises a preflight warning rather than +// silently accepting a setting that has no effect. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Stacking Order Warning (-9588)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_stacking.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); + + ReadH5OinaDataFilter filter; + + { + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, RefFrameZDir::k_HightoLow); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.warnings().size() == 1); + REQUIRE(preflightResult.outputActions.warnings()[0].code == -9588); + } + { + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, RefFrameZDir::k_LowtoHigh); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.warnings().empty()); + } +} + +//------------------------------------------------------------------------------ +// Class 2 independent readback against the archived production AZtec file. +// +// The expected cell values are read straight out of the .h5oina's own Data +// datasets with H5Lite, bypassing H5OINAReader, and the ensemble expectations are +// derived from the file's phase group (Laue 11 -> Cubic_High; lattice angles +// pi/2 radians -> 90 degrees). The archive's H5Oina_Test_Data.dream3d exemplar is +// deliberately NOT consulted -- it was written by this filter and would only pin +// the filter against itself. +// +// This file is a single 25 x 25 cubic scan, so it exercises the single-scan copy +// plumbing, the Phase widening (its Phase column contains both 0 and 1) and the +// ensemble fill on production data. It cannot exercise the hexagonal alignment or +// the multi-scan slab offsets; the toy fixtures above carry those. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Real AZtec File Readback", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "H5Oina_Test_Data.tar.gz", "H5Oina_Test_Data"); + + const fs::path inputFile = fs::path(fmt::format("{}/H5Oina_Test_Data/H5Oina_Test_Data.h5oina", unit_test::k_TestFilesDir)); + REQUIRE(fs::exists(inputFile)); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // Geometry: X Cells = Y Cells = 25, X Step = Y Step = 4.0, one scan selected. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + const auto& imageGeom = dataStructure.getDataRefAs(k_ImageGeomPath); + REQUIRE(imageGeom.getDimensions() == SizeVec3(25, 25, 1)); + REQUIRE(imageGeom.getSpacing() == FloatVec3(4.0F, 4.0F, 1.0F)); + REQUIRE(imageGeom.getOrigin() == FloatVec3(0.0F, 0.0F, 0.0F)); + REQUIRE(imageGeom.getUnits() == IGeometry::LengthUnit::Micrometer); + + // Every cell array is a verbatim copy of the corresponding Data dataset: this + // scan's only phase is Laue 11 (Cubic_High), so no point takes the alignment. + const std::string dataPrefix = fmt::format("/1/{}/{}/", ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::BandContrast)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandSlope), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::BandSlope)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Bands)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Error)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Euler)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), + ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::MeanAngularDeviation)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::X)); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Y)); + + // Phase is the file's uint8 column widened to int32. + { + const std::vector rawPhase = ReadRawDataset(inputFile, dataPrefix + ebsdlib::H5OINA::Phase); + std::vector expectedPhase(rawPhase.size()); + for(usize i = 0; i < rawPhase.size(); i++) + { + expectedPhase[i] = static_cast(rawPhase[i]); + } + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), expectedPhase); + } + + // Ensemble: slot 0 is the reserved Invalid Phase; slot 1 is the file's single + // "Titanium cubic" phase, Laue group 11 -> Cubic_High, lattice dimensions + // 3.192 and lattice angles pi/2 radians stored as 90 degrees. + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures), + {ebsdlib::CrystalStructure::UnknownCrystalStructure, ebsdlib::CrystalStructure::Cubic_High}); + CompareStringArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::MaterialName), {"Invalid Phase", "Titanium cubic"}); + CompareArrayValues(dataStructure, k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::LatticeConstants), + { + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 0.0F, + 3.192F, + 3.192F, + 3.192F, + 90.0F, + 90.0F, + 90.0F, + }); UnitTest::CheckArraysInheritTupleDims(dataStructure); } From 37f9ec2bae639286d67f81095ba6951a1237167a Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 11:33:43 -0400 Subject: [PATCH 02/10] VV: Document what the H5OINA importer actually reads and produces The documentation described a version restriction the reader does not have, named the wrong Euler angle for the hexagonal alignment, and listed no created outputs at all. - The reader does not gate on the file's Format Version: the value is optional and unused, and the columns read are the Format Version 2.0 set that later versions retain. Replaces the claim that only Format Version 2.0 files are understood. - The hexagonal alignment applies to phi2, the third Euler angle, and only to Hexagonal-High points. Since the file's angles are radians the value added is 30 degrees expressed in radians, not the number 30. - Adds a Created Outputs section naming every array with its type and component count, including that Phase is int32 by default and that tuple 0 of the ensemble arrays is the reserved invalid phase. - Records that the lattice angles are imported as degrees while the Euler angles stay in radians. - Records the multi-scan stacking rules, that the stacking order setting is not applied, and that pattern import is not yet supported for this format. Signed-off-by: Michael Jackson --- .../docs/ReadH5OinaDataFilter.md | 64 ++++++++++++++++++- 1 file changed, 61 insertions(+), 3 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index 191ca58f05..34cb087c0b 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -17,9 +17,31 @@ The file is EBSD (Electron Backscatter Diffraction) scan data. The most importan - **Pattern-quality metrics** — values such as Band Contrast, Band Slope, Bands, and Mean Angular Deviation describe how clear and reliable each measurement is. These are commonly used to flag unreliable pixels (see *Reference Frames* below). - **Per-phase (Ensemble) data** — the crystal structure, lattice constants, and material name for each phase. An **Ensemble** here means one distinct material/crystal type. +### Reading More Than One Scan + +An H5OINA file can hold several scans. Selecting more than one stacks them into a single +3D **Image Geometry**: the X and Y extents and step sizes come from the first selected +scan, the Z extent is the number of selected scans, and the Z spacing is the **Z Spacing** +parameter. Every selected scan must describe the same grid as the first one, and each must +be present in the file; the filter reports an error naming the offending scan otherwise. + +The scans are always stacked in the order they are listed. The **Stacking Order** setting +carried by the scan selection is not applied by this filter, and selecting *High To Low* +raises a warning; reverse the scan selection itself to change the stacking. + ### Limitations of the Filter -The current implementation only understands **FORMAT VERSION 2.0** of the H5OINA file. A user can still read a newer H5OINA file, but the filter will only extract the VERSION 2.0 headers and data. If additional data is needed from the file, the [Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter can be used to augment this filter. +The filter reads the header keys and the nine data columns defined by **FORMAT VERSION 2.0** +of the H5OINA specification, which later versions retain. The file's `Format Version` value +is not used to select what is read, and a file without that value is read the same way. Any +column outside that set — for example `Pattern Quality`, `Beam Position X`/`Y` or the +`Electron Image` tree — is ignored, and can be brought in with the +[Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter. + +**Importing diffraction patterns is not yet supported for H5OINA files.** Turning on +*Import Pattern Data* stops the filter with an error rather than producing a partial result. +A file's `Processed Patterns` or `Unprocessed Patterns` dataset can be read with the +[Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter. ![Overview of the user interface.](Images/ImportH5OinaFilter_1.png) @@ -39,7 +61,14 @@ The user also may want to assign un-indexed pixels to be ignored by flagging the ### Radians and Degrees -All orientation data in the H5OINA file are in radians. +All orientation data in the H5OINA file are in radians, and the imported `Euler` array is in +radians as well — no conversion is applied. + +The per-phase `LatticeConstants` array is the one place where a unit does change on import. +An H5OINA file stores its three lattice angles in radians; they are imported as **degrees**, +so that the array means the same thing no matter which EBSD format the phase came from. A +cubic phase therefore reports `90, 90, 90` rather than `1.5707964, 1.5707964, 1.5707964`. +The three lattice dimensions are imported unchanged. ### The Axis Alignment Issue for Hexagonal Symmetry [1] @@ -50,7 +79,7 @@ All orientation data in the H5OINA file are in radians. + Caution: it appears that the axis alignment is a choice that must be made when installing TSL software so determination of which convention is in use must be made on a case-by-case basis. It is fixed to the y-convention in the HKL software. + The main clue that something is wrong in a conversion is that either the 2110 & 1010 pole figures are transposed, or that a peak in the inverse pole figure that should be present at 2110 has shifted over to 1010. + DREAM3D-NX uses the TSL/EDAX convention. -+ __The result of this is that the filter will by default add 30 degrees to the second Euler Angle (phi2) when reading Oxford `.h5oina` files. This can be disabled by the user if necessary.__ ++ __The result of this is that the filter will by default add 30 degrees to the third Euler angle (phi2) of every point whose phase is Hexagonal-High when reading Oxford `.h5oina` files. Because the file's Euler angles are in radians, the value actually added is 30 degrees expressed in radians (pi/6, about 0.5235988). Points of any other symmetry, and un-indexed points, are never adjusted. This can be disabled by the user if necessary.__ | Figure 1 | |--------| @@ -65,6 +94,35 @@ Once the reference frames are correct, the imported Euler angles are typically c None — this filter reads directly from a `.h5oina` file on disk. +## Created Outputs + +The array names are the H5OINA dataset names, so several contain spaces. + +### Cell Attribute Matrix + +| Name | Type | Components | Notes | +|------|------|------------|-------| +| `Band Contrast` | uint8 | 1 | | +| `Band Slope` | uint8 | 1 | | +| `Bands` | uint8 | 1 | | +| `Error` | uint8 | 1 | 0 marks a successfully indexed point | +| `Euler` | float32 | 3 | Radians; phi2 optionally shifted for Hexagonal-High points | +| `Mean Angular Deviation` | float32 | 1 | | +| `Phase` | int32 or uint8 | 1 | int32 by default; uint8 when *Convert Phase Data to Int32* is off. 0 marks an un-indexed point | +| `X` | float32 | 1 | | +| `Y` | float32 | 1 | | + +### Ensemble Attribute Matrix + +One tuple per phase in the file, plus tuple 0, which is reserved for the invalid phase that +un-indexed points refer to. + +| Name | Type | Components | Notes | +|------|------|------------|-------| +| `CrystalStructures` | uint32 | 1 | Mapped from the file's Laue group; 999 in tuple 0 | +| `LatticeConstants` | float32 | 6 | a, b, c then alpha, beta, gamma in **degrees**; all zero in tuple 0 | +| `MaterialName` | string | 1 | `"Invalid Phase"` in tuple 0 | + % Auto generated parameter table will be inserted here ## Example Pipelines From 23d6a7776df51ce7fc73b8350e6526544e3363c6 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 12:00:56 -0400 Subject: [PATCH 03/10] VV: Add the V&V deliverables for ReadH5OinaDataFilter Report, deviations and provenance sidecar for the H5OINA importer. The filter has no DREAM3D 6.5.171 equivalent -- that tree contains no H5OINA or AZtec importer of any kind -- so no legacy comparison was run and none is possible. Its place is taken by a Class 2 independent readback: the archived production AZtec file is compared against a readback of its own data sets, performed once in the test with H5Lite and once out of band with h5py, rather than against the .dream3d exemplar that ships beside it. That exemplar was written by this filter, so comparing against it pinned the filter to itself, and it carried the pre-correction radian lattice angles, so it actively enforced one of the defects. The deviations file records the seven differences between what DREAM3D-NX 7.0.0 through 7.4.1 shipped and the correct behavior, four in the filter and three in EbsdLib's H5OINAReader, with the affected releases named from docs/dream3d_nx_release_dates.md. Three of them reach users only through EbsdLib 3.1.1, which the report and the deviations both record as a release dependency. Signed-off-by: Michael Jackson --- .../vv/ReadH5OinaDataFilter.md | 173 ++++++++++++++++++ .../vv/deviations/ReadH5OinaDataFilter.md | 171 +++++++++++++++++ .../vv/provenance/ReadH5OinaDataFilter.md | 133 ++++++++++++++ 3 files changed, 477 insertions(+) create mode 100644 src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md create mode 100644 src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md create mode 100644 src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..ac3d8186dd --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -0,0 +1,173 @@ +# V&V Report: ReadH5OinaDataFilter + +| | | +|--------|--------------| +| Plugin | OrientationAnalysis | +| SIMPLNX UUID | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| SIMPLNX Human Name | Read Oxford Aztec Data (.h5oina) | +| DREAM3D 6.5.171 equivalent | **None.** DREAM3D 6.5.171 has no H5OINA/AZtec importer of any kind (verified by a case-insensitive search of `DREAM3D/Source`, `SIMPL/Source` and `DREAM3D_Plugins` in the 6.5.171 tree: zero source hits). The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and carries no `FromSIMPLJson` and no legacy-UUID mapping entry. | +| Verified commit | ** | +| Status | READY FOR REVIEW | +| Sign-off | Michael A. Jackson — 2026-08-24. Second engineer: **. | + +## At a glance + +| Aspect | Current state | +|------------------------|---------------| +| Algorithm Relationship | **New filter, no legacy equivalent.** Nothing in DREAM3D 6.5.171 imports H5OINA, so there is no port to classify and no legacy behavior to inherit or defend. The filter is one of three siblings (`ReadH5OimData`, `ReadH5EspritData`) built on the shared `IEbsdOemReader` template; EbsdLib's `H5OINAReader` does the parsing. | +| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications (A, B, C) are written by the test itself with H5Lite and materialise into nineteen `.h5oina` files — nine positive-path files and ten guard files — and every expected value is derived from the fixture specification. The archived production AZtec file is compared against a readback of its own data sets. Encoded as 16 TEST_CASEs (8,890 assertions) in `test/ReadH5OinaDataTest.cpp`; all pass. SIMPLNX matched the oracle once the seven defects below were corrected. | +| Code paths enumerated | 21 of 26 paths exercised (see Code path coverage). The five gaps are two `-9582` return sites that share a statement with a covered row or need permission manipulation, the display-only preflight information values, the shared `-8971` empty-phase path (unreachable from this filter), and the cancel-signal early returns. | +| Tests today | 16 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a three-way Format Version sweep, nine value-add rejection cases with the error code pinned per section, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | +| Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained, SHA512 `346573ac…d140ea03`, unchanged.** Its genuine Oxford AZtec `.h5oina` is kept as an irreplaceable production input. Its `H5Oina_Test_Data.dream3d` exemplar is no longer consulted: that file was written by this filter, so comparing against it pinned the filter to itself. Documented in `vv/provenance/ReadH5OinaDataFilter.md`. | +| Legacy comparison | **Not run — no legacy equivalent (verified against the 6.5.171 tree).** Its place is taken by the Class 2 independent readback described under Oracle. | +| Bug flags | SIMPLNX, all releases 7.0.0 through 7.4.1: hexagonal φ2 shifted by 30 radians instead of 30 degrees (D1), multi-scan Euler slabs misplaced (D2), the hexagonal shift confined to the first scan and repeated (D3), pattern import advertised but impossible (D4), the third lattice angle discarded (D5), lattice angles left in radians while every other importer reports degrees (D6), and a crash on a phase group missing its lattice angles (D7). All resolved. | +| V&V phase | Discovery, relationship, oracle, reconciliation, algorithm review (fixes applied), tests, deviations, provenance, docs — **complete**. Tests pass 16/16 in the EbsdLib preset build. **Release dependency:** the EbsdLib-side corrections (D5/D6/D7) live on `topic/3_1_1_staging` and reach users only through EbsdLib 3.1.1; the `vcpkg.json` pin is raised to `>= 3.1.1` and the pull request is merge-blocked until that release exists. OOC waived for this batch. Second-engineer sign-off outstanding (PR review). | + +## Summary + +`ReadH5OinaDataFilter` ("Read Oxford Aztec Data (.h5oina)") imports one or more scans from an Oxford Instruments AZtec `.h5oina` file into a single Image Geometry: it builds the geometry from the first selected scan's header, creates the nine cell arrays and the three ensemble arrays, copies each scan's data into its own tuple slab, optionally widens the file's `uint8` Phase column to `int32`, and optionally applies the EDAX/TSL hexagonal x-axis alignment to φ2. Verification is Class 1 analytical plus Class 4 invariant on hand-authored `.h5oina` fixtures, with a Class 2 independent readback standing in for the legacy A/B comparison that cannot exist — DREAM3D 6.5.171 has no H5OINA importer. Headline result: seven defects were found, four in the filter and three in EbsdLib's `H5OINAReader`, including a crash and a silently wrong orientation for every hexagonal point at the shipped default settings; all are corrected and pinned, and all 16 tests pass. + +## Algorithm Relationship + +*Classification:* **New filter.** + +*Evidence:* A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits. The `aztec` hits in that tree are the `H5Aztec` file-version constant belonging to DREAM3D's own HDF5-CTF archive format, which is unrelated to Oxford's H5OINA. The filter has no `FromSIMPLJson`, no `SIMPLConversion` include, no entry in the plugin's legacy-UUID mapping and no SIMPL conversion fixtures — all consistent with a filter that never existed in SIMPL. It was added by PR #700 (`a51dd5f3d`, 2024-03-25). + +Because there is no legacy equivalent, the same-UUID equivalence claim that drives the Deviations gate does not apply; the Deviations file instead records the differences between what DREAM3D-NX 7.0.0–7.4.1 shipped and the corrected behavior. + +*Material PRs since introduction:* #996 (OEM reader error messages), #1088 (parameter versioning), #1152 (spacing/origin ordering), #1263 (phase info in preflight values), #1438 (microtexture cleanup), #1472 (EbsdLib 2.0.0 API migration), #1576 (error-message sweep). None of them touched the hexagonal-alignment constant, the multi-scan offsets or the pattern path; those three carried their defects from the initial import through every subsequent change. + +## Oracle + +*Class:* **1 (Analytical) + 4 (Invariant)**, plus **2 (Independent readback)** for the production file; EbsdLib parsing = **Class 2 boundary (trusted, not re-tested)**. + +### The EbsdLib boundary (what we do NOT re-test) + +EbsdLib's `H5OINAReader` owns HDF5 traversal, the header and phase-group parsing, the nine required data-set reads, the Laue-group-to-crystal-structure mapping in `CtfPhase`, and its own error codes. Those behaviors are upstream's to verify, and the tests pin only that their codes and messages reach the user. The filter's value-add — everything this oracle covers — is the deterministic plumbing on top: geometry construction from the first scan's header, array creation and typing, per-scan slab offsets, the verbatim column copies, the `uint8`→`int32` Phase widening, the hexagonal φ2 alignment, ensemble slot-0 defaults, and the value-add rejection paths. + +Three defects found during this work sit *inside* that boundary but corrupt user-visible SIMPLNX output or crash the process (D5, D6, D7). They were corrected upstream on `topic/3_1_1_staging` rather than worked around in the filter, which is what creates the EbsdLib 3.1.1 release dependency. + +### Applied + +Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. Each fixture carries exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism. + +- **Fixture A** — one scan, 3 × 2 cells, steps 0.25 / 0.5, a hexagonal phase (Laue 9) and a cubic phase (Laue 11), with Phase values `{1, 2, 0, 2, 1, 1}` so an unindexed point is present. Points 0, 1 and 2 all carry φ2 = `0.1F` on a hexagonal, a cubic and an unindexed point respectively, so the three expectations differ only through the alignment branch. The hexagonal phase's lattice angles are 90/90/120 degrees stored as radians, so γ ≠ β. +- **Fixture B** — two scans, 2 × 2 each, cubic only, with disjoint values per scan. With no hexagonal point present the Euler array must be a pure verbatim copy, which isolates slab placement. +- **Fixture C** — two scans, 2 × 2 each, every point hexagonal, so a shift applied to the wrong slab or applied twice changes pinned values. +- **Guard fixtures** — zero / negative cell counts, mismatched scan grids, a missing scan name in first and second position, phase groups not numbered 1..N, an 8-row data set behind a 16-cell header, an out-of-range phase byte, a missing `Data/Bands`, and a phase group missing `Lattice Angles`. +- **Format Version variants** — `"5.0"`, `"2.0"` and absent, which must all produce identical output. + +Expected values are derived from the fixture specification, not from observed output. Verbatim copies are asserted with exact equality because every fixture value is float32-exact. The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy (`ww_work/ReadH5OinaData/h5oina_oracle.py`, recorded output `oracle_spec.txt`) and embedded as literals with derivation comments. + +**Precision pinning.** Following the ReadCtfData lesson, φ2 values were chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. `0.25F`, `0.5F` and `0.75F` are carried alongside as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, recorded in `pi_over_6_discriminator.txt`. + +Class 4 invariants encoded: the ensemble matrix always carries exactly one more tuple than the file has phase groups; slot 0 always holds `UnknownCrystalStructure` / `"Invalid Phase"` / zeroed lattice constants; the Phase values are unchanged by either conversion option; and the hexagonal shift never reaches a cubic or an unindexed point. + +### The Class 2 independent readback (substitute for the legacy A/B) + +There is no DREAM3D 6.5.171 H5OINA importer, so the comparison that would normally establish behavioral continuity does not exist. Its place is taken by an independent readback of the production AZtec file: + +- `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` reads the archived `H5Oina_Test_Data.h5oina`'s own `Data` datasets with `H5Lite` — the file bytes, bypassing `H5OINAReader` entirely — and compares them element-wise against the filter's output. The file is a single 25 × 25 cubic scan with 625 points, both indexed and unindexed, and the comparison covers all nine cell arrays (6,966 assertions in that test case). +- `ww_work/ReadH5OinaData/h5oina_oracle.py` performs the equivalent readback with h5py in a second language and a second HDF5 binding, re-deriving the geometry, the ensemble values and the cell arrays from the documented rules. Its recorded output is `readback_real_file.txt`; it is the source of the ensemble literals pinned in that test case (`CrystalStructures {999, 1}`, `MaterialName {"Invalid Phase", "Titanium cubic"}`, `LatticeConstants {3.192, 3.192, 3.192, 90, 90, 90}`). + +This file cannot exercise the hexagonal alignment (its only phase is cubic) or the multi-scan slab offsets (it holds one scan); the toy fixtures carry those paths. + +*Second-engineer review:* Outstanding — to be recorded at PR review. The oracle design is auditable in the test source: the fixture specification, the derivation of every literal and the Python cross-check are all committed or recorded in the evidence folder. + +## Algorithm review + +Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflightImpl` after oracle reconciliation. All findings applied; all 16 tests pass afterwards. + +- **Correctness:** the hexagonal alignment now adds 30 degrees expressed in radians on a double intermediate (D1); the Euler slab offset is now an element offset rather than a tuple offset (D2); the alignment loop now walks the scan's own slab (D3). +- **Robustness:** six malformed-input rejections added (`-9584`, `-9585`, `-9586`, `-9587`, `-34971`, `-34972`), each naming the offending value, the scan and the file. +- **Dead code:** the execute-side pattern block was unreachable — preflight always failed first — and internally inconsistent, creating a `uint16` array that execute fetched as `UInt8Array`. It is removed along with its error code `-34970`, and preflight now rejects the parameter honestly (D4). +- **Progress and cancel:** the scan loop moved from the shared `IEbsdOemReader::execute()` into `ReadH5OinaData::operator()` so that per-scan progress messages and three cancel checks could be added without changing the sibling filters. The shared `readData()` is still used. +- **Message quality:** `-9582` now carries the file path and the scan name; `-8970`'s message already carried both and is pinned by a test. +- **Not changed:** `utilities/IEbsdOemReader.hpp` is untouched, so `ReadH5OimData` and `ReadH5EspritData` are unaffected by this work. See Follow-ups. + +## Code path coverage + +*21 of 26 enumerated paths exercised. Source: `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp` (256 lines) + preflight in `Filters/ReadH5OinaDataFilter.cpp` (317 lines).* Logical phases: **(a)** preflight, **(b)** execute read + ensemble population (shared `IEbsdOemReader::readData`), **(c)** per-scan cell-data copy. + +| # | Phase | Path | Test case | +|----|-------|------|-----------| +| 1 | (a) | `z_spacing <= 0` → `-9580` | `Parameter Rejections` (section "Non-positive Z Spacing") | +| 2 | (a) | empty scan-name list → `-9581` | `Parameter Rejections` (section "No Scan Names Selected") | +| 3 | (a) | `read_pattern_data` on → `-9583` | `Parameter Rejections` (section "Pattern Import Not Supported") | +| 4 | (a) | `readScanNames` failure → `-9582` | *Not directly tested — needs an unreadable-but-present file (permission manipulation). The same `-9582` return statement family is covered by rows 5 and 6.* | +| 5 | (a) | selected scan absent from the file → `-9586` | `Missing Scan Name rejected (-9586)` (sections: missing first scan; missing second scan) | +| 6 | (a) | first scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | +| 7 | (a) | `X Cells`/`Y Cells` < 1 → `-9584` | `Invalid Cell Counts rejected (-9584)` (sections: zero X; zero Y; negative X) | +| 8 | (a) | phase index outside `[1, phase count]` → `-9587` | `Phase Index Out Of Range rejected (-9587)` | +| 9 | (a) | later scan `readHeaderOnly` failure → `-9582` | *Not separately tested — same return statement as row 6, reached from the per-scan loop.* | +| 10 | (a) | later scan grid differs from the first → `-9585` | `Scan Header Mismatch rejected (-9585)` (sections: differing cell counts; differing step size) | +| 11 | (a) | stacking order not Low-to-High → warning `-9588` | `Stacking Order Warning (-9588)` (both branches: warning raised, and no warning for Low-to-High) | +| 12 | (a) | geometry action: dims (X, Y, scan count), spacing (X Step, Y Step, z_spacing), user origin | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement`; `Multi-Scan Hexagonal Alignment` | +| 13 | (a) | ensemble matrix sized phase count + 1; three ensemble array actions | `Class 1 Analytical Oracle` (3 tuples, 2 phases) | +| 14 | (a) | nine cell-array actions with the Phase type chosen by `convert_phase_to_int32` | `Conversion Option Combinations` (both branches assert the Phase array's type) | +| 15 | (a) | preflight scan/phase information values | *Exercised implicitly by every preflight; display-only, not asserted.* | +| 16 | (b) | `readFile()` failure → `-8970` | `EbsdLib Error Passthrough - Missing Data Column (-8970)` (code and message content pinned) | +| 17 | (b) | empty phase vector → `-8971` | *Not directly tested — `H5OINAReader` rejects a file with no phase groups at header-read time with `-90009`, so preflight `-9582` fires first and this shared-code path is unreachable from this filter.* | +| 18 | (b) | ensemble slot-0 defaults and per-phase fill (Laue mapping, name, lattice constants) | `Class 1 Analytical Oracle`; `Real AZtec File Readback` | +| 19 | (c) | data-set extent disagrees with the header → `-34971` | `Dataset Extent Mismatch rejected (-34971)` | +| 20 | (c) | four `uint8` verbatim copies into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | +| 21 | (c) | Euler copy at three times the tuple offset | `Multi-Scan Slab Placement` (24 values across two slabs) | +| 22 | (c) | phase value outside `[0, phase count]` → `-34972` | `Out-of-Range Phase Value rejected (-34972)` | +| 23 | (c) | Phase widened to `int32` / copied verbatim as `uint8` | `Conversion Option Combinations` (both branches) | +| 24 | (c) | three `float32` verbatim copies (MAD, X, Y) into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | +| 25 | (c) | hexagonal alignment on the scan's own slab, Hexagonal-High points only | `Class 1 Analytical Oracle`; `Conversion Option Combinations`; `Multi-Scan Hexagonal Alignment` | +| 26 | (b)/(c) | cancel checks (3 sites) | *Not directly tested. Requires cancel-signal injection; standard early-return pattern. Excluded from scope by direction.* | + +## Test inventory + +| Test case | Status | Notes | +|-----------|--------|-------| +| `OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle` | new-for-V&V | Class 1 + 4 over Fixture A. Geometry, all nine cell arrays and all three ensemble arrays asserted element-wise; the ensemble tuple-count invariant. | +| `…: Conversion Option Combinations` | new-for-V&V | Class 1 + 4. `GENERATE` over the 2 × 2 hexagonal-alignment × phase-conversion grid; pins the Phase array's type in each branch and that the alignment never reaches cubic or unindexed points. | +| `…: Multi-Scan Slab Placement` | new-for-V&V | Class 1 over Fixture B (cubic, two scans). 24 Euler values plus five other arrays across two slabs; regression pin for D2. | +| `…: Multi-Scan Hexagonal Alignment` | new-for-V&V | Class 1 over Fixture C (hexagonal, two scans). Regression pin for D3; 24 Euler values, eight of which carry the precision discrimination. | +| `…: Format Version Variants` | new-for-V&V | `GENERATE` over `"5.0"` / `"2.0"` / absent; pins that the reader does not gate on the version and that the dataset is optional. | +| `…: Parameter Rejections` | new-for-V&V | Three sections pinning `-9580`, `-9581`, `-9583`. Replaces the previous invalid-execution test, whose sections asserted only "some error" and whose input file was in neither extracted archive. | +| `…: Invalid Cell Counts rejected (-9584)` | new-for-V&V | Three sections: zero X Cells, zero Y Cells, negative X Cells. | +| `…: Scan Header Mismatch rejected (-9585)` | new-for-V&V | Two sections: differing cell counts, differing step size. | +| `…: Missing Scan Name rejected (-9586)` | new-for-V&V | Two sections: the missing name in first and in second position. | +| `…: Phase Index Out Of Range rejected (-9587)` | new-for-V&V | A single-phase file whose phase group is named `7`. | +| `…: Dataset Extent Mismatch rejected (-34971)` | new-for-V&V | A 4 × 4 header over 8-row data sets; deterministic, no file-mutation injection needed. | +| `…: Out-of-Range Phase Value rejected (-34972)` | new-for-V&V | A one-phase file with a Phase byte of 5. | +| `…: EbsdLib Error Passthrough - Missing Data Column (-8970)` | new-for-V&V | Pins the code and that the message names the scan and the file. | +| `…: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | new-for-V&V | Pins the code and the message content. This fixture crashed the process before D7 was corrected. | +| `…: Stacking Order Warning (-9588)` | new-for-V&V | Both branches: the warning for High-to-Low, and no warning for Low-to-High. | +| `…: Real AZtec File Readback` | modified | Class 2. Was `Valid Filter Execution`, which compared against the archive's `.dream3d` exemplar — a file this filter had written. Now compares all nine cell arrays element-wise against an `H5Lite` readback of the `.h5oina`'s own data sets, with the geometry and the ensemble values pinned from the h5py derivation. | +| *(retired)* `…: InValid Filter Execution` | retired | Its sentinel extracted `6_6_ImportH5Data.tar.gz` while its paths pointed into `H5Oina_Test_Data/`, and the file it named exists in neither archive, so the "incompatible manufacturer" section passed on file-not-found. Replaced by `Parameter Rejections` and the nine code-pinned rejection cases. | + +All 16 pass in the EbsdLib preset build `NX-Com-Qt69-Vtk96-Rel-EbsdLib` — 8,890 assertions in total. OOC is waived for this batch. + +## Exemplar archive + +- **Archive:** `H5Oina_Test_Data.tar.gz`, SHA512 `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03`, matching the `download_test_data()` entry at `test/CMakeLists.txt:146`. Unchanged — no re-upload. +- **Retained:** `H5Oina_Test_Data.h5oina`, a genuine Oxford AZtec export (Format Version 5.0, 25 × 25, 4 µm steps, one cubic titanium phase). Irreplaceable production realism, used as the input to the Class 2 readback. +- **No longer consulted:** `H5Oina_Test_Data.dream3d`. It was produced by running this filter on the sibling `.h5oina`, so it is a self-oracle. It also had the pre-correction radian lattice angles baked into `LatticeConstants`, which means the previous test actively enforced D6. +- **Provenance:** `src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md`. + +## Deviations from DREAM3D 6.5.171 + +**Not applicable — DREAM3D 6.5.171 has no H5OINA importer** (evidence in Algorithm Relationship). No legacy comparison was run and none is possible. + +`vv/deviations/ReadH5OinaDataFilter.md` instead records, in the same structured form, the seven differences between what DREAM3D-NX 7.0.0 through 7.4.1 shipped and the corrected behavior: + +- `ReadH5OinaDataFilter-D1` — the hexagonal alignment added 30 radians to a radian-valued φ2 instead of 30 degrees; the option ships ON. +- `ReadH5OinaDataFilter-D2` — in a multi-scan import the Euler block of scan 2 onward landed a third of the way into its slab. +- `ReadH5OinaDataFilter-D3` — the hexagonal alignment was applied to the first scan once per scan and never to the others. +- `ReadH5OinaDataFilter-D4` — "Import Pattern Data" could never succeed and reported a misleading reason. +- `ReadH5OinaDataFilter-D5` — the third lattice angle was discarded and γ echoed β. +- `ReadH5OinaDataFilter-D6` — lattice angles were reported in radians while every other importer reports degrees. +- `ReadH5OinaDataFilter-D7` — a phase group missing `Lattice Angles` crashed the process. + +D5, D6 and D7 are corrected in EbsdLib and reach users only through EbsdLib 3.1.1. + +## Follow-ups for the engineering team + +1. **Sibling exposure (not fixed here, by direction).** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. `utilities/IEbsdOemReader.hpp` was deliberately left untouched so this work changes no sibling behavior. +2. **`stackingOrder` is still not implemented** (see the HO-9 proposal in the task report). It now warns rather than being silently ignored; implementing it would require a member on the shared `ReadH5DataInputValues` and a change to the shared scan loop, which is sibling-affecting work. +3. **EbsdLib 3.1.1 release gate.** This pull request joins #1723 behind the same gate. +4. **`H5OINAReader::getPatternDims(std::array)` takes its argument by value** and `getPatternData()` returns `nullptr`. Implementing pattern import for H5OINA is a feature, not a fix, and is out of scope here. diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..28dcc871e6 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -0,0 +1,171 @@ +# Deviations: ReadH5OinaDataFilter + +**There is no DREAM3D 6.5.171 equivalent of this filter**, so there is no legacy comparison and no legacy-versus-SIMPLNX deviation to record. A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits; the filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and has no `FromSIMPLJson`, no legacy-UUID mapping entry and no SIMPL conversion fixtures. + +This file therefore records, in the same structured form, every difference between the behavior DREAM3D-NX shipped and the correct behavior. Entries are referenced by stable ID (`ReadH5OinaDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. + +Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1, D2, D3, D4, D5 and D6 are present in every release from 7.0.0 through 7.4.1 (2026-03-23), which is every release that contains the filter. **No released version carries these corrections yet**: D5, D6 and D7 are corrections to EbsdLib's `H5OINAReader` and reach users only through EbsdLib 3.1.1, and the DREAM3D-NX release that consumes it has not been made. + +--- + +## ReadH5OinaDataFilter-D1 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D1` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** With "Convert Hexagonal X-Axis to EDAX Standard" on — its shipped default — every scan point whose phase maps to `Hexagonal_High` had 30.0 added to its φ2. The file's Euler angles are radians, so this added **thirty radians**, not thirty degrees. Thirty radians is 4.867 radians modulo 2π, so the resulting orientation bears no relation to either convention: the correction is 9.55 times too large and lands at an arbitrary angle. Every downstream product of those orientations — pole figures, IPF colors, misorientations, grain segmentation — was wrong for every hexagonal point of every H5OINA file imported at default settings. Cubic and unindexed points were unaffected. + +**Root cause:** Bug. The correction is a 30 degree rotation about [0001] applied to φ2, and the `.ctf` importer, whose files store degrees, correctly adds the literal `30.0`. The H5OINA path reused that literal even though H5OINA stores radians and the filter performs no degrees-to-radians conversion anywhere. + +**Affected users:** Anyone importing an `.h5oina` file containing a hexagonal phase — titanium, magnesium, zirconium, zinc and hcp alloys generally — without turning the option off. Because the option defaults to on, this was the default outcome. Files with no hexagonal phase, including the archived 25 × 25 titanium-cubic test file, were never affected, which is why the pre-existing test suite could not see it. + +**Correct behavior:** The value added is 30 degrees expressed in radians, `30 × (π/180)`, computed on a double-precision intermediate so the stored float32 is the correctly-rounded result — the same arithmetic shape the `.ctf` importer uses. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, `…::"Conversion Option Combinations"` and `…::"Multi-Scan Hexagonal Alignment"`, whose φ2 expectations distinguish the correct value both from the literal-30 result and from a float32-intermediate result. + +**Recommendation:** Trust the corrected behavior. Results produced by 7.0.0 through 7.4.1 from `.h5oina` files with a hexagonal phase and the option left on must be regenerated; there is no post-hoc correction, because the shift was applied before any downstream analysis. Users who cannot upgrade should turn the option off and apply the 30 degree rotation with [Rotate Euler Reference Frame](../../docs/RotateEulerRefFrameFilter.md). + +--- + +## ReadH5OinaDataFilter-D2 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D2` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** When two or more scans were selected and stacked into one Image Geometry, the `Euler` array of every scan after the first landed at the wrong place. Scan *k* occupies the tuple slab starting at *k*·X·Y, so its Euler block belongs at element 3·*k*·X·Y; it was written at element *k*·X·Y instead — one third of the way into the slab, counted in tuples rather than elements. Scan 2's Euler block therefore overwrote the last two thirds of scan 1's Euler data, and the last two thirds of scan 2's own slab were left at their zero-initialized values. Every other cell array was placed correctly, so the corruption was confined to orientations and was silent: no error, no warning, and an output whose array sizes and geometry were all correct. Single-scan imports were unaffected, because the offset is zero. + +**Root cause:** Bug. The `Euler` copy correctly passes an element count of `totalPoints × 3` but passed the destination offset in tuples. The sibling `ReadH5OimData` writes the same interleave correctly as `(sliceTupleStart + i) * 3`. + +**Affected users:** Anyone importing more than one scan from an `.h5oina` file in a single filter invocation. Single-scan imports — the common case, and the only case the pre-existing test covered — were never affected. + +**Correct behavior:** The destination offset is three times the tuple offset. Pinned by `test/ReadH5OinaDataTest.cpp::"Multi-Scan Slab Placement"`, which uses a cubic-only two-scan fixture so no alignment transform can mask a placement error, and asserts all 24 Euler values across both slabs. + +**Recommendation:** Trust the corrected behavior. Multi-scan H5OINA imports produced by 7.0.0 through 7.4.1 must be regenerated. + +--- + +## ReadH5OinaDataFilter-D3 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D3` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** In a multi-scan import with the hexagonal alignment on, the alignment loop always walked scan-point indices 0 through X·Y — the first scan's tuples — regardless of which scan was being copied. In an *S*-scan stack, the first scan's hexagonal points therefore received the shift *S* times over, and no point of any later scan received it at all. Single-scan imports were unaffected. + +**Root cause:** Bug. `copyRawEbsdData` computed the slab offset for the copies but the alignment helper it called neither took nor applied that offset. + +**Affected users:** Anyone importing more than one scan from an `.h5oina` file containing a hexagonal phase. Compounds with D1 and D2 on the same imports. + +**Correct behavior:** The alignment loop iterates the scan's own slab, so each point is visited exactly once. Pinned by `test/ReadH5OinaDataTest.cpp::"Multi-Scan Hexagonal Alignment"`, in which every point of both scans is hexagonal, so a shift applied to the wrong slab or applied twice changes a pinned value. + +**Recommendation:** Trust the corrected behavior. + +--- + +## ReadH5OinaDataFilter-D4 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D4` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved — as a limitation, honestly reported | + +**Symptom:** Turning on "Import Pattern Data" always failed, on every file, with "The parameter 'Read Pattern Data' has been enabled but there does not seem to be any pattern data in the file for the scan name selected" — including for files that plainly do contain pattern data. The archived production AZtec file, for instance, carries a 625 × 512 × 622 `Processed Patterns` dataset and still produced that message. + +**Root cause:** Bug in the reported reason, over a missing feature. `H5OINAReader::getPatternData()` returns `nullptr` unconditionally, `getPatternDims(std::array)` takes its argument **by value** with an empty body, and the reader's pattern-reading block is commented out. Preflight's pattern dimensions therefore stayed `{0, 0}` and the failure fired regardless of file content, attributing a missing library feature to the user's file. Behind that failure the execute-side plumbing was itself inconsistent: preflight created `Unprocessed Patterns` as `uint16` while execute fetched it as `UInt8Array`, which would have thrown had it ever been reached, and its copy loop used the tuple offset where it needed the element offset. The filter also only ever targeted `Unprocessed Patterns`, while real AZtec exports may carry only `Processed Patterns`. + +**Affected users:** Anyone who turned the parameter on. No user ever obtained pattern data from an `.h5oina` file through this filter. + +**Correct behavior:** The parameter is retained for pipeline compatibility, and preflight now reports the true reason — pattern import is not yet supported for H5OINA files — naming the file and pointing at the SimplnxCore **Read HDF5 Dataset** filter as the way to read the patterns a file does contain. The unreachable, inconsistent execute block and its error code `-34970` are removed. The parameter's help text and the filter documentation state the limitation. Pinned by `test/ReadH5OinaDataTest.cpp::"Parameter Rejections"`, section "Pattern Import Not Supported (-9583)". + +**Recommendation:** Trust the corrected behavior. Implementing H5OINA pattern import is a feature request against EbsdLib, not a fix. + +--- + +## ReadH5OinaDataFilter-D5 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D5` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | + +**Symptom:** The `LatticeConstants` ensemble array reported each phase's γ angle as a copy of its β angle. The file's third lattice angle was never read. For a cubic phase, where α = β = γ, the error is invisible; for a hexagonal phase, whose angles are 90/90/120, the reported γ was 90 instead of 120, and the same applies to any monoclinic, triclinic, trigonal or rhombohedral cell whose γ differs from its β. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readHeader()` assembled the six lattice constants as `{a, b, c, angles[0], angles[1], angles[1]}`. + +**Affected users:** Anyone importing an `.h5oina` file whose phases are not cubic or tetragonal, and who reads `LatticeConstants` downstream. It does not affect orientations, phase indices or crystal-structure symmetry, which come from the Laue group and not from the lattice angles. + +**Correct behavior:** The gamma slot receives the third angle. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Store the third H5OINA lattice angle in the gamma slot`). Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, whose hexagonal fixture phase has γ ≠ β specifically so that a gamma slot echoing beta is visible. + +**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still reports the duplicated angle. + +--- + +## ReadH5OinaDataFilter-D6 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D6` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | + +**Symptom:** The three angle slots of `LatticeConstants` were reported in radians for H5OINA imports and in degrees for every other EBSD importer. A cubic phase imported from an `.h5oina` file reported `1.5707964, 1.5707964, 1.5707964`, while the same phase imported from a `.ctf` or `.ang` file reported `90, 90, 90`. The array's meaning therefore depended on which file format the phase happened to come from, with nothing in the data to say which. + +**Root cause:** Library inconsistency, not a misreading. An H5OINA file stores its lattice angles in radians, and that is correct for the format; `H5OINAReader` copied them through unchanged, while the `.ang` and `.ctf` importers populate the same slots from degree-valued file fields. **The H5OINA files themselves are correct**; what differed was the unit contract at the importer boundary. + +**Affected users:** Anyone comparing or combining phase information across file formats, and anyone reading `LatticeConstants` from an H5OINA import while assuming the degrees convention that the rest of DREAM3D-NX uses. + +**Correct behavior:** `H5OINAReader` converts the angles from radians to degrees on import, on a double-precision intermediate, so the array carries the same unit no matter which format the phase came from. Corrected in EbsdLib on `topic/3_1_1_staging` (`ENH: Convert H5OINA lattice angles to degrees on import`). Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"` (90, 90, 120 for the hexagonal fixture phase) and `…::"Real AZtec File Readback"` (90, 90, 90 for the production file's titanium-cubic phase). + +**Recommendation:** Trust the corrected behavior — the degrees convention is the one the rest of the toolkit uses and the one the other importers already produced. Consumers that had compensated by converting H5OINA lattice angles themselves must stop doing so once running against EbsdLib 3.1.1. The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). + +--- + +## ReadH5OinaDataFilter-D7 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D7` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | + +**Symptom:** An `.h5oina` file whose phase group was missing its `Lattice Dimensions` or `Lattice Angles` dataset **crashed the process**. This is empirically demonstrated, not inferred: running the fixture through the batch-base build produced a SEGFAULT, recorded as `OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582) (SEGFAULT)` in `ww_work/ReadH5OinaData/red_baseline.log`. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readHeader()` read the two vector datasets while discarding their error codes, then indexed elements 0 through 2 of the resulting vectors, which are empty when the dataset is absent. The `Laue Group` and `Space Group` reads discarded their error codes as well, and a failed phase-group open was not checked at all. + +**Affected users:** Anyone opening a truncated, partially written or otherwise malformed `.h5oina` file. The crash occurs during preflight, so it takes down the application as the file is selected, before any pipeline runs. + +**Correct behavior:** The four required phase reads are checked and reported with EbsdLib error codes `-90030` through `-90033`, each naming the phase and the dataset; `Space Group` remains optional because the reader only passes it through. The filter surfaces the failure as `-9582` with the file path and the scan name. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Check the phase dataset reads in H5OINAReader::readHeader()`). Pinned by `test/ReadH5OinaDataTest.cpp::"EbsdLib Error Passthrough - Missing Lattice Angles (-9582)"`, which is the fixture that used to crash. + +**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still crashes on such a file. + +--- + +## Malformed-input rejections added alongside these entries + +These are not behavioral deviations on well-formed files — every one of them is a new rejection path that replaces an out-of-range read, an out-of-range write, or a silently useless output. They are listed here so a reader auditing the error-code series has one place to find them. + +| Code | Rejects | Previously | +|---|---|---| +| `-9584` | `X Cells` or `Y Cells` below 1 in the first selected scan | The counts were cast to `usize` unchecked, producing a zero-sized geometry, or an enormous one from a negative count | +| `-9585` | A selected scan whose grid or step sizes differ from the first selected scan's | Only the first scan's header was ever read; the copy then spanned the later scan's reader buffers using the first scan's point count | +| `-9586` | A selected scan name that is not in the file | Only the first name was checked, at preflight; a bad later name failed part-way through execute with the earlier scans already written into the output arrays and no rollback | +| `-9587` | Phase groups not numbered 1 through N | The ensemble arrays are sized from the phase count but each phase is placed at the index in its group name, so a group named `7` in a one-phase file wrote past the end of the ensemble arrays | +| `-34971` | A `Data` dataset whose extent disagrees with the header's cell counts | The reader sizes its buffers to the actual extent while the copy spans the header's point count, reading past the end of those buffers | +| `-34972` | A phase value outside `[0, phase count]` | The value indexed `CrystalStructures` unchecked in the alignment loop, reading past the end of the ensemble array | + +Error code `-34970` (null pattern data) is retired with the unreachable execute-side pattern block described in D4. diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md new file mode 100644 index 0000000000..e8fbf0b25e --- /dev/null +++ b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md @@ -0,0 +1,133 @@ +# Exemplar Archive Provenance: ReadH5OinaDataFilter + +This sidecar documents the test data behind `ReadH5OinaDataFilter` and the disposition +of each part of it. + +--- + +## Archive identity + +| Field | Value | +|---|---| +| **Archive** | `H5Oina_Test_Data.tar.gz` | +| **SHA512** | `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03` | +| **`download_test_data()` entry** | `src/Plugins/OrientationAnalysis/test/CMakeLists.txt:146` | +| **Contents** | `H5Oina_Test_Data.h5oina`, `H5Oina_Test_Data.dream3d`, `H5Oina_Test_Data.xdmf`, `Acknowledgements.md` | +| **Used by** | `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` (the `.h5oina` only) | +| **Changed?** | **No.** The archive is unchanged and was not re-uploaded; the SHA512 above is the one already in `CMakeLists.txt`. | + +## The input file: retained + +`H5Oina_Test_Data.h5oina` is a genuine Oxford Instruments AZtec export and is retained as +an irreplaceable piece of production realism. Its properties, read directly with h5py: + +| Property | Value | +|---|---| +| `Format Version` | `5.0` | +| Scans | one, named `1` | +| Grid | `X Cells` 25, `Y Cells` 25 (625 points), `X Step` = `Y Step` = 4.0 | +| Phases | one group, named `1`: "Titanium cubic", Laue group 11 (Cubic-High), space group 229 | +| Lattice | dimensions 3.192, 3.192, 3.192; angles 1.5707964 rad each (90 degrees) | +| Phase column | values {0, 1} — the file contains both indexed and unindexed points | +| Euler | float32 (625, 3) in radians, observed range [0, 6.2773] | +| Extra columns | `Beam Position X`/`Y`, `Detector Distance`, `Pattern Center X`/`Y`, `Pattern Quality`, and a 625 × 512 × 622 `Processed Patterns` dataset — none of which this filter reads | + +Recorded probe output: `ww_work/ReadH5OinaData/probe_real_file.txt`. + +Note that the file carries `Processed Patterns` and **no** `Unprocessed Patterns`, while the +filter only ever targeted the latter — one of the observations behind deviation D4. + +## The exemplar file: no longer consulted + +`H5Oina_Test_Data.dream3d` is **no longer used as a comparison target**. It remains inside +the archive because the archive is unchanged, but no test reads it. + +**Why.** The file was produced by running this filter on the sibling `.h5oina` in the same +archive: its cell arrays are bit-identical to the raw file data. It is therefore a +self-oracle. It is *not* a forbidden legacy oracle — no legacy H5OINA importer has ever +existed, so no version of DREAM3D could have generated it — but it pins "the filter keeps +doing what it did", not "the filter is right", which is the same failure mode. + +It also had the pre-correction behavior baked in: its `LatticeConstants` reads +`[3.192, 3.192, 3.192, 1.5707964, 1.5707964, 1.5707964]`, the radian angles of deviation +D6. A test comparing against it did not merely fail to detect D6 — it actively enforced it. + +Its discriminating power against the defects found in this work was near zero: the file has +a single cubic phase, so the hexagonal alignment (D1, D3) never executes; it holds one scan, +so the slab offsets (D2) are always zero; pattern import was off, so D4 and its latent type +mismatch were never reached; and its angles are all equal, so the gamma slot (D5) is +invisible. What it did pin — single-scan copy plumbing, geometry wiring, the phase widening +and the ensemble slot-0 defaults against a real-world file — is preserved and strengthened by +its replacement. + +## Replacement oracle + +**Toy fixtures, written by the test at run time.** `test/ReadH5OinaDataTest.cpp` declares a +fixture specification as C++ structs and writes `.h5oina` files with `H5Support::H5Lite` +into the binary test-output directory. Three fixture specifications (A, B, C) materialise +into nineteen files: nine positive-path files and ten guard files. Nothing is committed as +binary test data and no archive upload was needed. Each fixture carries exactly the dataset set `H5OINAReader` +requires, plus the inert root datasets for realism. + +Every expected value is derived from the fixture specification. The hexagonal-alignment +expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, +derived independently with IEEE-754 semantics in NumPy and embedded as literals with +derivation comments beside them. + +**Class 2 independent readback for the production file.** +`…::"Real AZtec File Readback"` compares all nine cell arrays element-wise against a readback +of the `.h5oina`'s own `Data` datasets performed with `H5Lite` — the file bytes, bypassing +`H5OINAReader` entirely — for 6,966 assertions. The geometry and the three ensemble arrays +are pinned as literals derived from the h5py readback. + +## Independent-derivation script + +| Field | Value | +|---|---| +| **Script** | `ww_work/ReadH5OinaData/h5oina_oracle.py` (not committed; archived to the V&V working-folder remote) | +| **Interpreter** | `/opt/local/anaconda3/envs/dream3d/bin/python`, h5py 3.16.0, NumPy | +| **Author** | Michael Jackson | +| **Date** | 2026-08-24 | +| **`spec` mode** | Prints the fixture specification and every derived expectation as C++-ready float32 literals. Recorded output: `oracle_spec.txt`. This is the source of the pinned hexagonal-alignment constants. | +| **`readback` mode** | Re-reads an `.h5oina` with h5py and re-derives the expected NX arrays from the documented rules, without EbsdLib or simplnx. Recorded output for the production file: `readback_real_file.txt`. | + +The script encodes the derivation rules read out of the filter and algorithm source — geometry +from the header, slab placement, verbatim copies, the phase widening, the Laue-group mapping, +the ensemble slot-0 defaults, the radians-to-degrees lattice-angle conversion and the +hexagonal φ2 alignment — and applies them to the fixture specification or to a file's bytes. +It never runs the filter. + +Supporting evidence in the same folder: + +| File | What it records | +|---|---| +| `pi_over_6_discriminator.txt` | The exhaustive sweep of float32 values in `[0.25, 6.5)` that identified which φ2 values separate a double-precision intermediate from a float32 one (8,289,627 of 38,797,312 do), and confirmation that `30 × k_PiOver180D` is bit-identical to the nearest double to π/6 | +| `probe_real_file.txt` | The h5py structure dump of the production `.h5oina` | +| `red_baseline.log` | The full test suite run at the batch base commit: 14 of 16 failing, including the SEGFAULT that is deviation D7 | +| `mutation_table.md` | Seven mutations, each killing exactly its claimed test cases, each reverted to an empty diff | +| `base_oa_suite.log` / `full_oa_suite.log` | The OrientationAnalysis suite at base sources and at branch head, used to establish that all 29 suite failures are pre-existing | +| `ebsdlib_suite.log`, `simplnxcore_suite.log` | The EbsdLib and SimplnxCore suite runs | + +## Oracle-before-comparison ordering + +There was no comparison to order against: DREAM3D 6.5.171 has no H5OINA importer. The oracle +was nonetheless established before any filter output was observed — the fixture +specification, the derivation script and every pinned literal were written and recorded +(`oracle_spec.txt`, timestamped before the first build) ahead of the first test run, and the +first run of the suite against the batch-base code is the RED baseline in `red_baseline.log`. +No expected value in the test file was taken from observed output. + +## Second-engineer oracle review + +Outstanding — to be recorded at PR review. The oracle design is fully auditable from the +committed test source: the fixture specification, the derivation of every pinned literal and +the reason each discriminating φ2 value was chosen are all stated in comments beside the +values. + +## Archive disposition + +`H5Oina_Test_Data.tar.gz` stays in the GitHub +[Data_Archive release](https://github.com/BlueQuartzSoftware/simplnx/releases/tag/Data_Archive) +and on the www.dream3d.io mirror unchanged, and its `download_test_data()` entry stays in +`test/CMakeLists.txt` because the `.h5oina` input is still needed. Should the archive ever be +regenerated, the `.dream3d` and `.xdmf` members can be dropped: nothing reads them. From cba45390af437d3a8a92609c1e7c506b238c5fc3 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 12:15:26 -0400 Subject: [PATCH 04/10] VV: Correct the H5OINA Error-column guidance and tighten the V&V wording The filter documentation carried the .ctf convention that Error = 0 marks a successfully indexed point. That is not true of this format. In the AZtec export bundled with this filter's tests, all 587 indexed points carry Error = 1 and all 38 un-indexed points carry Error = 2, and no point carries 0, so the masking recipe the page recommended would have selected nothing. The page now recommends thresholding Phase > 0, which is reliable because phase 0 is the reserved invalid-phase slot, and records the observation about Error. The V&V wording is tightened where a claim was looser than the evidence: - The fixture split is recounted from the files the test actually writes: nineteen at run time, eight imported successfully and eleven backing a rejection or passthrough case. The "carries exactly the minimum dataset set" statement is scoped to the eight that import, since two of the others omit a required dataset on purpose. - "Every fixture value is float32-exact" is replaced by what is actually relied on -- every value is a float32 literal stored as float32, so the file round-trips it bit-for-bit. Five of the phi2 values are deliberately not exactly representable decimals, because a dyadic value has trailing zero mantissa bits and cannot separate a double intermediate from a float32 one. - The hexagonal correction is 57.3 times the intended one, not 9.55. - The Euler destination offset was one third of the correct offset; the earlier phrasing implied a position within the destination slab. - The 6.5.171 search result names its single non-source match. Signed-off-by: Michael Jackson --- .../docs/ReadH5OinaDataFilter.md | 6 ++- .../test/ReadH5OinaDataTest.cpp | 46 +++++++++++++------ .../vv/ReadH5OinaDataFilter.md | 15 +++--- .../vv/deviations/ReadH5OinaDataFilter.md | 4 +- .../vv/provenance/ReadH5OinaDataFilter.md | 10 ++-- 5 files changed, 52 insertions(+), 29 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index 34cb087c0b..3f40383ea8 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -57,7 +57,9 @@ Historical reference frame operations for Oxford data are the following: + Sample Reference Frame: 180o about the <010> Axis + Crystal Reference Frame: None -The user also may want to assign un-indexed pixels to be ignored by flagging them as "bad". The [Multi-Threshold Objects](../SimplnxCore/MultiThresholdObjectsFilter.md) filter can be used to define this *mask* by thresholding on values such as *Error* = 0. +The user also may want to assign un-indexed pixels to be ignored by flagging them as "bad". The [Multi-Threshold Objects](../SimplnxCore/MultiThresholdObjectsFilter.md) filter can be used to define this *mask*. For H5OINA data, threshold on `Phase` > 0: an un-indexed point carries phase 0, which is the reserved invalid-phase slot. + +Do not assume `Error` = 0 marks a good point in an H5OINA file. AZtec writes an enumerated status code there whose values are not the same as the `.ctf` convention: in the AZtec export bundled with this filter's tests, every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0. A mask built from `Error` = 0 would select nothing on that file. ### Radians and Degrees @@ -105,7 +107,7 @@ The array names are the H5OINA dataset names, so several contain spaces. | `Band Contrast` | uint8 | 1 | | | `Band Slope` | uint8 | 1 | | | `Bands` | uint8 | 1 | | -| `Error` | uint8 | 1 | 0 marks a successfully indexed point | +| `Error` | uint8 | 1 | AZtec status code; see the note on masking below — do not assume 0 means "indexed" | | `Euler` | float32 | 3 | Radians; phi2 optionally shifted for Hexagonal-High points | | `Mean Angular Deviation` | float32 | 1 | | | `Phase` | int32 or uint8 | 1 | int32 by default; uint8 when *Convert Phase Data to Int32* is off. 0 marks an un-indexed point | diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 733973765d..14f5d824c1 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -29,17 +29,27 @@ * ww_work/ReadH5OinaData/h5oina_oracle.py (readback mode); * its recorded output is readback_real_file.txt. * - * Precision pinning (the ReadCtfData lesson): fixture values are float32-exact. + * Precision pinning (the ReadCtfData lesson): every value written into a toy + * fixture below is a float32 literal stored as float32, so the file round-trips it + * bit-for-bit and the verbatim copies are asserted with exact equality. (The + * production file's values are whatever AZtec wrote; that test asserts exact + * equality too, but against an independent readback of the file rather than + * against literals.) + * * The hexagonal alignment adds 30 degrees expressed in radians, computed with a - * double-precision intermediate and stored back as float32. Three fixture phi2 - * values (0.1F, 0.34F, and Fixture C's 0.09F/0.22F/0.35F) are chosen because - * their correctly-rounded float32 results DIFFER between a double-precision + * double-precision intermediate and stored back as float32. Five fixture phi2 + * values -- 0.1F and 0.34F, used in Fixture A and in Fixture C's first scan, and + * 0.09F, 0.22F and 0.35F in Fixture C's second scan -- are chosen because their + * correctly-rounded float32 results DIFFER between a double-precision * intermediate and a float32 intermediate, so the expectations pin the shape of - * the arithmetic and not merely its magnitude. 0.25F is carried alongside as a - * dyadic control whose result is identical under either intermediate. Every such - * literal was derived with IEEE-754 float32/float64 semantics in NumPy; the - * derivation lives in ww_work/ReadH5OinaData/h5oina_oracle.py and its recorded - * output oracle_spec.txt. + * the arithmetic and not merely its magnitude. Note that none of those five is an + * exactly representable decimal, and that is deliberate: a dyadic value has + * trailing zero mantissa bits, so adding the constant rounds to the same float32 + * either way and cannot separate the two paths. 0.25F, 0.5F and 0.75F are exactly + * representable and are carried alongside on hexagonal points as controls whose + * results are identical under either intermediate. Every such literal was derived + * with IEEE-754 float32/float64 semantics in NumPy; the derivation lives in + * ww_work/ReadH5OinaData/h5oina_oracle.py and its recorded output oracle_spec.txt. * * The archive's H5Oina_Test_Data.dream3d exemplar is no longer consulted: it was * generated by this very filter, so it is a self-oracle that pins "the filter @@ -94,12 +104,18 @@ constexpr int32 k_LaueHexagonalHigh = 9; constexpr int32 k_LaueCubicHigh = 11; //------------------------------------------------------------------------------ -// Toy .h5oina fixture description. The dataset set below is exactly the minimum -// H5OINAReader requires: the four Header scalars, one or more Phases/ groups -// carrying Phase Name / Lattice Dimensions / Lattice Angles / Laue Group / -// Space Group, and the nine Data datasets it reads unconditionally. The root -// Manufacturer / Software Version / Index datasets are written for realism only -// -- the reader never reads them. +// Toy .h5oina fixture description. Written out in full, the dataset set below is +// exactly the minimum H5OINAReader requires: the four Header scalars, one or more +// Phases/ groups carrying Phase Name / Lattice Dimensions / Lattice Angles / +// Laue Group / Space Group, and the nine Data datasets it reads unconditionally. +// The root Manufacturer / Software Version / Index datasets are written for +// realism only -- the reader never reads them. +// +// The guard fixtures deliberately depart from that minimum: `omitBandsDataset` +// and `omitLatticeAngles` drop a required dataset outright, and `groupName` +// overrides the phase index that the group name carries. The remaining guard +// fixtures keep the full dataset set but give it values a well formed file would +// not have. //------------------------------------------------------------------------------ struct PhaseSpec { diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md index ac3d8186dd..45aa839b7d 100644 --- a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -5,7 +5,7 @@ | Plugin | OrientationAnalysis | | SIMPLNX UUID | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | SIMPLNX Human Name | Read Oxford Aztec Data (.h5oina) | -| DREAM3D 6.5.171 equivalent | **None.** DREAM3D 6.5.171 has no H5OINA/AZtec importer of any kind (verified by a case-insensitive search of `DREAM3D/Source`, `SIMPL/Source` and `DREAM3D_Plugins` in the 6.5.171 tree: zero source hits). The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and carries no `FromSIMPLJson` and no legacy-UUID mapping entry. | +| DREAM3D 6.5.171 equivalent | **None.** DREAM3D 6.5.171 has no H5OINA/AZtec importer of any kind (verified by a case-insensitive search of `DREAM3D/Source`, `SIMPL/Source` and `DREAM3D_Plugins` in the 6.5.171 tree at `/Users/mjackson/Workspace/D3D_v6.5.171`: zero source hits — the one match is a byte sequence inside a ZeissImport sample `.bmp`). The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and carries no `FromSIMPLJson` and no legacy-UUID mapping entry. | | Verified commit | ** | | Status | READY FOR REVIEW | | Sign-off | Michael A. Jackson — 2026-08-24. Second engineer: **. | @@ -15,9 +15,9 @@ | Aspect | Current state | |------------------------|---------------| | Algorithm Relationship | **New filter, no legacy equivalent.** Nothing in DREAM3D 6.5.171 imports H5OINA, so there is no port to classify and no legacy behavior to inherit or defend. The filter is one of three siblings (`ReadH5OimData`, `ReadH5EspritData`) built on the shared `IEbsdOemReader` template; EbsdLib's `H5OINAReader` does the parsing. | -| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications (A, B, C) are written by the test itself with H5Lite and materialise into nineteen `.h5oina` files — nine positive-path files and ten guard files — and every expected value is derived from the fixture specification. The archived production AZtec file is compared against a readback of its own data sets. Encoded as 16 TEST_CASEs (8,890 assertions) in `test/ReadH5OinaDataTest.cpp`; all pass. SIMPLNX matched the oracle once the seven defects below were corrected. | +| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications (A, B, C) are written by the test itself with H5Lite and materialise into nineteen `.h5oina` files at run time — eight that are imported successfully and eleven that back a rejection or passthrough case — and every expected value is derived from the fixture specification. The archived production AZtec file is compared against a readback of its own data sets. Encoded as 16 TEST_CASEs (8,890 assertions) in `test/ReadH5OinaDataTest.cpp`; all pass. SIMPLNX matched the oracle once the seven defects below were corrected. | | Code paths enumerated | 21 of 26 paths exercised (see Code path coverage). The five gaps are two `-9582` return sites that share a statement with a covered row or need permission manipulation, the display-only preflight information values, the shared `-8971` empty-phase path (unreachable from this filter), and the cancel-signal early returns. | -| Tests today | 16 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a three-way Format Version sweep, nine value-add rejection cases with the error code pinned per section, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | +| Tests today | 16 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a three-way Format Version sweep, seven value-add rejection cases with the error code pinned per section, a stacking-order warning case, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | | Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained, SHA512 `346573ac…d140ea03`, unchanged.** Its genuine Oxford AZtec `.h5oina` is kept as an irreplaceable production input. Its `H5Oina_Test_Data.dream3d` exemplar is no longer consulted: that file was written by this filter, so comparing against it pinned the filter to itself. Documented in `vv/provenance/ReadH5OinaDataFilter.md`. | | Legacy comparison | **Not run — no legacy equivalent (verified against the 6.5.171 tree).** Its place is taken by the Class 2 independent readback described under Oracle. | | Bug flags | SIMPLNX, all releases 7.0.0 through 7.4.1: hexagonal φ2 shifted by 30 radians instead of 30 degrees (D1), multi-scan Euler slabs misplaced (D2), the hexagonal shift confined to the first scan and repeated (D3), pattern import advertised but impossible (D4), the third lattice angle discarded (D5), lattice angles left in radians while every other importer reports degrees (D6), and a crash on a phase group missing its lattice angles (D7). All resolved. | @@ -31,7 +31,7 @@ *Classification:* **New filter.** -*Evidence:* A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits. The `aztec` hits in that tree are the `H5Aztec` file-version constant belonging to DREAM3D's own HDF5-CTF archive format, which is unrelated to Oxford's H5OINA. The filter has no `FromSIMPLJson`, no `SIMPLConversion` include, no entry in the plugin's legacy-UUID mapping and no SIMPL conversion fixtures — all consistent with a filter that never existed in SIMPL. It was added by PR #700 (`a51dd5f3d`, 2024-03-25). +*Evidence:* A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits; the single match is a byte sequence inside `DREAM3D_Plugins/ZeissImport/Data/ZeissImport/SampleMosaic/SampleMosaic_p0.bmp`. The `aztec` hits in that tree are the `H5Aztec` file-version constant belonging to DREAM3D's own HDF5-CTF archive format, which is unrelated to Oxford's H5OINA. The filter has no `FromSIMPLJson`, no `SIMPLConversion` include, no entry in the plugin's legacy-UUID mapping and no SIMPL conversion fixtures — all consistent with a filter that never existed in SIMPL. It was added by PR #700 (`a51dd5f3d`, 2024-03-25). Because there is no legacy equivalent, the same-UUID equivalence claim that drives the Deviations gate does not apply; the Deviations file instead records the differences between what DREAM3D-NX 7.0.0–7.4.1 shipped and the corrected behavior. @@ -49,7 +49,7 @@ Three defects found during this work sit *inside* that boundary but corrupt user ### Applied -Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. Each fixture carries exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism. +Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. The eight fixtures that are imported successfully carry exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism. (`Format Version` is not in that required set, which is why the variant that omits it still imports.) Of the eleven fixtures behind the rejection and passthrough cases, two omit a required dataset outright — `Data/Bands` and `Phases/1/Lattice Angles` — while the other nine carry the full set and are rejected on their values or on how they are selected. - **Fixture A** — one scan, 3 × 2 cells, steps 0.25 / 0.5, a hexagonal phase (Laue 9) and a cubic phase (Laue 11), with Phase values `{1, 2, 0, 2, 1, 1}` so an unindexed point is present. Points 0, 1 and 2 all carry φ2 = `0.1F` on a hexagonal, a cubic and an unindexed point respectively, so the three expectations differ only through the alignment branch. The hexagonal phase's lattice angles are 90/90/120 degrees stored as radians, so γ ≠ β. - **Fixture B** — two scans, 2 × 2 each, cubic only, with disjoint values per scan. With no hexagonal point present the Euler array must be a pure verbatim copy, which isolates slab placement. @@ -57,9 +57,9 @@ Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into - **Guard fixtures** — zero / negative cell counts, mismatched scan grids, a missing scan name in first and second position, phase groups not numbered 1..N, an 8-row data set behind a 16-cell header, an out-of-range phase byte, a missing `Data/Bands`, and a phase group missing `Lattice Angles`. - **Format Version variants** — `"5.0"`, `"2.0"` and absent, which must all produce identical output. -Expected values are derived from the fixture specification, not from observed output. Verbatim copies are asserted with exact equality because every fixture value is float32-exact. The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy (`ww_work/ReadH5OinaData/h5oina_oracle.py`, recorded output `oracle_spec.txt`) and embedded as literals with derivation comments. +Expected values are derived from the fixture specification, not from observed output. Every value written into a toy fixture is a float32 literal stored as float32, so the file round-trips it bit-for-bit and verbatim copies are asserted with exact equality. (The production file's values are whatever AZtec wrote; that test asserts exact equality too, but against an independent readback of the file rather than against literals.) The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy (`ww_work/ReadH5OinaData/h5oina_oracle.py`, recorded output `oracle_spec.txt`) and embedded as literals with derivation comments. -**Precision pinning.** Following the ReadCtfData lesson, φ2 values were chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. `0.25F`, `0.5F` and `0.75F` are carried alongside as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, recorded in `pi_over_6_discriminator.txt`. +**Precision pinning.** Following the ReadCtfData lesson, φ2 values were chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. None of those five is an exactly representable decimal, which is deliberate — a dyadic value has trailing zero mantissa bits, so the sum rounds to the same float32 under either intermediate and cannot separate the two paths. `0.25F`, `0.5F` and `0.75F` are exactly representable and are carried alongside on hexagonal points as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, recorded in `pi_over_6_discriminator.txt`. Class 4 invariants encoded: the ensemble matrix always carries exactly one more tuple than the file has phase groups; slot 0 always holds `UnknownCrystalStructure` / `"Invalid Phase"` / zeroed lattice constants; the Phase values are unchanged by either conversion option; and the hexagonal shift never reaches a cubic or an unindexed point. @@ -83,6 +83,7 @@ Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflig - **Dead code:** the execute-side pattern block was unreachable — preflight always failed first — and internally inconsistent, creating a `uint16` array that execute fetched as `UInt8Array`. It is removed along with its error code `-34970`, and preflight now rejects the parameter honestly (D4). - **Progress and cancel:** the scan loop moved from the shared `IEbsdOemReader::execute()` into `ReadH5OinaData::operator()` so that per-scan progress messages and three cancel checks could be added without changing the sibling filters. The shared `readData()` is still used. - **Message quality:** `-9582` now carries the file path and the scan name; `-8970`'s message already carried both and is pinned by a test. +- **Documentation:** the page claimed a Format Version restriction the reader does not have, named the wrong Euler angle for the hexagonal alignment, and listed no created outputs. It also carried the `.ctf` convention that `Error` = 0 marks a good point, which is wrong for this format: in the bundled AZtec export every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0, so the masking recipe it recommended would have selected nothing. The page now recommends thresholding `Phase` > 0 and states the observation. Evidence: `ww_work/ReadH5OinaData/error_column_check.txt`. - **Not changed:** `utilities/IEbsdOemReader.hpp` is untouched, so `ReadH5OimData` and `ReadH5EspritData` are unaffected by this work. See Follow-ups. ## Code path coverage diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md index 28dcc871e6..99abf7b4e4 100644 --- a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -17,7 +17,7 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter | **Affected releases** | 7.0.0 through 7.4.1 | | **Status** | resolved | -**Symptom:** With "Convert Hexagonal X-Axis to EDAX Standard" on — its shipped default — every scan point whose phase maps to `Hexagonal_High` had 30.0 added to its φ2. The file's Euler angles are radians, so this added **thirty radians**, not thirty degrees. Thirty radians is 4.867 radians modulo 2π, so the resulting orientation bears no relation to either convention: the correction is 9.55 times too large and lands at an arbitrary angle. Every downstream product of those orientations — pole figures, IPF colors, misorientations, grain segmentation — was wrong for every hexagonal point of every H5OINA file imported at default settings. Cubic and unindexed points were unaffected. +**Symptom:** With "Convert Hexagonal X-Axis to EDAX Standard" on — its shipped default — every scan point whose phase maps to `Hexagonal_High` had 30.0 added to its φ2. The file's Euler angles are radians, so this added **thirty radians**, not thirty degrees. Thirty radians is 4.867 radians modulo 2π, so the resulting orientation bears no relation to either convention: the correction is 57.3 times the intended one (the ratio is exactly 180/π) and lands at an arbitrary angle. Every downstream product of those orientations — pole figures, IPF colors, misorientations, grain segmentation — was wrong for every hexagonal point of every H5OINA file imported at default settings. Cubic and unindexed points were unaffected. **Root cause:** Bug. The correction is a 30 degree rotation about [0001] applied to φ2, and the `.ctf` importer, whose files store degrees, correctly adds the literal `30.0`. The H5OINA path reused that literal even though H5OINA stores radians and the filter performs no degrees-to-radians conversion anywhere. @@ -38,7 +38,7 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter | **Affected releases** | 7.0.0 through 7.4.1 | | **Status** | resolved | -**Symptom:** When two or more scans were selected and stacked into one Image Geometry, the `Euler` array of every scan after the first landed at the wrong place. Scan *k* occupies the tuple slab starting at *k*·X·Y, so its Euler block belongs at element 3·*k*·X·Y; it was written at element *k*·X·Y instead — one third of the way into the slab, counted in tuples rather than elements. Scan 2's Euler block therefore overwrote the last two thirds of scan 1's Euler data, and the last two thirds of scan 2's own slab were left at their zero-initialized values. Every other cell array was placed correctly, so the corruption was confined to orientations and was silent: no error, no warning, and an output whose array sizes and geometry were all correct. Single-scan imports were unaffected, because the offset is zero. +**Symptom:** When two or more scans were selected and stacked into one Image Geometry, the `Euler` array of every scan after the first landed at the wrong place. Scan *k* occupies the tuple slab starting at *k*·X·Y, so its Euler block belongs at element 3·*k*·X·Y; it was written at element *k*·X·Y instead — one third of the correct offset, because the offset was counted in tuples rather than in elements. Scan 2's Euler block therefore overwrote the last two thirds of scan 1's Euler data, and the last two thirds of scan 2's own slab were left at their zero-initialized values. Every other cell array was placed correctly, so the corruption was confined to orientations and was silent: no error, no warning, and an output whose array sizes and geometry were all correct. Single-scan imports were unaffected, because the offset is zero. **Root cause:** Bug. The `Euler` copy correctly passes an element count of `totalPoints × 3` but passed the destination offset in tuples. The sibling `ReadH5OimData` writes the same interleave correctly as `(sliceTupleStart + i) * 3`. diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md index e8fbf0b25e..7786073eb4 100644 --- a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md @@ -65,9 +65,13 @@ its replacement. **Toy fixtures, written by the test at run time.** `test/ReadH5OinaDataTest.cpp` declares a fixture specification as C++ structs and writes `.h5oina` files with `H5Support::H5Lite` into the binary test-output directory. Three fixture specifications (A, B, C) materialise -into nineteen files: nine positive-path files and ten guard files. Nothing is committed as -binary test data and no archive upload was needed. Each fixture carries exactly the dataset set `H5OINAReader` -requires, plus the inert root datasets for realism. +into nineteen files at run time: eight that are imported successfully and eleven that back a +rejection or passthrough case. Nothing is committed as binary test data and no archive +upload was needed. The eight that import carry exactly the dataset set `H5OINAReader` +requires, plus the inert root datasets for realism. Of the eleven behind the rejection and +passthrough cases, two omit a required dataset outright -- `Data/Bands` and +`Phases/1/Lattice Angles` -- while the other nine carry the full set and are rejected on +their values or on how they are selected. Every expected value is derived from the fixture specification. The hexagonal-alignment expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, From a82e166f82733dab0b574746281374020e5fc7b0 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 13:05:41 -0400 Subject: [PATCH 05/10] VV: Validate the phase groups of every selected H5OINA scan The ensemble fill in the shared IEbsdOemReader::readData runs once per selected scan and writes crystalStructures[phaseId], materialNames[phaseId] and latticeConstants component phaseId, where phaseId is the integer in that scan's HDF5 phase group name. The three ensemble arrays are sized from the FIRST selected scan's phase count alone, so the phase-index guard has to hold for every selected scan and not only for the first. Two shapes of well formed multi-scan file reached that unguarded write: - A later scan carrying a phase group named outside 1..N. This is the same defect the first-scan guard already rejected, simply moved into scan 2. - A later scan declaring more phase groups than the first scan. This needs no malformed input at all: different scans of a real AZtec export may legitimately declare different phase lists. Both now fail preflight. The phase-index check is applied to every selected scan with the bound taken from the first scan's phase count, and a scan whose phase group count differs from the first scan's is rejected with the new code -9589, which also covers the case of a later scan declaring FEWER phases: the shared ensemble arrays would then describe that scan's phases while the earlier scan's points still referred to them. The dataset-extent probe's contract is stated accurately: it re-reads the file's extents because no reader API exposes the buffer sizes the reader allocated, and it rejects a dataset longer than the header describes as well as a shorter one. Its H5ScopedFileSentinel construction no longer casts away a const that the sentinel does not require, and the raw-dataset readback helper in the test no longer writes through a const_cast to a const hid_t, which was undefined behaviour on all nine of its call sites. Fixture B's second scan now differs from the first in its Phase, X and Y columns as well, so a slab-placement error in any of those columns is detectable across the two slabs rather than only in the columns that already differed. Signed-off-by: Michael Jackson --- .../Filters/Algorithms/ReadH5OinaData.cpp | 15 +- .../Filters/ReadH5OinaDataFilter.cpp | 40 ++++- .../test/ReadH5OinaDataTest.cpp | 158 +++++++++++++++--- 3 files changed, 174 insertions(+), 39 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp index 3e84b965b8..4b5d37aad3 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp @@ -41,18 +41,25 @@ const std::vector> k_RequiredDataSets = { * H5OINAReader sizes its buffers to whatever extent each dataset actually has, while * the geometry and the destination arrays are sized from the header's X Cells and * Y Cells. If a file's datasets are shorter than the header claims, copying - * totalPoints elements out of those buffers reads past their end. The extents are - * read straight from the file with H5Lite so no reader API is needed to expose them. + * totalPoints elements out of those buffers reads past their end. + * + * No reader API exposes the sizes the reader actually allocated, so the extents are + * re-probed from the file with H5Lite. That is the file's extents as of this call + * rather than the buffer sizes themselves; the two agree unless the file changed + * between H5OINAReader::readFile() and this call. A dataset LONGER than the header + * describes is rejected as well as a shorter one: the reader would size its buffers + * to the longer extent and the extra rows would be silently dropped, which is not a + * result a caller can distinguish from a correct import. */ Result<> validateDataSetExtents(const std::filesystem::path& filePath, const std::string& scanName, usize totalPoints) { - const hid_t fileId = H5Support::H5Utilities::openFile(filePath.string(), true); + hid_t fileId = H5Support::H5Utilities::openFile(filePath.string(), true); if(fileId < 0) { return MakeErrorResult( -34971, fmt::format("The file '{}' could not be reopened to verify the extents of scan '{}'. The file may have been moved or changed since preflight.", filePath.string(), scanName)); } - H5Support::H5ScopedFileSentinel sentinel(const_cast(fileId), false); + H5Support::H5ScopedFileSentinel sentinel(fileId, false); const std::string dataGroupPath = fmt::format("/{}/{}/{}", scanName, ebsdlib::H5OINA::EBSD, ebsdlib::H5OINA::Data); for(const auto& [dataSetName, componentCount] : k_RequiredDataSets) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp index b02517fced..0ce9d9f90a 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp @@ -187,18 +187,36 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure } // The Ensemble Attribute Matrix is sized from the number of phase groups in the - // file, but each phase is placed at the index carried by its group name. A file - // whose phase groups are not numbered 1..N would place a phase past the end of - // the ensemble arrays. + // FIRST selected scan, but the shared ensemble fill in IEbsdOemReader::readData runs + // once per selected scan and places each phase at the index carried by its HDF5 group + // name. Both properties below therefore have to hold for EVERY selected scan, not just + // the first: a later scan with a group named outside 1..N, or with more phase groups + // than the first scan, writes past the end of the ensemble arrays at execute. const auto phases = reader.getPhaseVector(); - for(const auto& phase : phases) - { - const int32 phaseIndex = phase->getPhaseIndex(); - if(phaseIndex < 1 || static_cast(phaseIndex) > phases.size()) + const usize ensemblePhaseCount = phases.size(); + const auto validatePhaseGroups = [&](const std::string& scanName, const auto& scanPhases) -> Result<> { + if(scanPhases.size() != ensemblePhaseCount) + { + return MakeErrorResult(-9589, fmt::format("Scan '{}' in '{}' declares {} phase group(s), but scan '{}' declares {}. Every selected scan must declare the same phase groups, because the single " + "Ensemble Attribute Matrix that all of the stacked scans share is sized and filled from those groups. Import scans with differing phase lists " + "separately.", + scanName, inputFilePath, scanPhases.size(), firstScanName, ensemblePhaseCount)); + } + for(const auto& phase : scanPhases) { - return MakePreflightErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares {} phase(s), but one of them carries index {}. The phase groups of an H5OINA file must be named 1 through {}.", - firstScanName, inputFilePath, phases.size(), phaseIndex, phases.size())); + const int32 phaseIndex = phase->getPhaseIndex(); + if(phaseIndex < 1 || static_cast(phaseIndex) > ensemblePhaseCount) + { + return MakeErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares {} phase(s), but one of them carries index {}. The phase groups of an H5OINA file must be named 1 through {}.", scanName, + inputFilePath, ensemblePhaseCount, phaseIndex, ensemblePhaseCount)); + } } + return {}; + }; + + if(Result<> phaseCheck = validatePhaseGroups(firstScanName, phases); phaseCheck.invalid()) + { + return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); } // Every other selected scan has to describe the same grid, because the geometry @@ -230,6 +248,10 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure scanName, inputFilePath, scanCheckReader.getXDimension(), scanCheckReader.getYDimension(), scanCheckReader.getXStep(), scanCheckReader.getYStep(), firstScanName, reader.getXDimension(), reader.getYDimension(), reader.getXStep(), reader.getYStep())); } + if(Result<> phaseCheck = validatePhaseGroups(scanName, scanCheckReader.getPhaseVector()); phaseCheck.invalid()) + { + return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); + } } } diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 14f5d824c1..70514c307b 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -24,17 +24,19 @@ * - Real AZtec file : Class 2 independent readback. The archived production * H5Oina_Test_Data.h5oina is compared against a readback of * its own Data group datasets performed with H5Lite, i.e. the - * raw file bytes, bypassing H5OINAReader entirely. The - * equivalent h5py readback is - * ww_work/ReadH5OinaData/h5oina_oracle.py (readback mode); - * its recorded output is readback_real_file.txt. + * raw file bytes, bypassing H5OINAReader entirely. An + * equivalent h5py readback in a second language is recorded + * in the V&V working folder, described in + * vv/provenance/ReadH5OinaDataFilter.md. * - * Precision pinning (the ReadCtfData lesson): every value written into a toy - * fixture below is a float32 literal stored as float32, so the file round-trips it - * bit-for-bit and the verbatim copies are asserted with exact equality. (The - * production file's values are whatever AZtec wrote; that test asserts exact - * equality too, but against an independent readback of the file rather than - * against literals.) + * Precision pinning (the ReadCtfData lesson): every floating-point value written + * into a toy fixture below is a float32 literal stored as float32, so the file + * round-trips it bit-for-bit and the verbatim copies are asserted with exact + * equality. (The cell counts and the Laue and space group numbers are int32 + * scalars and five of the nine data columns are uint8, all of which round-trip + * exactly as well. The production file's values are whatever AZtec wrote; that + * test asserts exact equality too, but against an independent readback of the + * file rather than against literals.) * * The hexagonal alignment adds 30 degrees expressed in radians, computed with a * double-precision intermediate and stored back as float32. Five fixture phi2 @@ -47,9 +49,12 @@ * trailing zero mantissa bits, so adding the constant rounds to the same float32 * either way and cannot separate the two paths. 0.25F, 0.5F and 0.75F are exactly * representable and are carried alongside on hexagonal points as controls whose - * results are identical under either intermediate. Every such literal was derived - * with IEEE-754 float32/float64 semantics in NumPy; the derivation lives in - * ww_work/ReadH5OinaData/h5oina_oracle.py and its recorded output oracle_spec.txt. + * results are identical under either intermediate. The hexagonal phase's third + * lattice angle, 2.0943952F -> 120.0F, is a sixth such discriminator: a float32 + * intermediate in the radians-to-degrees conversion would give 120.00000762939453F. + * Every such literal was derived with IEEE-754 float32/float64 semantics in NumPy; + * the derivation script and its recorded output are described in + * vv/provenance/ReadH5OinaDataFilter.md. * * The archive's H5Oina_Test_Data.dream3d exemplar is no longer consulted: it was * generated by this very filter, so it is a self-oracle that pins "the filter @@ -330,6 +335,13 @@ ScanSpec MakeFixtureBScan2() scan.error = {4, 5, 6, 7}; scan.euler = {2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F}; scan.mad = {1.125F, 1.25F, 1.375F, 1.5F}; + // Every column has to differ from scan 1's, or a slab-placement error in that + // column cannot be detected across the two slabs. The phase group set is kept + // identical because the ensemble arrays are shared by every stacked scan; only + // the per-point phase VALUES differ, using the reserved unindexed value 0. + scan.phase = {1, 0, 1, 0}; + scan.x = {1.0F, 1.25F, 1.0F, 1.25F}; + scan.y = {1.0F, 1.0F, 1.5F, 1.5F}; return scan; } @@ -457,12 +469,14 @@ void CompareStringArrayValues(const DataStructure& dataStructure, const DataPath template std::vector ReadRawDataset(const fs::path& filePath, const std::string& datasetPath) { - const hid_t fileId = H5Utilities::openFile(filePath.string(), true); + // H5Utilities::closeFile() takes its argument by non-const reference and assigns -1 to + // it, so this identifier cannot be const. + hid_t fileId = H5Utilities::openFile(filePath.string(), true); REQUIRE(fileId >= 0); std::vector data; const herr_t err = H5Lite::readVectorDataset(fileId, datasetPath, data); REQUIRE(err >= 0); - REQUIRE(H5Utilities::closeFile(const_cast(fileId)) >= 0); + REQUIRE(H5Utilities::closeFile(fileId) >= 0); return data; } } // namespace @@ -613,7 +627,9 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Multi-Scan Slab Placement" CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Bands), {1, 2, 3, 4, 5, 6, 7, 8}); CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Error), {0, 1, 2, 3, 4, 5, 6, 7}); CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::MeanAngularDeviation), {0.125F, 0.25F, 0.375F, 0.5F, 1.125F, 1.25F, 1.375F, 1.5F}); - CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 1, 1, 1, 1, 1, 1, 1}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), {1, 1, 1, 1, 1, 0, 1, 0}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::X), {0.0F, 0.25F, 0.0F, 0.25F, 1.0F, 1.25F, 1.0F, 1.25F}); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Y), {0.0F, 0.0F, 0.5F, 0.5F, 1.0F, 1.0F, 1.5F, 1.5F}); UnitTest::CheckArraysInheritTupleDims(dataStructure); } @@ -840,20 +856,43 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Missing Scan Name rejected // ensemble matrix is sized from the phase COUNT. A file whose groups are not // numbered 1..N is rejected instead of writing past the end of the ensemble // arrays. +// +// The ensemble fill in the shared IEbsdOemReader::readData runs once per SELECTED +// scan, so the check has to run for every selected scan and not only the first. +// The "Later selected scan" section is the same defect shape moved into scan "2": +// with the check applied to the first scan only, that file passes preflight and +// crashes the process at execute. //------------------------------------------------------------------------------ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Phase Index Out Of Range rejected (-9587)", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); - ScanSpec scan = MakeFixtureBScan1(); - scan.phases = {k_CubicPhase}; - scan.phases[0].groupName = "7"; // one phase, but it claims index 7 + std::vector scans; + std::list scanNames; + std::string fileName; + + SECTION("First selected scan") + { + ScanSpec scan = MakeFixtureBScan1(); + scan.phases[0].groupName = "7"; // one phase, but it claims index 7 + scans = {scan}; + scanNames = {"1"}; + fileName = "read_h5oina_vv_phase_index.h5oina"; + } + SECTION("Later selected scan") + { + ScanSpec scan2 = MakeFixtureBScan2(); + scan2.phases[0].groupName = "7"; + scans = {MakeFixtureBScan1(), scan2}; + scanNames = {"1", "2"}; + fileName = "read_h5oina_vv_phase_index_scan2.h5oina"; + } - const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_index.h5oina", {scan}); + const fs::path inputFile = WriteH5OinaFixture(fileName, scans); ReadH5OinaDataFilter filter; DataStructure dataStructure; - const Arguments args = MakeArgs(inputFile, {"1"}); + const Arguments args = MakeArgs(inputFile, scanNames); auto preflightResult = filter.preflight(dataStructure, args); SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); @@ -862,6 +901,48 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Phase Index Out Of Range r UnitTest::CheckArraysInheritTupleDims(dataStructure); } +//------------------------------------------------------------------------------ +// Value-add guard: the ensemble arrays are sized from the FIRST selected scan's +// phase count, and every selected scan fills them from its own phase groups. A +// later scan that declares more phase groups than the first therefore writes past +// the end of all three ensemble arrays, and a later scan that declares fewer +// silently leaves the first scan's phases in place under different definitions. +// Neither is a malformed file: different scans of a real multi-scan AZtec export +// may legitimately declare different phase lists. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Phase Count Mismatch rejected (-9589)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan1 = MakeFixtureBScan1(); + ScanSpec scan2 = MakeFixtureBScan2(); + std::string fileName; + + SECTION("Later scan declares more phases") + { + // Groups "1", "2" and "3" against the first scan's single group "1". + scan2.phases = {k_CubicPhase, k_HexPhase, k_CubicPhase}; + fileName = "read_h5oina_vv_phase_count_more.h5oina"; + } + SECTION("Later scan declares fewer phases") + { + scan1.phases = {k_CubicPhase, k_HexPhase}; + fileName = "read_h5oina_vv_phase_count_fewer.h5oina"; + } + + const fs::path inputFile = WriteH5OinaFixture(fileName, {scan1, scan2}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9589); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + //------------------------------------------------------------------------------ // Value-add guard: EbsdLib sizes its data buffers to the ACTUAL dataset extent // while the geometry is sized from the header's cell counts. A file whose Data @@ -966,25 +1047,50 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough // Error propagation from the trusted EbsdLib boundary at header-read time: a // phase group missing its Lattice Angles dataset is reported through -9582, and // the message carries the file path and scan name. +// +// The header of every selected scan is read at preflight, from two different +// return sites: the first selected scan's read and the per-scan loop's read. Both +// are pinned here, and each section asserts the scan name its own site injects. //------------------------------------------------------------------------------ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); - ScanSpec scan = MakeFixtureAScan(); - scan.phases[0].omitLatticeAngles = true; + std::vector scans; + std::list scanNames; + std::string fileName; + std::string expectedScanName; - const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_missing_angles.h5oina", {scan}); + SECTION("First selected scan") + { + ScanSpec scan = MakeFixtureAScan(); + scan.phases[0].omitLatticeAngles = true; + scans = {scan}; + scanNames = {"1"}; + fileName = "read_h5oina_vv_missing_angles.h5oina"; + expectedScanName = "'1'"; + } + SECTION("Later selected scan") + { + ScanSpec scan2 = MakeFixtureBScan2(); + scan2.phases[0].omitLatticeAngles = true; + scans = {MakeFixtureBScan1(), scan2}; + scanNames = {"1", "2"}; + fileName = "read_h5oina_vv_missing_angles_scan2.h5oina"; + expectedScanName = "'2'"; + } + + const fs::path inputFile = WriteH5OinaFixture(fileName, scans); ReadH5OinaDataFilter filter; DataStructure dataStructure; - const Arguments args = MakeArgs(inputFile, {"1"}); + const Arguments args = MakeArgs(inputFile, scanNames); auto preflightResult = filter.preflight(dataStructure, args); SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); REQUIRE(preflightResult.outputActions.errors()[0].code == -9582); REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); - REQUIRE(preflightResult.outputActions.errors()[0].message.find("'1'") != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(expectedScanName) != std::string::npos); UnitTest::CheckArraysInheritTupleDims(dataStructure); } From 451be31d8a077fb1b45dd4827f642f91b4165398 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 13:06:32 -0400 Subject: [PATCH 06/10] VV: Apply the H5OINA stacking order to the stacked scans The Stacking Order carried by the scan-selection parameter chooses which end of the selection list lands in tuple slab 0. This filter read it only to decide whether to raise a warning, and stacked the scans in list order either way, so High To Low was a setting the user could choose and the filter would not honour. The scan loop already lives in ReadH5OinaData::operator(), and stackingOrder already reaches the algorithm on ReadH5DataInputValues::SelectedScanNames, whose type OEMEbsdScanSelectionParameter::ValueType carries it. Honouring the setting is therefore a reversal of that loop's iteration order and needs no change to the shared IEbsdOemReader header, so the two sibling OEM readers are unaffected. Low To High reads the scans in the order they are listed; High To Low reads them in the reverse of that order, so the last selected scan occupies slab 0. Warning -9588, which told the user to reverse the scan selection by hand, is retired. Because the destination slab index no longer matches a scan's position in the selection list, the algorithm carries the name of the scan currently in the reader's buffers rather than re-deriving it by walking the selection list, which also makes the per-scan lookup constant time instead of linear. Signed-off-by: Michael Jackson --- .../docs/ReadH5OinaDataFilter.md | 15 +++-- .../Filters/Algorithms/ReadH5OinaData.cpp | 33 ++++++---- .../Filters/Algorithms/ReadH5OinaData.hpp | 11 ++++ .../Filters/ReadH5OinaDataFilter.cpp | 11 ---- .../test/ReadH5OinaDataTest.cpp | 65 ++++++++++++++----- 5 files changed, 89 insertions(+), 46 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index 3f40383ea8..ed1acb63bc 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -22,12 +22,15 @@ The file is EBSD (Electron Backscatter Diffraction) scan data. The most importan An H5OINA file can hold several scans. Selecting more than one stacks them into a single 3D **Image Geometry**: the X and Y extents and step sizes come from the first selected scan, the Z extent is the number of selected scans, and the Z spacing is the **Z Spacing** -parameter. Every selected scan must describe the same grid as the first one, and each must -be present in the file; the filter reports an error naming the offending scan otherwise. - -The scans are always stacked in the order they are listed. The **Stacking Order** setting -carried by the scan selection is not applied by this filter, and selecting *High To Low* -raises a warning; reverse the scan selection itself to change the stacking. +parameter. Every selected scan must describe the same grid as the first one and declare +the same phase groups, and each must be present in the file; the filter reports an error +naming the offending scan otherwise. Scans whose phase lists differ have to be imported +separately, because all of the stacked scans share one **Ensemble Attribute Matrix**. + +The **Stacking Order** setting carried by the scan selection chooses which end of the +list lands at Z = 0. *Low To High* stacks the scans in the order they are listed, so the +first selected scan is at Z = 0. *High To Low* stacks them in the reverse of that order, +so the last selected scan is at Z = 0. ### Limitations of the Filter diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp index 4b5d37aad3..cf82f80459 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp @@ -9,6 +9,7 @@ #include +#include #include #include #include @@ -149,18 +150,29 @@ Result<> ReadH5OinaData::operator()() imageGeom.setUnits(IGeometry::LengthUnit::Micrometer); // The scan loop is kept here rather than in IEbsdOemReader::execute() so that the - // cancel checks and the progress messages below apply to this filter only. - const usize scanCount = m_InputValues->SelectedScanNames.scanNames.size(); - int index = 0; - for(const auto& currentScanName : m_InputValues->SelectedScanNames.scanNames) + // cancel checks, the progress messages and the stacking order below apply to this + // filter only. + // + // The stacking order chooses which end of the selection list lands in tuple slab 0: + // Low-to-High reads the scans in the order they are listed, High-to-Low reads them in + // the reverse of that order, so the last selected scan occupies slab 0. + std::vector orderedScanNames(m_InputValues->SelectedScanNames.scanNames.cbegin(), m_InputValues->SelectedScanNames.scanNames.cend()); + if(m_InputValues->SelectedScanNames.stackingOrder == RefFrameZDir::k_HightoLow) + { + std::reverse(orderedScanNames.begin(), orderedScanNames.end()); + } + + const usize scanCount = orderedScanNames.size(); + for(usize index = 0; index < scanCount; index++) { if(m_ShouldCancel) { return {}; } - m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", currentScanName, index + 1, scanCount)}); - Result<> readResults = readData(currentScanName); + m_CurrentScanName = orderedScanNames[index]; + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", m_CurrentScanName, index + 1, scanCount)}); + Result<> readResults = readData(m_CurrentScanName); if(readResults.invalid()) { return readResults; @@ -171,14 +183,12 @@ Result<> ReadH5OinaData::operator()() return {}; } - m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", currentScanName, index + 1, scanCount)}); - Result<> copyDataResults = copyRawEbsdData(index); + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", m_CurrentScanName, index + 1, scanCount)}); + Result<> copyDataResults = copyRawEbsdData(static_cast(index)); if(copyDataResults.invalid()) { return copyDataResults; } - - ++index; } return {}; } @@ -191,8 +201,7 @@ Result<> ReadH5OinaData::copyRawEbsdData(int index) // Scan `index` occupies the tuple slab [index * totalPoints, (index + 1) * totalPoints). const usize tupleOffset = static_cast(index) * totalPoints; - const auto scanNameIterator = std::next(m_InputValues->SelectedScanNames.scanNames.cbegin(), index); - const std::string& scanName = *scanNameIterator; + const std::string& scanName = m_CurrentScanName; if(Result<> extentResults = validateDataSetExtents(m_InputValues->SelectedScanNames.inputFilePath, scanName, totalPoints); extentResults.invalid()) { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp index f0dbb9cf5d..fc27d488ed 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp @@ -3,6 +3,8 @@ #include "OrientationAnalysis/OrientationAnalysis_export.hpp" #include "OrientationAnalysis/utilities/IEbsdOemReader.hpp" +#include + namespace nx::core { @@ -26,6 +28,15 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaData : public IEbsdOemReader operator()(); Result<> copyRawEbsdData(int index) override; + +private: + /** + * @brief The scan name whose data is currently in the reader's buffers. The shared + * copyRawEbsdData(int) signature carries only the destination slab index, and under a + * High-to-Low stacking order that index no longer matches the scan's position in the + * selection list, so the name is carried here instead of being re-derived from it. + */ + std::string m_CurrentScanName; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp index 0ce9d9f90a..a2987e46a0 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp @@ -255,17 +255,6 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure } } - // The stacking order is carried by the scan-selection parameter and shared with - // the sibling OEM readers, but this filter always stacks the scans in the order - // they appear in the list. - if(pSelectedScanNamesValue.stackingOrder != RefFrameZDir::k_LowtoHigh) - { - resultOutputActions.warnings().push_back( - {-9588, fmt::format("The stacking order is set to High To Low, which this filter does not apply: the {} selected scans are always stacked in the order they are listed. Reverse the scan " - "selection itself to change the stacking.", - pSelectedScanNamesValue.scanNames.size())}); - } - // create the Image Geometry and it's attribute matrices const CreateImageGeometryAction::DimensionType dims = {static_cast(reader.getXDimension()), static_cast(reader.getYDimension()), pSelectedScanNamesValue.scanNames.size()}; const ShapeType tupleDims = {dims[2], dims[1], dims[0]}; diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 70514c307b..5ad8bf822e 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -1096,34 +1096,65 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough } //------------------------------------------------------------------------------ -// The scan-selection parameter carries a stacking order that this filter does -// not act on: scans are always stacked in the order they appear in the list. -// Selecting High-to-Low therefore raises a preflight warning rather than -// silently accepting a setting that has no effect. +// The stacking order carried by the scan-selection parameter chooses which end of +// the scan list lands in tuple slab 0. Low-to-High stacks the scans in the order +// they are listed; High-to-Low stacks them in the reverse of that order, so the +// LAST selected scan occupies slab 0. +// +// Fixture B's two scans are disjoint in every column, so each order is pinned by +// values that the other order cannot produce. The geometry is identical under both +// orders -- only the slab assignment changes. //------------------------------------------------------------------------------ -TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Stacking Order Warning (-9588)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Stacking Order", "[OrientationAnalysis][ReadH5OinaDataFilter]") { UnitTest::LoadPlugins(); const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_stacking.h5oina", {MakeFixtureBScan1(), MakeFixtureBScan2()}); - ReadH5OinaDataFilter filter; + const std::vector k_Scan1Euler = {0.125F, 0.25F, 0.375F, 0.5F, 0.625F, 0.75F, 0.875F, 1.0F, 1.125F, 1.25F, 1.375F, 1.5F}; + const std::vector k_Scan2Euler = {2.125F, 2.25F, 2.375F, 2.5F, 2.625F, 2.75F, 2.875F, 3.0F, 3.125F, 3.25F, 3.375F, 3.5F}; + + uint32 stackingOrder = RefFrameZDir::k_LowtoHigh; + std::vector expectedEuler; + std::vector expectedBandContrast; + std::vector expectedPhase; + SECTION("Low to High stacks the scans in list order") { - DataStructure dataStructure; - const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, RefFrameZDir::k_HightoLow); - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - REQUIRE(preflightResult.outputActions.warnings().size() == 1); - REQUIRE(preflightResult.outputActions.warnings()[0].code == -9588); + stackingOrder = RefFrameZDir::k_LowtoHigh; + expectedEuler = k_Scan1Euler; + expectedEuler.insert(expectedEuler.end(), k_Scan2Euler.cbegin(), k_Scan2Euler.cend()); + expectedBandContrast = {10, 11, 12, 13, 110, 111, 112, 113}; + expectedPhase = {1, 1, 1, 1, 1, 0, 1, 0}; } + SECTION("High to Low stacks the scans in reverse list order") { - DataStructure dataStructure; - const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, RefFrameZDir::k_LowtoHigh); - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - REQUIRE(preflightResult.outputActions.warnings().empty()); + stackingOrder = RefFrameZDir::k_HightoLow; + expectedEuler = k_Scan2Euler; + expectedEuler.insert(expectedEuler.end(), k_Scan1Euler.cbegin(), k_Scan1Euler.cend()); + expectedBandContrast = {110, 111, 112, 113, 10, 11, 12, 13}; + expectedPhase = {1, 0, 1, 0, 1, 1, 1, 1}; } + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}, 1.0F, true, true, false, stackingOrder); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.warnings().empty()); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ImageGeomPath)); + REQUIRE(dataStructure.getDataRefAs(k_ImageGeomPath).getDimensions() == SizeVec3(2, 2, 2)); + + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Euler), expectedEuler); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::BandContrast), expectedBandContrast); + CompareArrayValues(dataStructure, k_CellAMPath.createChildPath(ebsdlib::H5OINA::Phase), expectedPhase); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } //------------------------------------------------------------------------------ From 5220a1017c6759186c5609caa06d6fc47040742d Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 13:17:51 -0400 Subject: [PATCH 07/10] DOC: Record the H5OINA lattice-angle unit change as a migration note The lattice angles of an H5OINA import are reported in degrees, matching every other EBSD importer. DREAM3D-NX 7.0.0 through 7.4.1 reported them in radians, so the three angle slots of the LatticeConstants array change value for every H5OINA import: any .dream3d file, regression baseline or pipeline comparison produced by those releases reads a different number after the change. The page now says so where it describes the unit, and carries a Migration Notes section covering both that change and the hexagonal alignment correction, in the form the sibling ReadCtfData page uses. The Format Version paragraph is scoped to what is actually evidenced: the reader reads a fixed key and column set, which is the set Oxford documents for FORMAT VERSION 2.0 and which the bundled Format Version 5.0 export still carries. The earlier wording asserted that all later versions retain it. Signed-off-by: Michael Jackson --- .../docs/ReadH5OinaDataFilter.md | 36 ++++++++++++++++--- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index ed1acb63bc..635c392af6 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -34,11 +34,12 @@ so the last selected scan is at Z = 0. ### Limitations of the Filter -The filter reads the header keys and the nine data columns defined by **FORMAT VERSION 2.0** -of the H5OINA specification, which later versions retain. The file's `Format Version` value -is not used to select what is read, and a file without that value is read the same way. Any -column outside that set — for example `Pattern Quality`, `Beam Position X`/`Y` or the -`Electron Image` tree — is ignored, and can be brought in with the +The filter reads a fixed set of header keys and nine data columns, the set Oxford documents +for **FORMAT VERSION 2.0**. The file's `Format Version` value is not used to select what is +read, and a file without that value is read the same way, so a file imports exactly when it +carries that fixed set — the Format Version 5.0 export bundled with this filter's tests +still does. Any column outside the set — for example `Pattern Quality`, `Beam Position +X`/`Y` or the `Electron Image` tree — is ignored, and can be brought in with the [Read HDF5 Dataset](../SimplnxCore/ReadHDF5DatasetFilter.md) filter. **Importing diffraction patterns is not yet supported for H5OINA files.** Turning on @@ -75,6 +76,12 @@ so that the array means the same thing no matter which EBSD format the phase cam cubic phase therefore reports `90, 90, 90` rather than `1.5707964, 1.5707964, 1.5707964`. The three lattice dimensions are imported unchanged. +**This is a breaking change to a published output, and it is not in any released +version.** DREAM3D-NX 7.0.0 through 7.4.1 reported those three slots in radians for +H5OINA imports. See the migration notes below before comparing an H5OINA import against +a `.dream3d` file, a regression baseline, or a pipeline result produced by one of those +releases. + ### The Axis Alignment Issue for Hexagonal Symmetry [1] + The issue with hexagonal materials is the alignment of the Cartesian coordinate system used for calculations with the crystal coordinate system (the Bravais lattice). @@ -130,6 +137,25 @@ un-indexed points refer to. % Auto generated parameter table will be inserted here +## Migration Notes + +Documented behavioral differences from DREAM3D-NX 7.0.0 through 7.4.1 are maintained as +Deviation entries in the source tree at +`src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md`. Two of them +change values that an existing pipeline may compare against. + +- **`LatticeConstants` angle slots are degrees, not radians** (`ReadH5OinaDataFilter-D6`). + Every `.dream3d` file written by 7.0.0 through 7.4.1 carries radians in components 3, 4 + and 5 of that array, so any saved exemplar, regression baseline or pipeline comparison + that reads them changes value. Multiply a stored radian value by 180/pi to compare it + against a new import, and remove any downstream conversion that was compensating for + the radian values. Nothing else in the import changes unit: the `Euler` array is still + radians and the three lattice dimensions are still unconverted. +- **The hexagonal x-axis alignment adds 30 degrees, not 30 radians** + (`ReadH5OinaDataFilter-D1`). Orientations imported from a file with a hexagonal phase + by 7.0.0 through 7.4.1 with *Convert Hexagonal X-Axis to EDAX Standard* left on — its + default — are wrong and cannot be corrected after the fact; re-import the file. + ## Example Pipelines ## References From ee60d1b0d0bfeeb883efcfde6d1c8f0a277e6239 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 13:29:37 -0400 Subject: [PATCH 08/10] VV: State the H5OINA V&V deliverables against the shipped behavior The deliverables now record eleven deviations rather than seven. Four were missing: - D8, the multi-scan phase-group write. A selection whose scans do not all declare the same phase groups crashed the process, from a well formed file. The report previously claimed this filter's preflight prevented the unguarded ensemble write, which was true only for the first selected scan. - D9, the Stacking Order setting, which was accepted and never applied. - D10 and D11, the two H5OINAReader corrections that carried user-visible behavior changes -- every composed error message discarded, and readData() returning a code that did not match the one it set while never validating its column count -- neither of which had an entry. D6 is labelled a breaking change and carries a release note and migration paragraph: it changes the value of a published output array for every H5OINA import, and no released version carries it. Two coverage justifications were false. Row 9's later-scan -9582 return is a different statement in a different object from row 6's, not the same one, and row 4's note named the wrong rows. Row 9 is now covered by a test section that asserts the scan name that site injects, and row 4 is enumerated as the genuinely untested site it is. The table gains the second -34971 return, the extent probe's skip branch, the stacking-order path, the -9589 rejection and the three EbsdLib phase-read codes with no constructible fixture: 23 of 30, not 21 of 26. Every count is recounted against the sources: 23 fixture files of which eight import and fifteen back a rejection or passthrough; 17 test cases; five discriminating phi2 values and three dyadic controls, not eight discriminators; eight arrays beside Euler in the multi-scan comparison, not five; the assertion count given both as measured in one process and as summed over ctest's per-process runs, with the reason they differ. The claim that the OrientationAnalysis suite carries 29 pre-existing failures does not survive a full build of the preset directory. Twenty-eight of them were PIPELINE:: and PY:: tests, which run nxrunner and the Python bindings -- targets that building only the unit-test target leaves stale. After a full build the suite is 308 tests with one failure, ComputeSchmidsFilter, which is the known EbsdLib CubicOps precision drift, and SimplnxCore is 985 of 985. The provenance records that, and keeps the controlled EbsdLib rollback for what it does establish: rolling EbsdLib back to the batch base changes the non-H5OINA failure set not at all. The provenance sidecar states once that its evidence filenames live in an uncommitted working folder archived to the V&V remote, and no committed document cites a path under that folder or under .superpowers/. It also gains the Generated by / Generated on / canonical-oracle rows the provenance template requires, the EbsdLib and h5py/NumPy versions behind the trusted boundary, and a complete list of the suite logs. Wording throughout states what the code does rather than which pass changed it, per #1725; the exemplar bit-identity claim is scoped to the Euler array, which is what was measured. Signed-off-by: Michael Jackson --- .../vv/ReadH5OinaDataFilter.md | 122 ++++++++-------- .../vv/deviations/ReadH5OinaDataFilter.md | 104 ++++++++++++-- .../vv/provenance/ReadH5OinaDataFilter.md | 132 ++++++++++++++---- 3 files changed, 268 insertions(+), 90 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md index 45aa839b7d..86690a9e72 100644 --- a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -15,17 +15,17 @@ | Aspect | Current state | |------------------------|---------------| | Algorithm Relationship | **New filter, no legacy equivalent.** Nothing in DREAM3D 6.5.171 imports H5OINA, so there is no port to classify and no legacy behavior to inherit or defend. The filter is one of three siblings (`ReadH5OimData`, `ReadH5EspritData`) built on the shared `IEbsdOemReader` template; EbsdLib's `H5OINAReader` does the parsing. | -| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications (A, B, C) are written by the test itself with H5Lite and materialise into nineteen `.h5oina` files at run time — eight that are imported successfully and eleven that back a rejection or passthrough case — and every expected value is derived from the fixture specification. The archived production AZtec file is compared against a readback of its own data sets. Encoded as 16 TEST_CASEs (8,890 assertions) in `test/ReadH5OinaDataTest.cpp`; all pass. SIMPLNX matched the oracle once the seven defects below were corrected. | -| Code paths enumerated | 21 of 26 paths exercised (see Code path coverage). The five gaps are two `-9582` return sites that share a statement with a covered row or need permission manipulation, the display-only preflight information values, the shared `-8971` empty-phase path (unreachable from this filter), and the cancel-signal early returns. | -| Tests today | 16 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a three-way Format Version sweep, seven value-add rejection cases with the error code pinned per section, a stacking-order warning case, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | +| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications are written by the test itself with H5Lite and materialise into twenty-three `.h5oina` files at run time, and the archived production AZtec file is compared against a readback of its own data sets. Encoded as 17 TEST_CASEs in `test/ReadH5OinaDataTest.cpp`; all pass. | +| Code paths enumerated | 23 of 30 paths exercised (see Code path coverage). The seven gaps are one `-9582` return site needing permission manipulation, the display-only preflight information values, the shared `-8971` empty-phase path (unreachable from this filter), two defensive branches of the data-set extent probe, the cancel-signal early returns, and three EbsdLib phase-read codes with no constructible fixture. | +| Tests today | 17 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a stacking-order case, a three-way Format Version sweep, eight value-add rejection cases with the error code pinned per section, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | | Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained, SHA512 `346573ac…d140ea03`, unchanged.** Its genuine Oxford AZtec `.h5oina` is kept as an irreplaceable production input. Its `H5Oina_Test_Data.dream3d` exemplar is no longer consulted: that file was written by this filter, so comparing against it pinned the filter to itself. Documented in `vv/provenance/ReadH5OinaDataFilter.md`. | | Legacy comparison | **Not run — no legacy equivalent (verified against the 6.5.171 tree).** Its place is taken by the Class 2 independent readback described under Oracle. | -| Bug flags | SIMPLNX, all releases 7.0.0 through 7.4.1: hexagonal φ2 shifted by 30 radians instead of 30 degrees (D1), multi-scan Euler slabs misplaced (D2), the hexagonal shift confined to the first scan and repeated (D3), pattern import advertised but impossible (D4), the third lattice angle discarded (D5), lattice angles left in radians while every other importer reports degrees (D6), and a crash on a phase group missing its lattice angles (D7). All resolved. | -| V&V phase | Discovery, relationship, oracle, reconciliation, algorithm review (fixes applied), tests, deviations, provenance, docs — **complete**. Tests pass 16/16 in the EbsdLib preset build. **Release dependency:** the EbsdLib-side corrections (D5/D6/D7) live on `topic/3_1_1_staging` and reach users only through EbsdLib 3.1.1; the `vcpkg.json` pin is raised to `>= 3.1.1` and the pull request is merge-blocked until that release exists. OOC waived for this batch. Second-engineer sign-off outstanding (PR review). | +| Bug flags | D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11 — every one a bug under the root-cause taxonomy, every one present in DREAM3D-NX 7.0.0 through 7.4.1, all resolved. Two of them (D1, D6) change the value of a published output; D6 is labelled a breaking change. | +| V&V phase | Discovery, relationship, oracle, reconciliation, algorithm review, tests, deviations, provenance, docs — **complete**. The EbsdLib-side corrections (D5, D6, D7, D10, D11) reach users only through EbsdLib 3.1.1, which the `vcpkg.json` pin requires. In-core build and tests pass in `NX-Com-Qt69-Vtk96-Rel-EbsdLib`; OOC build skipped — the filter's writes are single-pass forward-sequential into freshly created arrays, and the plan for this batch scopes verification to in-core reader plumbing (approved in the batch plan for this filter). Second-engineer sign-off outstanding. | ## Summary -`ReadH5OinaDataFilter` ("Read Oxford Aztec Data (.h5oina)") imports one or more scans from an Oxford Instruments AZtec `.h5oina` file into a single Image Geometry: it builds the geometry from the first selected scan's header, creates the nine cell arrays and the three ensemble arrays, copies each scan's data into its own tuple slab, optionally widens the file's `uint8` Phase column to `int32`, and optionally applies the EDAX/TSL hexagonal x-axis alignment to φ2. Verification is Class 1 analytical plus Class 4 invariant on hand-authored `.h5oina` fixtures, with a Class 2 independent readback standing in for the legacy A/B comparison that cannot exist — DREAM3D 6.5.171 has no H5OINA importer. Headline result: seven defects were found, four in the filter and three in EbsdLib's `H5OINAReader`, including a crash and a silently wrong orientation for every hexagonal point at the shipped default settings; all are corrected and pinned, and all 16 tests pass. +`ReadH5OinaDataFilter` ("Read Oxford Aztec Data (.h5oina)") imports one or more scans from an Oxford Instruments AZtec `.h5oina` file into a single Image Geometry: it builds the geometry from the first selected scan's header, creates the nine cell arrays and the three ensemble arrays, copies each scan's data into its own tuple slab in the order the stacking-order setting asks for, optionally widens the file's `uint8` Phase column to `int32`, and optionally applies the EDAX/TSL hexagonal x-axis alignment to φ2. Verification is Class 1 analytical plus Class 4 invariant on hand-authored `.h5oina` fixtures, with a Class 2 independent readback standing in for the legacy A/B comparison that cannot exist — DREAM3D 6.5.171 has no H5OINA importer. Headline result: eleven defects were found, six in the filter and five in EbsdLib's `H5OINAReader`, including two crashes and a silently wrong orientation for every hexagonal point at the shipped default settings; all are corrected and pinned, and all 17 tests pass. ## Algorithm Relationship @@ -35,7 +35,7 @@ Because there is no legacy equivalent, the same-UUID equivalence claim that drives the Deviations gate does not apply; the Deviations file instead records the differences between what DREAM3D-NX 7.0.0–7.4.1 shipped and the corrected behavior. -*Material PRs since introduction:* #996 (OEM reader error messages), #1088 (parameter versioning), #1152 (spacing/origin ordering), #1263 (phase info in preflight values), #1438 (microtexture cleanup), #1472 (EbsdLib 2.0.0 API migration), #1576 (error-message sweep). None of them touched the hexagonal-alignment constant, the multi-scan offsets or the pattern path; those three carried their defects from the initial import through every subsequent change. +*PRs since introduction that touched these files:* the full list between `a51dd5f3d` and `d65d859b5` is #874, #934, #937, #996, #1088, #1152, #1187, #1238, #1263, #1438, #1439, #1472 and #1576. The ones that changed behavior rather than form are #996 (OEM reader error messages), #1088 (parameter versioning), #1152 (spacing/origin ordering), #1263 (phase info in preflight values), #1472 (EbsdLib 2.0.0 API migration) and #1576 (error-message sweep). None of them touched the hexagonal-alignment constant, the multi-scan offsets or the pattern path; those three carried their defects from the initial import through every subsequent change. ## Oracle @@ -43,81 +43,86 @@ Because there is no legacy equivalent, the same-UUID equivalence claim that driv ### The EbsdLib boundary (what we do NOT re-test) -EbsdLib's `H5OINAReader` owns HDF5 traversal, the header and phase-group parsing, the nine required data-set reads, the Laue-group-to-crystal-structure mapping in `CtfPhase`, and its own error codes. Those behaviors are upstream's to verify, and the tests pin only that their codes and messages reach the user. The filter's value-add — everything this oracle covers — is the deterministic plumbing on top: geometry construction from the first scan's header, array creation and typing, per-scan slab offsets, the verbatim column copies, the `uint8`→`int32` Phase widening, the hexagonal φ2 alignment, ensemble slot-0 defaults, and the value-add rejection paths. +EbsdLib's `H5OINAReader` owns HDF5 traversal, the header and phase-group parsing, the nine required data-set reads, the Laue-group-to-crystal-structure mapping in `CtfPhase`, and its own error codes. Those behaviors are upstream's to verify, and the tests pin only that their codes and messages reach the user. The filter's value-add — everything this oracle covers — is the deterministic plumbing on top: geometry construction from the first scan's header, array creation and typing, per-scan slab offsets and their stacking order, the verbatim column copies, the `uint8`→`int32` Phase widening, the hexagonal φ2 alignment, ensemble slot-0 defaults, and the value-add rejection paths. -Three defects found during this work sit *inside* that boundary but corrupt user-visible SIMPLNX output or crash the process (D5, D6, D7). They were corrected upstream on `topic/3_1_1_staging` rather than worked around in the filter, which is what creates the EbsdLib 3.1.1 release dependency. +Five of the eleven deviations sit *inside* that boundary but corrupt user-visible SIMPLNX output, crash the process, or blank out the reason a file was rejected (D5, D6, D7, D10, D11). They are corrected upstream on `topic/3_1_1_staging` rather than worked around in the filter, which is what creates the EbsdLib 3.1.1 release dependency. ### Applied -Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. The eight fixtures that are imported successfully carry exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism. (`Format Version` is not in that required set, which is why the variant that omits it still imports.) Of the eleven fixtures behind the rejection and passthrough cases, two omit a required dataset outright — `Data/Bands` and `Phases/1/Lattice Angles` — while the other nine carry the full set and are rejected on their values or on how they are selected. +Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. Three fixture specifications materialise into twenty-three files at run time: **eight** that are imported successfully and **fifteen** that back a rejection or passthrough case. The eight that import carry exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism, and all but one of them also carry `Format Version`, which is not in the required set (which is why the variant that omits it still imports). Of the fifteen behind the rejection and passthrough cases, three omit a required dataset outright — `Data/Bands` once and `Phases//Lattice Angles` twice — while the other twelve carry the full dataset set and are rejected on a parameter value, on a header or phase value, or on how the scans are selected. - **Fixture A** — one scan, 3 × 2 cells, steps 0.25 / 0.5, a hexagonal phase (Laue 9) and a cubic phase (Laue 11), with Phase values `{1, 2, 0, 2, 1, 1}` so an unindexed point is present. Points 0, 1 and 2 all carry φ2 = `0.1F` on a hexagonal, a cubic and an unindexed point respectively, so the three expectations differ only through the alignment branch. The hexagonal phase's lattice angles are 90/90/120 degrees stored as radians, so γ ≠ β. -- **Fixture B** — two scans, 2 × 2 each, cubic only, with disjoint values per scan. With no hexagonal point present the Euler array must be a pure verbatim copy, which isolates slab placement. +- **Fixture B** — two scans, 2 × 2 each, cubic only, with every one of the nine columns disjoint between the two scans. With no hexagonal point present the Euler array must be a pure verbatim copy, which isolates slab placement and stacking order. - **Fixture C** — two scans, 2 × 2 each, every point hexagonal, so a shift applied to the wrong slab or applied twice changes pinned values. -- **Guard fixtures** — zero / negative cell counts, mismatched scan grids, a missing scan name in first and second position, phase groups not numbered 1..N, an 8-row data set behind a 16-cell header, an out-of-range phase byte, a missing `Data/Bands`, and a phase group missing `Lattice Angles`. +- **Guard fixtures** — zero / negative cell counts, mismatched scan grids, a missing scan name in first and second position, a phase group named outside 1..N in the first and in a later scan, a later scan whose phase-group count differs from the first scan's in both directions, an 8-row data set behind a 16-cell header, an out-of-range phase byte, a missing `Data/Bands`, and a phase group missing `Lattice Angles` in the first and in a later scan. - **Format Version variants** — `"5.0"`, `"2.0"` and absent, which must all produce identical output. -Expected values are derived from the fixture specification, not from observed output. Every value written into a toy fixture is a float32 literal stored as float32, so the file round-trips it bit-for-bit and verbatim copies are asserted with exact equality. (The production file's values are whatever AZtec wrote; that test asserts exact equality too, but against an independent readback of the file rather than against literals.) The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy (`ww_work/ReadH5OinaData/h5oina_oracle.py`, recorded output `oracle_spec.txt`) and embedded as literals with derivation comments. +Expected values are derived from the fixture specification, not from observed output. Every floating-point value written into a toy fixture is a float32 literal stored as float32, so the file round-trips it bit-for-bit and verbatim copies are asserted with exact equality; the cell counts and the Laue and space group numbers are `int32` scalars and five of the nine data columns are `uint8`, all of which round-trip exactly as well. (The production file's values are whatever AZtec wrote; that test asserts exact equality too, but against an independent readback of the file rather than against literals.) The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy and embedded as literals with derivation comments. The derivation script and its recorded output are described in `vv/provenance/ReadH5OinaDataFilter.md`. -**Precision pinning.** Following the ReadCtfData lesson, φ2 values were chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. None of those five is an exactly representable decimal, which is deliberate — a dyadic value has trailing zero mantissa bits, so the sum rounds to the same float32 under either intermediate and cannot separate the two paths. `0.25F`, `0.5F` and `0.75F` are exactly representable and are carried alongside on hexagonal points as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, recorded in `pi_over_6_discriminator.txt`. +**Precision pinning.** φ2 values are chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. None of those five is an exactly representable decimal, which is deliberate — a dyadic value has trailing zero mantissa bits, so the sum rounds to the same float32 under either intermediate and cannot separate the two paths. `0.25F`, `0.5F` and `0.75F` are exactly representable and are carried alongside on hexagonal points as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The hexagonal fixture phase's third lattice angle is a sixth discriminator of the same kind, on the EbsdLib radians-to-degrees conversion rather than on the alignment: `2.0943952F → 120.0F` under a double intermediate and `120.00000762939453F` under a float32 one. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, of which 8,289,627 of 38,797,312 separate the two paths. -Class 4 invariants encoded: the ensemble matrix always carries exactly one more tuple than the file has phase groups; slot 0 always holds `UnknownCrystalStructure` / `"Invalid Phase"` / zeroed lattice constants; the Phase values are unchanged by either conversion option; and the hexagonal shift never reaches a cubic or an unindexed point. +Class 4 invariants encoded: the ensemble matrix always carries exactly one more tuple than the file has phase groups; slot 0 always holds `UnknownCrystalStructure` / `"Invalid Phase"` / zeroed lattice constants; the Phase values are unchanged by either conversion option; the hexagonal shift never reaches a cubic or an unindexed point; and the geometry is identical under both stacking orders. ### The Class 2 independent readback (substitute for the legacy A/B) There is no DREAM3D 6.5.171 H5OINA importer, so the comparison that would normally establish behavioral continuity does not exist. Its place is taken by an independent readback of the production AZtec file: - `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` reads the archived `H5Oina_Test_Data.h5oina`'s own `Data` datasets with `H5Lite` — the file bytes, bypassing `H5OINAReader` entirely — and compares them element-wise against the filter's output. The file is a single 25 × 25 cubic scan with 625 points, both indexed and unindexed, and the comparison covers all nine cell arrays (6,966 assertions in that test case). -- `ww_work/ReadH5OinaData/h5oina_oracle.py` performs the equivalent readback with h5py in a second language and a second HDF5 binding, re-deriving the geometry, the ensemble values and the cell arrays from the documented rules. Its recorded output is `readback_real_file.txt`; it is the source of the ensemble literals pinned in that test case (`CrystalStructures {999, 1}`, `MaterialName {"Invalid Phase", "Titanium cubic"}`, `LatticeConstants {3.192, 3.192, 3.192, 90, 90, 90}`). +- The same readback is performed out of band with h5py, in a second language and a second HDF5 binding, re-deriving the geometry, the ensemble values and the cell arrays from the documented rules. It is the source of the ensemble literals pinned in that test case (`CrystalStructures {999, 1}`, `MaterialName {"Invalid Phase", "Titanium cubic"}`, `LatticeConstants {3.192, 3.192, 3.192, 90, 90, 90}`). The script and its recorded output are described in `vv/provenance/ReadH5OinaDataFilter.md`. -This file cannot exercise the hexagonal alignment (its only phase is cubic) or the multi-scan slab offsets (it holds one scan); the toy fixtures carry those paths. +This file cannot exercise the hexagonal alignment (its only phase is cubic), the multi-scan slab offsets or the stacking order (it holds one scan); the toy fixtures carry those paths. -*Second-engineer review:* Outstanding — to be recorded at PR review. The oracle design is auditable in the test source: the fixture specification, the derivation of every literal and the Python cross-check are all committed or recorded in the evidence folder. +*Second-engineer review:* Outstanding — to be recorded at PR review. The oracle design is auditable in the test source: the fixture specification, the derivation of every literal and the Python cross-check are all committed or recorded in the working folder described in the provenance sidecar. ## Algorithm review -Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflightImpl` after oracle reconciliation. All findings applied; all 16 tests pass afterwards. +Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflightImpl`. -- **Correctness:** the hexagonal alignment now adds 30 degrees expressed in radians on a double intermediate (D1); the Euler slab offset is now an element offset rather than a tuple offset (D2); the alignment loop now walks the scan's own slab (D3). -- **Robustness:** six malformed-input rejections added (`-9584`, `-9585`, `-9586`, `-9587`, `-34971`, `-34972`), each naming the offending value, the scan and the file. -- **Dead code:** the execute-side pattern block was unreachable — preflight always failed first — and internally inconsistent, creating a `uint16` array that execute fetched as `UInt8Array`. It is removed along with its error code `-34970`, and preflight now rejects the parameter honestly (D4). -- **Progress and cancel:** the scan loop moved from the shared `IEbsdOemReader::execute()` into `ReadH5OinaData::operator()` so that per-scan progress messages and three cancel checks could be added without changing the sibling filters. The shared `readData()` is still used. -- **Message quality:** `-9582` now carries the file path and the scan name; `-8970`'s message already carried both and is pinned by a test. -- **Documentation:** the page claimed a Format Version restriction the reader does not have, named the wrong Euler angle for the hexagonal alignment, and listed no created outputs. It also carried the `.ctf` convention that `Error` = 0 marks a good point, which is wrong for this format: in the bundled AZtec export every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0, so the masking recipe it recommended would have selected nothing. The page now recommends thresholding `Phase` > 0 and states the observation. Evidence: `ww_work/ReadH5OinaData/error_column_check.txt`. -- **Not changed:** `utilities/IEbsdOemReader.hpp` is untouched, so `ReadH5OimData` and `ReadH5EspritData` are unaffected by this work. See Follow-ups. +- **Correctness:** the hexagonal alignment adds 30 degrees expressed in radians on a double intermediate (D1); the Euler slab offset is an element offset rather than a tuple offset (D2); the alignment loop walks the scan's own slab (D3); the scan iteration honors the stacking order (D9). +- **Robustness:** seven malformed-input rejections (`-9584`, `-9585`, `-9586`, `-9587`, `-9589`, `-34971`, `-34972`), each naming the offending value, the scan and the file. The phase-group checks run for every selected scan, not only the first, because the ensemble arrays are sized from the first scan and filled from all of them (D8). +- **Dead code:** the execute-side pattern block was unreachable — preflight always failed first — and internally inconsistent, creating a `uint16` array that execute fetched as `UInt8Array`. It is removed along with its error code `-34970`, and preflight rejects the parameter honestly (D4). +- **Progress and cancel:** the scan loop lives in `ReadH5OinaData::operator()` rather than in the shared `IEbsdOemReader::execute()`, which is what lets the per-scan progress messages, the three cancel checks and the stacking order apply to this filter without changing the sibling filters. The shared `readData()` is still used. +- **Message quality:** the two `-9582` sites that name a scan carry the file path and the scan name. The third, on `readScanNames()`, carries the file path only, because no scan has been selected at that point. `-8970`'s message carries both and is pinned by a test. +- **Preflight cost:** preflight performs one `readScanNames` open, one first-scan header open and one further open per additional selected scan, and preflight fires on every GUI parameter edit. That is `O(S)` file opens per keystroke for a selection of `S` scans. It is the price of validating every selected scan's header rather than only the first, and it is bounded by the number of scans a file contains. +- **Documentation:** the page states that the reader ignores the file's `Format Version` value, names φ2 as the third Euler angle the hexagonal alignment shifts, lists the created outputs with their types and component counts, and recommends thresholding `Phase` > 0 to mask unindexed points. That last point is format-specific: the `.ctf` convention that `Error` = 0 marks a good point does not hold here. In the bundled AZtec export every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0, so an `Error` = 0 mask selects nothing; the page says so. A Migration Notes section covers the two deviations that change a value a saved pipeline may compare against. +- **Not changed:** `utilities/IEbsdOemReader.hpp` is untouched, so `ReadH5OimData` and `ReadH5EspritData` are unaffected. See Follow-ups. ## Code path coverage -*21 of 26 enumerated paths exercised. Source: `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp` (256 lines) + preflight in `Filters/ReadH5OinaDataFilter.cpp` (317 lines).* Logical phases: **(a)** preflight, **(b)** execute read + ensemble population (shared `IEbsdOemReader::readData`), **(c)** per-scan cell-data copy. +*23 of 30 enumerated paths exercised. Source: `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp` (272 lines) + preflight in `Filters/ReadH5OinaDataFilter.cpp` (328 lines) + the shared read and ensemble fill in `utilities/IEbsdOemReader.hpp` (147 lines), which rows 16–18 live in.* Logical phases: **(a)** preflight, **(b)** execute read + ensemble population (shared `IEbsdOemReader::readData`), **(c)** per-scan cell-data copy. | # | Phase | Path | Test case | |----|-------|------|-----------| | 1 | (a) | `z_spacing <= 0` → `-9580` | `Parameter Rejections` (section "Non-positive Z Spacing") | | 2 | (a) | empty scan-name list → `-9581` | `Parameter Rejections` (section "No Scan Names Selected") | | 3 | (a) | `read_pattern_data` on → `-9583` | `Parameter Rejections` (section "Pattern Import Not Supported") | -| 4 | (a) | `readScanNames` failure → `-9582` | *Not directly tested — needs an unreadable-but-present file (permission manipulation). The same `-9582` return statement family is covered by rows 5 and 6.* | +| 4 | (a) | `readScanNames` failure → `-9582` | *Not directly tested — the file has to be present but unreadable, which needs permission manipulation. This is a distinct return statement from rows 6 and 9, and it is the one `-9582` site whose message carries the file path without a scan name.* | | 5 | (a) | selected scan absent from the file → `-9586` | `Missing Scan Name rejected (-9586)` (sections: missing first scan; missing second scan) | -| 6 | (a) | first scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | +| 6 | (a) | first scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` (section "First selected scan") | | 7 | (a) | `X Cells`/`Y Cells` < 1 → `-9584` | `Invalid Cell Counts rejected (-9584)` (sections: zero X; zero Y; negative X) | -| 8 | (a) | phase index outside `[1, phase count]` → `-9587` | `Phase Index Out Of Range rejected (-9587)` | -| 9 | (a) | later scan `readHeaderOnly` failure → `-9582` | *Not separately tested — same return statement as row 6, reached from the per-scan loop.* | +| 8 | (a) | phase group of any selected scan named outside `[1, first scan's phase count]` → `-9587` | `Phase Index Out Of Range rejected (-9587)` (sections: first selected scan; later selected scan) | +| 9 | (a) | later scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` (section "Later selected scan"; asserts the scan name this site injects) | | 10 | (a) | later scan grid differs from the first → `-9585` | `Scan Header Mismatch rejected (-9585)` (sections: differing cell counts; differing step size) | -| 11 | (a) | stacking order not Low-to-High → warning `-9588` | `Stacking Order Warning (-9588)` (both branches: warning raised, and no warning for Low-to-High) | +| 11 | (a) | later scan phase-group count differs from the first → `-9589` | `Scan Phase Count Mismatch rejected (-9589)` (sections: more phases; fewer phases) | | 12 | (a) | geometry action: dims (X, Y, scan count), spacing (X Step, Y Step, z_spacing), user origin | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement`; `Multi-Scan Hexagonal Alignment` | | 13 | (a) | ensemble matrix sized phase count + 1; three ensemble array actions | `Class 1 Analytical Oracle` (3 tuples, 2 phases) | | 14 | (a) | nine cell-array actions with the Phase type chosen by `convert_phase_to_int32` | `Conversion Option Combinations` (both branches assert the Phase array's type) | -| 15 | (a) | preflight scan/phase information values | *Exercised implicitly by every preflight; display-only, not asserted.* | +| 15 | (a) | preflight scan/phase information values | *Not directly tested — display-only values, exercised implicitly by every preflight and asserted by none.* | | 16 | (b) | `readFile()` failure → `-8970` | `EbsdLib Error Passthrough - Missing Data Column (-8970)` (code and message content pinned) | | 17 | (b) | empty phase vector → `-8971` | *Not directly tested — `H5OINAReader` rejects a file with no phase groups at header-read time with `-90009`, so preflight `-9582` fires first and this shared-code path is unreachable from this filter.* | | 18 | (b) | ensemble slot-0 defaults and per-phase fill (Laue mapping, name, lattice constants) | `Class 1 Analytical Oracle`; `Real AZtec File Readback` | -| 19 | (c) | data-set extent disagrees with the header → `-34971` | `Dataset Extent Mismatch rejected (-34971)` | -| 20 | (c) | four `uint8` verbatim copies into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | -| 21 | (c) | Euler copy at three times the tuple offset | `Multi-Scan Slab Placement` (24 values across two slabs) | -| 22 | (c) | phase value outside `[0, phase count]` → `-34972` | `Out-of-Range Phase Value rejected (-34972)` | -| 23 | (c) | Phase widened to `int32` / copied verbatim as `uint8` | `Conversion Option Combinations` (both branches) | -| 24 | (c) | three `float32` verbatim copies (MAD, X, Y) into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | -| 25 | (c) | hexagonal alignment on the scan's own slab, Hexagonal-High points only | `Class 1 Analytical Oracle`; `Conversion Option Combinations`; `Multi-Scan Hexagonal Alignment` | -| 26 | (b)/(c) | cancel checks (3 sites) | *Not directly tested. Requires cancel-signal injection; standard early-return pattern. Excluded from scope by direction.* | +| 19 | (b)/(c) | stacking order: Low-to-High reads the selection in list order, High-to-Low in reverse | `Stacking Order` (both sections; each pins values only its own order can produce) | +| 20 | (c) | the file cannot be reopened for the extent probe → `-34971` | *Not directly tested — the file was opened successfully moments earlier by the reader, so reaching this needs the file to be deleted or its permissions changed mid-execute.* | +| 21 | (c) | `getDatasetInfo` fails for a `Data` dataset → skip that dataset | *Not directly tested — a missing `Data` dataset is fatal inside `H5OINAReader`, which has already run, so `-8970` fires first (row 16) and this branch is defensive only.* | +| 22 | (c) | data-set extent disagrees with the header, in either direction → `-34971` | `Dataset Extent Mismatch rejected (-34971)` | +| 23 | (c) | four `uint8` verbatim copies into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | +| 24 | (c) | Euler copy at three times the tuple offset | `Multi-Scan Slab Placement` (24 values across two slabs) | +| 25 | (c) | phase value outside `[0, phase count]` → `-34972` | `Out-of-Range Phase Value rejected (-34972)` | +| 26 | (c) | Phase widened to `int32` / copied verbatim as `uint8` | `Conversion Option Combinations` (both branches) | +| 27 | (c) | three `float32` verbatim copies (MAD, X, Y) into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | +| 28 | (c) | hexagonal alignment on the scan's own slab, Hexagonal-High points only | `Class 1 Analytical Oracle`; `Conversion Option Combinations`; `Multi-Scan Hexagonal Alignment` | +| 29 | (b)/(c) | cancel checks (3 sites) | *Not directly tested. Requires cancel-signal injection; standard early-return pattern. Excluded from scope by direction.* | +| 30 | (a) | EbsdLib phase-read failures `-90030` (unopenable phase group), `-90031` (`Lattice Dimensions`) and `-90033` (`Laue Group`) surfaced through `-9582` | *Not directly tested — the fixture writer emits every phase dataset or omits `Lattice Angles`, which is `-90032` and is covered by rows 6 and 9. `-90030` needs an HDF5 object that cannot be opened, which `H5Lite` cannot write; the other two need fixture switches this suite does not carry.* | ## Test inventory @@ -125,23 +130,24 @@ Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflig |-----------|--------|-------| | `OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle` | new-for-V&V | Class 1 + 4 over Fixture A. Geometry, all nine cell arrays and all three ensemble arrays asserted element-wise; the ensemble tuple-count invariant. | | `…: Conversion Option Combinations` | new-for-V&V | Class 1 + 4. `GENERATE` over the 2 × 2 hexagonal-alignment × phase-conversion grid; pins the Phase array's type in each branch and that the alignment never reaches cubic or unindexed points. | -| `…: Multi-Scan Slab Placement` | new-for-V&V | Class 1 over Fixture B (cubic, two scans). 24 Euler values plus five other arrays across two slabs; regression pin for D2. | -| `…: Multi-Scan Hexagonal Alignment` | new-for-V&V | Class 1 over Fixture C (hexagonal, two scans). Regression pin for D3; 24 Euler values, eight of which carry the precision discrimination. | +| `…: Multi-Scan Slab Placement` | new-for-V&V | Class 1 over Fixture B (cubic, two scans). 24 Euler values plus eight other arrays across two slabs; regression pin for D2. | +| `…: Multi-Scan Hexagonal Alignment` | new-for-V&V | Class 1 over Fixture C (hexagonal, two scans). Regression pin for D3; 24 Euler values, five of which carry the precision discrimination and three of which are the dyadic controls. | | `…: Format Version Variants` | new-for-V&V | `GENERATE` over `"5.0"` / `"2.0"` / absent; pins that the reader does not gate on the version and that the dataset is optional. | | `…: Parameter Rejections` | new-for-V&V | Three sections pinning `-9580`, `-9581`, `-9583`. Replaces the previous invalid-execution test, whose sections asserted only "some error" and whose input file was in neither extracted archive. | | `…: Invalid Cell Counts rejected (-9584)` | new-for-V&V | Three sections: zero X Cells, zero Y Cells, negative X Cells. | | `…: Scan Header Mismatch rejected (-9585)` | new-for-V&V | Two sections: differing cell counts, differing step size. | | `…: Missing Scan Name rejected (-9586)` | new-for-V&V | Two sections: the missing name in first and in second position. | -| `…: Phase Index Out Of Range rejected (-9587)` | new-for-V&V | A single-phase file whose phase group is named `7`. | +| `…: Phase Index Out Of Range rejected (-9587)` | new-for-V&V | Two sections: a phase group named `7` in the first selected scan, and the same in a later selected scan. The second fixture passes preflight and crashes the process against the pre-correction filter; regression pin for D8. | +| `…: Scan Phase Count Mismatch rejected (-9589)` | new-for-V&V | Two sections: a later scan declaring more phase groups than the first, and one declaring fewer. The first fixture passes preflight and crashes the process against the pre-correction filter; regression pin for D8. | | `…: Dataset Extent Mismatch rejected (-34971)` | new-for-V&V | A 4 × 4 header over 8-row data sets; deterministic, no file-mutation injection needed. | | `…: Out-of-Range Phase Value rejected (-34972)` | new-for-V&V | A one-phase file with a Phase byte of 5. | | `…: EbsdLib Error Passthrough - Missing Data Column (-8970)` | new-for-V&V | Pins the code and that the message names the scan and the file. | -| `…: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | new-for-V&V | Pins the code and the message content. This fixture crashed the process before D7 was corrected. | -| `…: Stacking Order Warning (-9588)` | new-for-V&V | Both branches: the warning for High-to-Low, and no warning for Low-to-High. | -| `…: Real AZtec File Readback` | modified | Class 2. Was `Valid Filter Execution`, which compared against the archive's `.dream3d` exemplar — a file this filter had written. Now compares all nine cell arrays element-wise against an `H5Lite` readback of the `.h5oina`'s own data sets, with the geometry and the ensemble values pinned from the h5py derivation. | -| *(retired)* `…: InValid Filter Execution` | retired | Its sentinel extracted `6_6_ImportH5Data.tar.gz` while its paths pointed into `H5Oina_Test_Data/`, and the file it named exists in neither archive, so the "incompatible manufacturer" section passed on file-not-found. Replaced by `Parameter Rejections` and the nine code-pinned rejection cases. | +| `…: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | new-for-V&V | Two sections, one per `-9582` return site, each asserting the file path and the scan name that site injects. This fixture crashed the process before D7 was corrected. | +| `…: Stacking Order` | new-for-V&V | Two sections over Fixture B, one per order, each pinning Euler, Band Contrast and Phase values that only that order can produce; regression pin for D9. Replaces `Stacking Order Warning (-9588)`, which pinned a warning that no longer exists. | +| `…: Real AZtec File Readback` | kept | Class 2. Was `Valid Filter Execution`, which compared against the archive's `.dream3d` exemplar — a file this filter had written. It now compares all nine cell arrays element-wise against an `H5Lite` readback of the `.h5oina`'s own data sets, with the geometry and the ensemble values pinned from the h5py derivation; 6,966 assertions. | +| *(retired)* `…: InValid Filter Execution` | retired | Its sentinel extracted `6_6_ImportH5Data.tar.gz` while its paths pointed into `H5Oina_Test_Data/`, and the file it named exists in neither archive, so the "incompatible manufacturer" section passed on file-not-found. Replaced by `Parameter Rejections` and the eight code-pinned rejection cases plus the two EbsdLib passthroughs. | -All 16 pass in the EbsdLib preset build `NX-Com-Qt69-Vtk96-Rel-EbsdLib` — 8,890 assertions in total. OOC is waived for this batch. +All 17 pass in the EbsdLib preset build `NX-Com-Qt69-Vtk96-Rel-EbsdLib`, built and run at the tree state one commit before the `vcpkg.json` pin is raised to EbsdLib 3.1.1 — the pin makes the head unconfigurable until that release exists, so every measurement here is from the pre-pin tree, whose sources are otherwise identical. The whole `OrientationAnalysis::` suite is 308 tests with one failure at that state, `ComputeSchmidsFilter`, which is the known EbsdLib-staging numerical drift and is analysed in `vv/provenance/ReadH5OinaDataFilter.md`. The filter suite reports **9,374 assertions** when the 17 test cases run in one process. A `ctest -V` run sums to **9,390** instead, because `UnitTest::LoadPlugins()` guards itself per process, so its single assertion is counted once in each of ctest's 17 processes rather than once overall; both numbers are correct for how they were measured. OOC build skipped, for the reason recorded in the V&V phase row. ## Exemplar archive @@ -154,21 +160,25 @@ All 16 pass in the EbsdLib preset build `NX-Com-Qt69-Vtk96-Rel-EbsdLib` — 8,89 **Not applicable — DREAM3D 6.5.171 has no H5OINA importer** (evidence in Algorithm Relationship). No legacy comparison was run and none is possible. -`vv/deviations/ReadH5OinaDataFilter.md` instead records, in the same structured form, the seven differences between what DREAM3D-NX 7.0.0 through 7.4.1 shipped and the corrected behavior: +`vv/deviations/ReadH5OinaDataFilter.md` instead records, in the same structured form, the eleven differences between what DREAM3D-NX 7.0.0 through 7.4.1 shipped and the corrected behavior: - `ReadH5OinaDataFilter-D1` — the hexagonal alignment added 30 radians to a radian-valued φ2 instead of 30 degrees; the option ships ON. - `ReadH5OinaDataFilter-D2` — in a multi-scan import the Euler block of scan 2 onward landed a third of the way into its slab. - `ReadH5OinaDataFilter-D3` — the hexagonal alignment was applied to the first scan once per scan and never to the others. - `ReadH5OinaDataFilter-D4` — "Import Pattern Data" could never succeed and reported a misleading reason. - `ReadH5OinaDataFilter-D5` — the third lattice angle was discarded and γ echoed β. -- `ReadH5OinaDataFilter-D6` — lattice angles were reported in radians while every other importer reports degrees. +- `ReadH5OinaDataFilter-D6` — lattice angles were reported in radians while every other importer reports degrees. **Breaking change**; see its release note and migration section. - `ReadH5OinaDataFilter-D7` — a phase group missing `Lattice Angles` crashed the process. +- `ReadH5OinaDataFilter-D8` — a multi-scan selection whose scans declared different phase groups wrote past the end of the ensemble arrays and crashed the process. +- `ReadH5OinaDataFilter-D9` — the Stacking Order setting was accepted and never applied. +- `ReadH5OinaDataFilter-D10` — every error message `H5OINAReader` composed was discarded, so failures reached the user with a blank reason. +- `ReadH5OinaDataFilter-D11` — `H5OINAReader::readData()` returned a code that did not match the one it set, and never validated its column count. -D5, D6 and D7 are corrected in EbsdLib and reach users only through EbsdLib 3.1.1. +D5, D6, D7, D10 and D11 are corrected in EbsdLib and reach users only through EbsdLib 3.1.1. ## Follow-ups for the engineering team -1. **Sibling exposure (not fixed here, by direction).** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. `utilities/IEbsdOemReader.hpp` was deliberately left untouched so this work changes no sibling behavior. -2. **`stackingOrder` is still not implemented** (see the HO-9 proposal in the task report). It now warns rather than being silently ignored; implementing it would require a member on the shared `ReadH5DataInputValues` and a change to the shared scan loop, which is sibling-affecting work. -3. **EbsdLib 3.1.1 release gate.** This pull request joins #1723 behind the same gate. -4. **`H5OINAReader::getPatternDims(std::array)` takes its argument by value** and `getPatternData()` returns `nullptr`. Implementing pattern import for H5OINA is a feature, not a fix, and is out of scope here. +1. **Sibling exposure.** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes, `utilities/IEbsdOemReader.hpp` is unchanged, so the two siblings behave exactly as they did. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; neither honors the Stacking Order setting the scan-selection parameter carries (D9's shape); and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. +2. **`-9587` and `-9589` are preflight checks over a shared write.** Both guard an out-of-range write that happens in the shared `IEbsdOemReader::readData` at execute. A file modified between preflight and execute would still reach that write. Closing it at the point of the write means bounds-checking the shared header, which changes all three filters. +3. **EbsdLib 3.1.1.** The corrections behind D5, D6, D7, D10 and D11 live on `topic/3_1_1_staging` and reach users only through that release; the `vcpkg.json` pin requires it. +4. **`H5OINAReader::getPatternDims(std::array)` takes its argument by value** and `getPatternData()` returns `nullptr`. Implementing pattern import for H5OINA is a feature, not a fix. diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md index 99abf7b4e4..cc6f1c2d36 100644 --- a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -4,7 +4,7 @@ This file therefore records, in the same structured form, every difference between the behavior DREAM3D-NX shipped and the correct behavior. Entries are referenced by stable ID (`ReadH5OinaDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. -Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1, D2, D3, D4, D5 and D6 are present in every release from 7.0.0 through 7.4.1 (2026-03-23), which is every release that contains the filter. **No released version carries these corrections yet**: D5, D6 and D7 are corrections to EbsdLib's `H5OINAReader` and reach users only through EbsdLib 3.1.1, and the DREAM3D-NX release that consumes it has not been made. +Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1 through D11 are present in every release from 7.0.0 through 7.4.1 (2026-03-23), which is every release that contains the filter. **No released version carries these corrections yet**: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINAReader` and reach users only through EbsdLib 3.1.1, and the DREAM3D-NX release that consumes it has not been made. --- @@ -121,6 +121,7 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | | **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Breaking change** | **Yes.** The value of a published output array changes for every H5OINA import. | **Symptom:** The three angle slots of `LatticeConstants` were reported in radians for H5OINA imports and in degrees for every other EBSD importer. A cubic phase imported from an `.h5oina` file reported `1.5707964, 1.5707964, 1.5707964`, while the same phase imported from a `.ctf` or `.ang` file reported `90, 90, 90`. The array's meaning therefore depended on which file format the phase happened to come from, with nothing in the data to say which. @@ -130,7 +131,9 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter **Correct behavior:** `H5OINAReader` converts the angles from radians to degrees on import, on a double-precision intermediate, so the array carries the same unit no matter which format the phase came from. Corrected in EbsdLib on `topic/3_1_1_staging` (`ENH: Convert H5OINA lattice angles to degrees on import`). Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"` (90, 90, 120 for the hexagonal fixture phase) and `…::"Real AZtec File Readback"` (90, 90, 90 for the production file's titanium-cubic phase). -**Recommendation:** Trust the corrected behavior — the degrees convention is the one the rest of the toolkit uses and the one the other importers already produced. Consumers that had compensated by converting H5OINA lattice angles themselves must stop doing so once running against EbsdLib 3.1.1. The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). +**Release note and migration:** This is a **breaking change to a published output**, and the first release that carries it is the first DREAM3D-NX release built against EbsdLib 3.1.1; no released version carries it today. Every `.dream3d` file written by 7.0.0 through 7.4.1 has radians in components 3, 4 and 5 of `LatticeConstants`, so any saved exemplar, regression baseline or pipeline comparison that reads those components changes value on upgrade. To compare a stored radian value against a new import, multiply it by 180/π. Consumers that compensated by converting H5OINA lattice angles themselves must stop doing so. Nothing else in the import changes unit: `Euler` is still radians and the three lattice dimensions are still unconverted. The user-facing migration note is in `docs/ReadH5OinaDataFilter.md` under "Migration Notes". The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). + +**Recommendation:** Trust the corrected behavior — the degrees convention is the one the rest of the toolkit uses and the one the other importers already produced. --- @@ -143,7 +146,7 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter | **Affected releases** | 7.0.0 through 7.4.1 | | **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | -**Symptom:** An `.h5oina` file whose phase group was missing its `Lattice Dimensions` or `Lattice Angles` dataset **crashed the process**. This is empirically demonstrated, not inferred: running the fixture through the batch-base build produced a SEGFAULT, recorded as `OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582) (SEGFAULT)` in `ww_work/ReadH5OinaData/red_baseline.log`. +**Symptom:** An `.h5oina` file whose phase group was missing its `Lattice Dimensions` or `Lattice Angles` dataset **crashed the process**. This is empirically demonstrated, not inferred: running the fixture against a build with the corrected filter sources but the pre-correction `H5OINAReader` reports the regression test as `OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582) (SEGFAULT)` rather than as a failure. The suite log recording that run is listed in `vv/provenance/ReadH5OinaDataFilter.md`. **Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readHeader()` read the two vector datasets while discarding their error codes, then indexed elements 0 through 2 of the resulting vectors, which are empty when the dataset is absent. The `Laue Group` and `Space Group` reads discarded their error codes as well, and a failed phase-group open was not checked at all. @@ -155,17 +158,102 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter --- -## Malformed-input rejections added alongside these entries +## ReadH5OinaDataFilter-D8 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D8` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** Selecting two or more scans from a file whose scans do not all declare the same phase groups **crashed the process** at execute, with no preflight error and no warning. No malformed input is involved: an AZtec file may legitimately carry scans with different phase lists, and a scan that declares more phases than the *first* selected scan is enough. + +**Root cause:** Bug. The single Ensemble Attribute Matrix that all of the stacked scans share is sized from the first selected scan's phase count, `phases.size() + 1`. The shared ensemble fill in `utilities/IEbsdOemReader.hpp` then runs once per selected scan and writes `crystalStructures[phaseId]`, `materialNames[phaseId]` and `latticeConstants` component `phaseId`, where `phaseId` is the integer in that scan's HDF5 phase group name. Any later scan whose phase index exceeds the first scan's phase count writes past the end of all three arrays. Preflight range-checked those indices for the first selected scan only, and the execute-side `-34972` check inspects the phase *column values*, not the phase *group names*, and runs after the ensemble fill has already happened. + +**Affected users:** Anyone selecting more than one scan from an `.h5oina` file whose scans differ in their phase lists. Single-scan imports and multi-scan imports of files with one uniform phase list — which is every file in the shipped test data — were never affected, which is why the pre-existing suite could not see it. + +**Correct behavior:** Preflight applies the phase-index range check to **every** selected scan, with the bound taken from the first scan's phase count (`-9587`), and rejects a selected scan whose phase-group count differs from the first scan's (`-9589`), telling the user to import scans with differing phase lists separately. Pinned by `test/ReadH5OinaDataTest.cpp::"Phase Index Out Of Range rejected (-9587)"` section "Later selected scan" and `…::"Scan Phase Count Mismatch rejected (-9589)"`, whose fixtures both pass preflight and crash the process against the pre-correction filter. + +**Recommendation:** Trust the corrected behavior. A multi-scan import that completed under 7.0.0 through 7.4.1 was not affected — the crash is the only outcome the defect produced. + +--- + +## ReadH5OinaDataFilter-D9 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D9` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** The **Stacking Order** setting carried by the scan selection had no effect. Choosing *High To Low* produced exactly the same output as *Low To High*, with no error and no warning, so the Z order of a multi-scan stack could not be changed from the parameter that appears to control it. + +**Root cause:** Bug. The scan loop iterated `SelectedScanNames.scanNames` in list order unconditionally and never read `SelectedScanNames.stackingOrder`. The setting reached the algorithm — it is a member of `OEMEbsdScanSelectionParameter::ValueType` — and was simply not consulted. + +**Affected users:** Anyone importing more than one scan who set the stacking order to *High To Low*. Single-scan imports are unaffected, because the two orders coincide. + +**Correct behavior:** *Low To High* stacks the scans in the order they are listed; *High To Low* stacks them in the reverse of that order, so the last selected scan occupies tuple slab 0. Pinned by `test/ReadH5OinaDataTest.cpp::"Stacking Order"`, whose two sections assert cell values that only the corresponding order can produce. + +**Recommendation:** Trust the corrected behavior. A pipeline saved under 7.0.0 through 7.4.1 with *High To Low* selected now produces a Z-reversed stack relative to what it used to produce, which is what it always asked for; check any such pipeline before re-running it. The two sibling filters `ReadH5OimData` and `ReadH5EspritData` still ignore the same setting — see the V&V report's Follow-ups. + +--- + +## ReadH5OinaDataFilter-D10 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D10` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | + +**Symptom:** Every failure reported out of `H5OINAReader` reached the user with a blank `Message:` field. A malformed `.h5oina` produced an error code and an empty explanation, so nothing in the message said which dataset, phase or scan was at fault. + +**Root cause:** Bug in the trusted parsing boundary. Ten failure paths in `H5OINAReader` were written as `std::string str; std::stringstream ss(str); ss << …; setErrorMessage(str);`. `std::stringstream(str)` copies `str` into the stream's own buffer, so everything composed went into that copy and the still-empty original was handed to `setErrorMessage()`. A related defect sat one level up: `readFile()` replaced whatever `readHeader()` or `readData()` had set with a generic "could not read header" / "could not read data", discarding the specific reason even where one had been composed correctly. + +**Affected users:** Anyone who hit any `H5OINAReader` error — a malformed file, a missing dataset, an absent scan. The filter's own `-8970` and `-9582` messages name the file and the scan, so the file was identifiable; the reason was not. + +**Correct behavior:** Each of the ten sites composes into a stream of its own and reports that stream's contents, and `readFile()`'s wrappers name the scan and append the inner message rather than replacing it. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Report the error messages H5OINAReader builds`). + +**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still reports the blank message. No test asserts the text `H5OINAReader` composes — the filter-side assertions match the scan name and file path that the DREAM3D-NX format strings inject, which are independent of the reader's message. + +--- + +## ReadH5OinaDataFilter-D11 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D11` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | + +**Symptom:** Two defects in one code path. A scan whose header declared zero rows was rejected with error code `-90301` recorded on the reader but `-301` handed back as the return value, so a caller reporting the return value and a caller reading `getErrorCode()` disagreed about what had happened. A scan whose header declared a **negative** column count was not rejected at all: the count was widened to `size_t`, producing an enormous allocation request. + +**Root cause:** Bug in the trusted parsing boundary. `H5OINAReader::readData()` validated the row count only, and its rejection returned a literal that did not match the code it had just set. + +**Affected users:** Anyone opening a truncated or otherwise malformed `.h5oina` file. DREAM3D-NX reports the code it receives, so the mismatched value was the one the user saw. + +**Correct behavior:** Both the row and the column count are validated and the rejection returns the code it sets. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Validate H5OINAReader::readData()'s row and column counts`). The filter rejects the same shape earlier and independently: `-9584` rejects `X Cells` or `Y Cells` below 1 at preflight, which is pinned by `test/ReadH5OinaDataTest.cpp::"Invalid Cell Counts rejected (-9584)"`, including the negative case. + +**Recommendation:** Trust the corrected behavior. The filter's `-9584` guard fires first for a file selected through DREAM3D-NX, so this entry matters to other `H5OINAReader` callers. + +--- + +## Malformed-input rejections -These are not behavioral deviations on well-formed files — every one of them is a new rejection path that replaces an out-of-range read, an out-of-range write, or a silently useless output. They are listed here so a reader auditing the error-code series has one place to find them. +These are not behavioral deviations on well-formed files — every one of them is a rejection path that stands where 7.0.0 through 7.4.1 performed an out-of-range read, an out-of-range write, or produced a silently useless output. They are listed here so a reader auditing the error-code series has one place to find them. The multi-scan phase-group cases are the exception and are recorded as D8, because they are reachable from a well-formed file. | Code | Rejects | Previously | |---|---|---| | `-9584` | `X Cells` or `Y Cells` below 1 in the first selected scan | The counts were cast to `usize` unchecked, producing a zero-sized geometry, or an enormous one from a negative count | | `-9585` | A selected scan whose grid or step sizes differ from the first selected scan's | Only the first scan's header was ever read; the copy then spanned the later scan's reader buffers using the first scan's point count | | `-9586` | A selected scan name that is not in the file | Only the first name was checked, at preflight; a bad later name failed part-way through execute with the earlier scans already written into the output arrays and no rollback | -| `-9587` | Phase groups not numbered 1 through N | The ensemble arrays are sized from the phase count but each phase is placed at the index in its group name, so a group named `7` in a one-phase file wrote past the end of the ensemble arrays | -| `-34971` | A `Data` dataset whose extent disagrees with the header's cell counts | The reader sizes its buffers to the actual extent while the copy spans the header's point count, reading past the end of those buffers | +| `-9587` | A phase group of **any** selected scan named outside 1 through N, where N is the first selected scan's phase count | The check covered the first selected scan only, so a group named `7` in a later scan wrote past the end of the ensemble arrays — see D8 | +| `-9589` | A selected scan whose phase-group count differs from the first selected scan's | The ensemble arrays were sized from the first scan alone and filled from every scan — see D8 | +| `-34971` | A `Data` dataset whose extent disagrees with the header's cell counts, in either direction | The reader sizes its buffers to the actual extent while the copy spans the header's point count, reading past the end of those buffers when the dataset is short and silently dropping the surplus rows when it is long | | `-34972` | A phase value outside `[0, phase count]` | The value indexed `CrystalStructures` unchecked in the alignment loop, reading past the end of the ensemble array | -Error code `-34970` (null pattern data) is retired with the unreachable execute-side pattern block described in D4. +Error codes `-34970` (null pattern data) and `-9588` (stacking order not applied) are retired: the first with the unreachable execute-side pattern block described in D4, the second with the stacking-order implementation described in D9. diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md index 7786073eb4..cfb5613b43 100644 --- a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md @@ -3,6 +3,12 @@ This sidecar documents the test data behind `ReadH5OinaDataFilter` and the disposition of each part of it. +**Working folder.** Every file named in this document under "Recorded output", "Supporting +evidence" or "Suite logs" lives in the V&V working folder `ww_work/ReadH5OinaData/`, which +is **not committed to this repository**; it is archived to the V&V working-folder remote. +The committed deliverables are this sidecar, `vv/ReadH5OinaDataFilter.md` and +`vv/deviations/ReadH5OinaDataFilter.md`. + --- ## Archive identity @@ -14,7 +20,10 @@ of each part of it. | **`download_test_data()` entry** | `src/Plugins/OrientationAnalysis/test/CMakeLists.txt:146` | | **Contents** | `H5Oina_Test_Data.h5oina`, `H5Oina_Test_Data.dream3d`, `H5Oina_Test_Data.xdmf`, `Acknowledgements.md` | | **Used by** | `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` (the `.h5oina` only) | -| **Changed?** | **No.** The archive is unchanged and was not re-uploaded; the SHA512 above is the one already in `CMakeLists.txt`. | +| **Generated by** | **Not generated for this V&V.** The archive predates it and is unchanged, so there is no generating pipeline, script or commit to record on this side. The `.h5oina` is a vendor export, not a generated file; the `.dream3d` and `.xdmf` were produced by running this filter over it, which is why they are no longer consulted (below). | +| **Generated on / at commit** | Unknown — the archive was uploaded before the provenance convention existed. Its identity is pinned by the SHA512 above, which is byte-for-byte the value in `CMakeLists.txt`. | +| **Canonical oracle output** | **None from this archive.** The oracle for this filter is analytical: expected values are derived from the fixture specification in the test source, and the production file is compared against a readback of its own bytes. See "Replacement oracle" below. | +| **Changed?** | **No.** The archive is unchanged and was not re-uploaded. | ## The input file: retained @@ -32,7 +41,7 @@ an irreplaceable piece of production realism. Its properties, read directly with | Euler | float32 (625, 3) in radians, observed range [0, 6.2773] | | Extra columns | `Beam Position X`/`Y`, `Detector Distance`, `Pattern Center X`/`Y`, `Pattern Quality`, and a 625 × 512 × 622 `Processed Patterns` dataset — none of which this filter reads | -Recorded probe output: `ww_work/ReadH5OinaData/probe_real_file.txt`. +Recorded probe output: `probe_real_file.txt`. Note that the file carries `Processed Patterns` and **no** `Unprocessed Patterns`, while the filter only ever targeted the latter — one of the observations behind deviation D4. @@ -42,36 +51,40 @@ filter only ever targeted the latter — one of the observations behind deviatio `H5Oina_Test_Data.dream3d` is **no longer used as a comparison target**. It remains inside the archive because the archive is unchanged, but no test reads it. -**Why.** The file was produced by running this filter on the sibling `.h5oina` in the same -archive: its cell arrays are bit-identical to the raw file data. It is therefore a -self-oracle. It is *not* a forbidden legacy oracle — no legacy H5OINA importer has ever -existed, so no version of DREAM3D could have generated it — but it pins "the filter keeps -doing what it did", not "the filter is right", which is the same failure mode. +**Why.** The file was produced by running this filter on the sibling `.h5oina`, so it is a +self-oracle. Its `Euler` array is bit-identical to the raw file data — all 1,875 values, +compared as `uint32` bit patterns — which is what a verbatim copy of that column should +produce; the remaining eight cell arrays were not measured for bit-identity, because the +file's disposition does not turn on it. It is *not* a forbidden legacy oracle — no legacy +H5OINA importer has ever existed, so no version of DREAM3D could have generated it — but it +pins "the filter keeps doing what it did", not "the filter is right", which is the same +failure mode. It also had the pre-correction behavior baked in: its `LatticeConstants` reads `[3.192, 3.192, 3.192, 1.5707964, 1.5707964, 1.5707964]`, the radian angles of deviation D6. A test comparing against it did not merely fail to detect D6 — it actively enforced it. +Recorded measurement: `exemplar_selforacle_check.txt`. -Its discriminating power against the defects found in this work was near zero: the file has +Its discriminating power against the eleven deviations is near zero: the file has a single cubic phase, so the hexagonal alignment (D1, D3) never executes; it holds one scan, -so the slab offsets (D2) are always zero; pattern import was off, so D4 and its latent type -mismatch were never reached; and its angles are all equal, so the gamma slot (D5) is -invisible. What it did pin — single-scan copy plumbing, geometry wiring, the phase widening -and the ensemble slot-0 defaults against a real-world file — is preserved and strengthened by -its replacement. +so the slab offsets (D2), the stacking order (D9) and the multi-scan ensemble write (D8) are +never reached; pattern import was off, so D4 and its latent type mismatch were never +reached; and its angles are all equal, so the gamma slot (D5) is invisible. What it did pin +— single-scan copy plumbing, geometry wiring, the phase widening and the ensemble slot-0 +defaults against a real-world file — is preserved and strengthened by its replacement. ## Replacement oracle **Toy fixtures, written by the test at run time.** `test/ReadH5OinaDataTest.cpp` declares a fixture specification as C++ structs and writes `.h5oina` files with `H5Support::H5Lite` -into the binary test-output directory. Three fixture specifications (A, B, C) materialise -into nineteen files at run time: eight that are imported successfully and eleven that back a +into the binary test-output directory. Three fixture specifications materialise into +twenty-three files at run time: eight that are imported successfully and fifteen that back a rejection or passthrough case. Nothing is committed as binary test data and no archive upload was needed. The eight that import carry exactly the dataset set `H5OINAReader` -requires, plus the inert root datasets for realism. Of the eleven behind the rejection and -passthrough cases, two omit a required dataset outright -- `Data/Bands` and -`Phases/1/Lattice Angles` -- while the other nine carry the full set and are rejected on -their values or on how they are selected. +requires, plus the inert root datasets for realism. Of the fifteen behind the rejection and +passthrough cases, three omit a required dataset outright — `Data/Bands` once and +`Phases//Lattice Angles` twice — while the other twelve carry the full set and are +rejected on a parameter value, on a header or phase value, or on how the scans are selected. Every expected value is derived from the fixture specification. The hexagonal-alignment expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, @@ -88,11 +101,11 @@ are pinned as literals derived from the h5py readback. | Field | Value | |---|---| -| **Script** | `ww_work/ReadH5OinaData/h5oina_oracle.py` (not committed; archived to the V&V working-folder remote) | -| **Interpreter** | `/opt/local/anaconda3/envs/dream3d/bin/python`, h5py 3.16.0, NumPy | +| **Script** | `h5oina_oracle.py` | +| **Interpreter** | `/opt/local/anaconda3/envs/dream3d/bin/python`, h5py 3.16.0, NumPy 2.5.2 | | **Author** | Michael Jackson | | **Date** | 2026-08-24 | -| **`spec` mode** | Prints the fixture specification and every derived expectation as C++-ready float32 literals. Recorded output: `oracle_spec.txt`. This is the source of the pinned hexagonal-alignment constants. | +| **`spec` mode** | Prints the fixture specification and every derived expectation as C++-ready float32 literals. Recorded output: `oracle_spec.txt`, reproduced bit-for-bit as `oracle_spec_rerun.txt`. This is the source of the pinned hexagonal-alignment constants. | | **`readback` mode** | Re-reads an `.h5oina` with h5py and re-derives the expected NX arrays from the documented rules, without EbsdLib or simplnx. Recorded output for the production file: `readback_real_file.txt`. | The script encodes the derivation rules read out of the filter and algorithm source — geometry @@ -101,16 +114,83 @@ the ensemble slot-0 defaults, the radians-to-degrees lattice-angle conversion an hexagonal φ2 alignment — and applies them to the fixture specification or to a file's bytes. It never runs the filter. -Supporting evidence in the same folder: +## Trusted-boundary versions + +| Boundary | Version | +|---|---| +| **EbsdLib** | `topic/3_1_1_staging`, the branch that becomes EbsdLib 3.1.1. This is the primary Class 2 trusted boundary: it owns HDF5 traversal, header and phase parsing, the required-dataset reads and the error codes. Five of this filter's eleven deviations are corrections inside it. | +| h5py / NumPy | 3.16.0 / 2.5.2, under `/opt/local/anaconda3/envs/dream3d/bin/python` | +| HDF5 | as vendored by the `NX-Com-Qt69-Vtk96-Rel-EbsdLib` preset's vcpkg manifest | + +## Supporting evidence | File | What it records | |---|---| | `pi_over_6_discriminator.txt` | The exhaustive sweep of float32 values in `[0.25, 6.5)` that identified which φ2 values separate a double-precision intermediate from a float32 one (8,289,627 of 38,797,312 do), and confirmation that `30 × k_PiOver180D` is bit-identical to the nearest double to π/6 | | `probe_real_file.txt` | The h5py structure dump of the production `.h5oina` | -| `red_baseline.log` | The full test suite run at the batch base commit: 14 of 16 failing, including the SEGFAULT that is deviation D7 | +| `error_column_check.txt` | The `Error` column census of the production file — 587 points at 1, 38 at 2, none at 0 — behind the documentation's masking correction | +| `exemplar_selforacle_check.txt` | The measurement behind the exemplar's retirement: `Euler` bit-identity and the radian `LatticeConstants` | | `mutation_table.md` | Seven mutations, each killing exactly its claimed test cases, each reverted to an empty diff | -| `base_oa_suite.log` / `full_oa_suite.log` | The OrientationAnalysis suite at base sources and at branch head, used to establish that all 29 suite failures are pre-existing | -| `ebsdlib_suite.log`, `simplnxcore_suite.log` | The EbsdLib and SimplnxCore suite runs | + +## Suite logs + +| File | What it records | +|---|---| +| `red_baseline.log` | The RED baseline: the full suite run against the batch-base sources with the oracle suite in place — 14 of 16 failing, including the SEGFAULT that is deviation D7 | +| `fixround1_red_baseline.log` | The RED baseline for the multi-scan phase-group guards and the stacking-order implementation: three of 17 failing before those changes | +| `fixround1_green_readh5oina.log` | The green `ctest -V` run of the filter's suite, 17 of 17 | +| `fixround1_oa_suite.log` | The full `OrientationAnalysis::` suite at the verified tree state, after a full build of the preset directory: 308 tests, one failure | +| `fixround1_simplnxcore_suite.log` | The full `SimplnxCore::` suite at the same state | +| `gate_oa_head.log`, `gate_oa_ebsdlib539.log`, `gate_head_fail_names.txt`, `gate_539_fail_names.txt` | The controlled EbsdLib rollback described below | +| `ebsdlib_suite.log`, `simplnxcore_suite.log` | Earlier EbsdLib and SimplnxCore suite runs | + +### The pre-existing OrientationAnalysis failure + +**After a full build of the preset directory (`cmake --build . --target all`), the +`OrientationAnalysis::` suite is 308 tests with exactly one failure: +`OrientationAnalysis::ComputeSchmidsFilter`, on `Schmid_Lambdas`.** That is the known +EbsdLib-staging numerical drift from the `CubicOps` precision correction; it reads no +`.h5oina` file and shares no code with this filter. Log: `fixround1_oa_suite.log`. + +Two earlier runs of the same suite in the same directory recorded **twenty-nine** failures: +`ComputeSchmidsFilter` plus twenty `PIPELINE::OrientationAnalysis::` entries and eight +`PY::OrientationAnalysis::` entries. Those twenty-eight are **artifacts of a partially built +directory, not defects.** `PIPELINE::` tests invoke `Bin/nxrunner` and `PY::` tests invoke +the Python bindings, and neither target is built by +`cmake --build . --target OrientationAnalysisUnitTest`, which is what those runs had built. +Once `--target all` rebuilt `nxrunner` against the current libraries, all twenty-eight +passed with no source change. Any claim about this build directory's baseline has to be +measured after a full build. + +The one genuine failure is shown to be independent of the H5OINA corrections by a +**controlled rollback of the EbsdLib side alone**, which is the only variable in them that +could reach an unrelated test: + +| Configuration | Result | Non-H5OINA failure set | +|---|---|---| +| simplnx and EbsdLib both at the batch head | 307 tests, 29 fail | 29 names -- `gate_head_fail_names.txt` | +| simplnx at the batch head, EbsdLib rolled back to `539ddfc` (the batch's EbsdLib base) | 307 tests, 32 fail | **the same 29 names -- `diff` of the two sorted lists is empty** | + +Logs: `gate_oa_head.log` and `gate_oa_ebsdlib539.log`, with the sorted name lists in +`gate_head_fail_names.txt` and `gate_539_fail_names.txt`. Both runs were made in the +partially built state described above, so their absolute counts are inflated; what the +comparison establishes is that rolling EbsdLib back changes the non-H5OINA failure set not +at all. The three extra failures at `539ddfc` are the three ReadH5Oina tests that the +EbsdLib corrections are required for -- `Class 1 Analytical Oracle`, `Real AZtec File +Readback` and `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)`, the last as a +SEGFAULT -- which is the evidence the `vcpkg.json` pin to EbsdLib 3.1.1 rests on. The +structural argument agrees: `git diff --stat 539ddfc..HEAD` on EbsdLib is one file, +`Source/EbsdLib/IO/HKL/H5OINAReader.cpp`, with no header and no API change, so nothing +outside the H5OINA read path can be reached. + +The same applies to `SimplnxCore::`, which is **985 of 985 passing** after the full build +(`fixround1_simplnxcore_suite.log`). An earlier partially built run of it reported +`PIPELINE::SimplnxCore::002_ApplyTransformation_Image` failing — another `nxrunner` +pipeline test of exactly the shape described above. + +`Real AZtec File Readback` is intermittently flaky in this build dir on the test-data +sentinel extraction, not on any assertion; a failure of that shape clears on an immediate +re-run with no source change. ## Oracle-before-comparison ordering From 235ee4f8c3292b6f2c9f39cd2d8e10089016e743 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Mon, 24 Aug 2026 13:48:57 -0400 Subject: [PATCH 09/10] VV: Scope the D8 recommendation and disclose the phase-content restriction * Every reproduction of the D8 defect crashed before writing output, but the write is undefined behavior, so the recommendation no longer claims the crash is the only possible outcome * Disclose the residual restriction: the guards compare phase-group names and counts across scans, not their contents; differing definitions keep the last scan's values and should be imported separately * Repair the sibling-exposure follow-up's sentence structure Signed-off-by: Michael Jackson --- src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md | 2 +- .../vv/deviations/ReadH5OinaDataFilter.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md index 86690a9e72..700960a735 100644 --- a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -178,7 +178,7 @@ D5, D6, D7, D10 and D11 are corrected in EbsdLib and reach users only through Eb ## Follow-ups for the engineering team -1. **Sibling exposure.** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes, `utilities/IEbsdOemReader.hpp` is unchanged, so the two siblings behave exactly as they did. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; neither honors the Stacking Order setting the scan-selection parameter carries (D9's shape); and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. +1. **Sibling exposure.** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes. That header is unchanged, so the two siblings behave exactly as they did. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; neither honors the Stacking Order setting the scan-selection parameter carries (D9's shape); and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. 2. **`-9587` and `-9589` are preflight checks over a shared write.** Both guard an out-of-range write that happens in the shared `IEbsdOemReader::readData` at execute. A file modified between preflight and execute would still reach that write. Closing it at the point of the write means bounds-checking the shared header, which changes all three filters. 3. **EbsdLib 3.1.1.** The corrections behind D5, D6, D7, D10 and D11 live on `topic/3_1_1_staging` and reach users only through that release; the `vcpkg.json` pin requires it. 4. **`H5OINAReader::getPatternDims(std::array)` takes its argument by value** and `getPatternData()` returns `nullptr`. Implementing pattern import for H5OINA is a feature, not a fix. diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md index cc6f1c2d36..32bc44e711 100644 --- a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -1,10 +1,11 @@ # Deviations: ReadH5OinaDataFilter -**There is no DREAM3D 6.5.171 equivalent of this filter**, so there is no legacy comparison and no legacy-versus-SIMPLNX deviation to record. A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits; the filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and has no `FromSIMPLJson`, no legacy-UUID mapping entry and no SIMPL conversion fixtures. +**There is no DREAM3D 6.5.171 equivalent of this filter**, so there is no legacy comparison and no legacy-versus-SIMPLNX deviation to record. The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and has no `FromSIMPLJson`, no legacy-UUID mapping entry and no SIMPL conversion fixtures. This file therefore records, in the same structured form, every difference between the behavior DREAM3D-NX shipped and the correct behavior. Entries are referenced by stable ID (`ReadH5OinaDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. -Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1 through D11 are present in every release from 7.0.0 through 7.4.1 (2026-03-23), which is every release that contains the filter. **No released version carries these corrections yet**: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINAReader` and reach users only through EbsdLib 3.1.1, and the DREAM3D-NX release that consumes it has not been made. +Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1 through D11 are present in every release from 7.0.0 through 7.4.1 (2026-03-23). +EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINAReader` and every release of DREAM3D-NX up through and including 7.4.1 will have these bugs. --- @@ -175,7 +176,7 @@ Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter **Correct behavior:** Preflight applies the phase-index range check to **every** selected scan, with the bound taken from the first scan's phase count (`-9587`), and rejects a selected scan whose phase-group count differs from the first scan's (`-9589`), telling the user to import scans with differing phase lists separately. Pinned by `test/ReadH5OinaDataTest.cpp::"Phase Index Out Of Range rejected (-9587)"` section "Later selected scan" and `…::"Scan Phase Count Mismatch rejected (-9589)"`, whose fixtures both pass preflight and crash the process against the pre-correction filter. -**Recommendation:** Trust the corrected behavior. A multi-scan import that completed under 7.0.0 through 7.4.1 was not affected — the crash is the only outcome the defect produced. +**Recommendation:** Trust the corrected behavior. Every reproduction of the defect crashed before any output was written, so a multi-scan import that *completed* under 7.0.0 through 7.4.1 is very unlikely to have been affected — but the underlying write is undefined behavior, so silent corruption cannot be strictly excluded. Residual restriction: the guards require every selected scan to declare the same phase-group *names and count*; the group *contents* (material name, Laue class, lattice constants) are not compared across scans — the shared ensemble arrays keep the values of the last scan read, and the preflight phase display always reflects the first-listed scan. Scans whose phase definitions differ in content should be imported separately. --- From 1045f95ab18d2721b792d4ab8fed57ce26744585 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Wed, 26 Aug 2026 14:30:59 -0400 Subject: [PATCH 10/10] VV: Complete the H5OINA internal review - Validate finite scan spacing and identical phase definitions across stacked scans - Report invalid phase-group and phase-value data with actionable file context - Add focused regression tests for each internal-review defect - Require EbsdLib 3.1.2 for the corrected H5OINA reader - Rewrite the report around the independent oracle and the DREAM3D-NX 7.0.0 through 7.4.1 defects Signed-off-by: Michael Jackson --- .../docs/ReadH5OinaDataFilter.md | 11 +- .../Filters/Algorithms/ReadH5OinaData.cpp | 48 +-- .../Filters/Algorithms/ReadH5OinaData.hpp | 7 +- .../Filters/ReadH5OinaDataFilter.cpp | 84 ++++- .../Filters/ReadH5OinaDataFilter.hpp | 1 - .../test/ReadH5OinaDataTest.cpp | 139 ++++++++- .../vv/ReadH5OinaDataFilter.md | 288 +++++++++--------- .../vv/deviations/ReadH5OinaDataFilter.md | 81 +++-- .../vv/provenance/ReadH5OinaDataFilter.md | 27 +- 9 files changed, 459 insertions(+), 227 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md index 635c392af6..ccecac72e3 100644 --- a/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ReadH5OinaDataFilter.md @@ -22,10 +22,11 @@ The file is EBSD (Electron Backscatter Diffraction) scan data. The most importan An H5OINA file can hold several scans. Selecting more than one stacks them into a single 3D **Image Geometry**: the X and Y extents and step sizes come from the first selected scan, the Z extent is the number of selected scans, and the Z spacing is the **Z Spacing** -parameter. Every selected scan must describe the same grid as the first one and declare -the same phase groups, and each must be present in the file; the filter reports an error -naming the offending scan otherwise. Scans whose phase lists differ have to be imported -separately, because all of the stacked scans share one **Ensemble Attribute Matrix**. +parameter. The selected scans become the slices of one 3D microstructure. Therefore, +every scan must describe the same grid and use the same phase definitions. The phase-group +names, material names, Laue groups, space groups, and lattice constants must match. All +scans share one **Ensemble Attribute Matrix**, so different phase definitions cannot be +represented correctly. Import scans with different phase definitions separately. The **Stacking Order** setting carried by the scan selection chooses which end of the list lands at Z = 0. *Low To High* stacks the scans in the order they are listed, so the @@ -160,7 +161,7 @@ change values that an existing pipeline may compare against. ## References -[1] Rollett, A.D. Lecture Slides located at [http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf](http://pajarito.materials.cmu.edu/rollett/27750/L17-EBSD-analysis-31Mar16.pdf) +[1] Wright, S. I. and De Graef, M., "Electron backscatter diffraction," *International Tables for Crystallography*. [EBSD reference-frame conventions](https://onlinelibrary.wiley.com/iucr/itc/Cc/wf5160/). ## DREAM3D-NX Help diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp index cf82f80459..c19d7a95d7 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp @@ -98,8 +98,8 @@ void copyRawData(const ReadH5DataInputValues* inputValues, usize count, DataStru auto& dataRef = dataStructure.getDataRefAs(inputValues->CellAttributeMatrixPath.createChildPath(name)); auto* dataStorePtr = dataRef.getDataStore(); - const nonstd::span rawDataPtr(reinterpret_cast(reader.getPointerByName(name)), count); - std::copy(rawDataPtr.begin(), rawDataPtr.end(), dataStorePtr->begin() + offset); + const nonstd::span rawDataSpan(reinterpret_cast(reader.getPointerByName(name)), count); + std::copy(rawDataSpan.begin(), rawDataSpan.end(), dataStorePtr->begin() + offset); } /** @@ -122,12 +122,12 @@ void convertHexEulerAngle(const ReadH5DataInputValues* inputValues, usize totalP // Only this scan's slab is visited. Looping from 0 every time would shift the first // scan's points once per scan and never reach the later scans' points. - for(usize i = tupleOffset; i < tupleOffset + totalPoints; i++) + for(usize tupleIdx = tupleOffset; tupleIdx < tupleOffset + totalPoints; tupleIdx++) { - if(crystalStructuresDSRef[cellPhasesDSRef[i]] == ebsdlib::CrystalStructure::Hexagonal_High) + if(crystalStructuresDSRef[cellPhasesDSRef[tupleIdx]] == ebsdlib::CrystalStructure::Hexagonal_High) { - const auto phi2 = static_cast(eulerDataStoreRef[3 * i + 2]); - eulerDataStoreRef[3 * i + 2] = static_cast(phi2 + k_HexagonalAlignmentRadians); + const auto phi2 = static_cast(eulerDataStoreRef[3 * tupleIdx + 2]); + eulerDataStoreRef[3 * tupleIdx + 2] = static_cast(phi2 + k_HexagonalAlignmentRadians); } } } @@ -135,8 +135,8 @@ void convertHexEulerAngle(const ReadH5DataInputValues* inputValues, usize totalP } // namespace // ----------------------------------------------------------------------------- -ReadH5OinaData::ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues) -: IEbsdOemReader(dataStructure, mesgHandler, shouldCancel, inputValues) +ReadH5OinaData::ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues) +: IEbsdOemReader(dataStructure, messageHandler, shouldCancel, inputValues) { } @@ -163,15 +163,15 @@ Result<> ReadH5OinaData::operator()() } const usize scanCount = orderedScanNames.size(); - for(usize index = 0; index < scanCount; index++) + for(usize scanIdx = 0; scanIdx < scanCount; scanIdx++) { if(m_ShouldCancel) { return {}; } - m_CurrentScanName = orderedScanNames[index]; - m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", m_CurrentScanName, index + 1, scanCount)}); + m_CurrentScanName = orderedScanNames[scanIdx]; + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading scan '{}' ({} of {})", m_CurrentScanName, scanIdx + 1, scanCount)}); Result<> readResults = readData(m_CurrentScanName); if(readResults.invalid()) { @@ -183,8 +183,8 @@ Result<> ReadH5OinaData::operator()() return {}; } - m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", m_CurrentScanName, index + 1, scanCount)}); - Result<> copyDataResults = copyRawEbsdData(static_cast(index)); + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Copying the cell data of scan '{}' ({} of {})", m_CurrentScanName, scanIdx + 1, scanCount)}); + Result<> copyDataResults = copyRawEbsdData(static_cast(scanIdx)); if(copyDataResults.invalid()) { return copyDataResults; @@ -194,12 +194,12 @@ Result<> ReadH5OinaData::operator()() } // ----------------------------------------------------------------------------- -Result<> ReadH5OinaData::copyRawEbsdData(int index) +Result<> ReadH5OinaData::copyRawEbsdData(int scanIndex) { const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const usize totalPoints = imageGeom.getNumXCells() * imageGeom.getNumYCells(); - // Scan `index` occupies the tuple slab [index * totalPoints, (index + 1) * totalPoints). - const usize tupleOffset = static_cast(index) * totalPoints; + // Scan `scanIndex` occupies the tuple slab [scanIndex * totalPoints, (scanIndex + 1) * totalPoints). + const usize tupleOffset = static_cast(scanIndex) * totalPoints; const std::string& scanName = m_CurrentScanName; @@ -223,25 +223,25 @@ Result<> ReadH5OinaData::copyRawEbsdData(int index) { const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::AngFile::CrystalStructures)); const usize ensembleTupleCount = crystalStructures.getNumberOfTuples(); - const nonstd::span rawPhasePtr(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); - for(usize i = 0; i < totalPoints; i++) + const nonstd::span rawPhaseSpan(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); + for(usize tupleIdx = 0; tupleIdx < totalPoints; tupleIdx++) { - if(static_cast(rawPhasePtr[i]) >= ensembleTupleCount) + if(static_cast(rawPhaseSpan[tupleIdx]) >= ensembleTupleCount) { - return MakeErrorResult(-34972, fmt::format("Scan point {} of scan '{}' carries phase value {}, which is outside the valid range [0, {}] established by the file's phase definitions.", i, - scanName, rawPhasePtr[i], ensembleTupleCount - 1)); + return MakeErrorResult(-34972, fmt::format("Scan point {} of scan '{}' in '{}' carries phase value {}, which is outside the valid range [0, {}] established by the file's phase definitions.", + tupleIdx, scanName, m_InputValues->SelectedScanNames.inputFilePath.string(), rawPhaseSpan[tupleIdx], ensembleTupleCount - 1)); } } } if(m_InputValues->ConvertPhaseToInt32) { - const nonstd::span rawDataPtr(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); + const nonstd::span rawDataSpan(reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5OINA::Phase)), totalPoints); auto& dataRef = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5OINA::Phase)); auto* dataStorePtr = dataRef.getDataStore(); - for(usize i = 0; i < totalPoints; i++) + for(usize tupleIdx = 0; tupleIdx < totalPoints; tupleIdx++) { - dataStorePtr->setValue(i + tupleOffset, static_cast(rawDataPtr[i])); + dataStorePtr->setValue(tupleIdx + tupleOffset, static_cast(rawDataSpan[tupleIdx])); } } else diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp index fc27d488ed..efeb9d46cb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.hpp @@ -10,14 +10,13 @@ namespace nx::core /** * @class ReadH5OinaData - * @brief This filter will read a single .h5 file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the - * intermediate .h5ebsd file. + * @brief Reads one or more scans from an H5OINA file into one Image Geometry. */ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaData : public IEbsdOemReader { public: - ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues); + ReadH5OinaData(DataStructure& dataStructure, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel, ReadH5DataInputValues* inputValues); ~ReadH5OinaData() noexcept override; ReadH5OinaData(const ReadH5OinaData&) = delete; @@ -27,7 +26,7 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaData : public IEbsdOemReader operator()(); - Result<> copyRawEbsdData(int index) override; + Result<> copyRawEbsdData(int scanIndex) override; private: /** diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp index a2987e46a0..30394bf172 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.cpp @@ -23,10 +23,11 @@ #include #include -#include +#include #include #include -namespace fs = std::filesystem; + +#include using namespace nx::core; @@ -96,7 +97,9 @@ Parameters ReadH5OinaDataFilter::parameters() const //------------------------------------------------------------------------------ IFilter::VersionType ReadH5OinaDataFilter::parametersVersion() const { - return 1; + // Version 2: Pattern import is explicitly unsupported, and multi-scan inputs + // must have compatible geometry and identical phase definitions. + return 2; } //------------------------------------------------------------------------------ @@ -121,19 +124,18 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure DataPath cellEnsembleAMPath = pImageGeometryNameValue.createChildPath(pCellEnsembleAttributeMatrixNameValue); DataPath cellAMPath = pImageGeometryNameValue.createChildPath(pCellAttributeMatrixNameValue); - PreflightResult preflightResult; nx::core::Result resultOutputActions; std::vector preflightUpdatedValues; const std::string inputFilePath = pSelectedScanNamesValue.inputFilePath.string(); - if(pZSpacingValue <= 0) + if(!std::isfinite(pZSpacingValue) || pZSpacingValue <= 0) { - return MakePreflightErrorResult(-9580, fmt::format("The Z Spacing field contains a value ({}) that is non-positive. The Z Spacing field must be set to a positive value.", pZSpacingValue)); + return MakePreflightErrorResult(-9580, fmt::format("The Z Spacing value ({}) must be finite and positive.", pZSpacingValue)); } if(pSelectedScanNamesValue.scanNames.empty()) { - return MakePreflightErrorResult(-9581, "At least one scan must be chosen. Please select a scan from the list."); + return MakePreflightErrorResult(-9581, "At least one scan must be chosen. Please select a scan from the list."); } if(pReadPatternDataValue) { @@ -186,6 +188,21 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure firstScanName, inputFilePath, reader.getXDimension(), reader.getYDimension())); } + const auto validateSpacing = [&](const std::string& scanName, ebsdlib::H5OINAReader& scanReader) -> Result<> { + const float32 xStep = scanReader.getXStep(); + const float32 yStep = scanReader.getYStep(); + if(!std::isfinite(xStep) || !std::isfinite(yStep) || xStep <= 0.0F || yStep <= 0.0F) + { + return MakeErrorResult(-9591, fmt::format("Scan '{}' in '{}' reports X Step = {} and Y Step = {}. Both values must be finite and positive.", scanName, inputFilePath, xStep, yStep)); + } + return {}; + }; + + if(Result<> spacingCheck = validateSpacing(firstScanName, reader); spacingCheck.invalid()) + { + return MakePreflightErrorResult(spacingCheck.errors().front().code, spacingCheck.errors().front().message); + } + // The Ensemble Attribute Matrix is sized from the number of phase groups in the // FIRST selected scan, but the shared ensemble fill in IEbsdOemReader::readData runs // once per selected scan and places each phase at the index carried by its HDF5 group @@ -194,7 +211,7 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure // than the first scan, writes past the end of the ensemble arrays at execute. const auto phases = reader.getPhaseVector(); const usize ensemblePhaseCount = phases.size(); - const auto validatePhaseGroups = [&](const std::string& scanName, const auto& scanPhases) -> Result<> { + const auto validatePhaseGroups = [&](const std::string& scanName, const auto& scanPhases, bool comparePhaseDefinitions) -> Result<> { if(scanPhases.size() != ensemblePhaseCount) { return MakeErrorResult(-9589, fmt::format("Scan '{}' in '{}' declares {} phase group(s), but scan '{}' declares {}. Every selected scan must declare the same phase groups, because the single " @@ -210,11 +227,52 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure return MakeErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares {} phase(s), but one of them carries index {}. The phase groups of an H5OINA file must be named 1 through {}.", scanName, inputFilePath, ensemblePhaseCount, phaseIndex, ensemblePhaseCount)); } + + if(comparePhaseDefinitions) + { + const auto referencePhaseIter = std::find_if(phases.cbegin(), phases.cend(), [phaseIndex](const auto& referencePhase) { return referencePhase->getPhaseIndex() == phaseIndex; }); + if(referencePhaseIter == phases.cend()) + { + return MakeErrorResult(-9587, fmt::format("Scan '{}' in '{}' declares phase index {}, but scan '{}' does not. Every selected scan must use the same phase group names.", scanName, + inputFilePath, phaseIndex, firstScanName)); + } + + const auto& referencePhase = *referencePhaseIter; + if(phase->getPhaseName() != referencePhase->getPhaseName()) + { + return MakeErrorResult(-9590, + fmt::format("Phase group {} of scan '{}' in '{}' has material name '{}', but the same group of scan '{}' has material name '{}'. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, phase->getPhaseName(), firstScanName, referencePhase->getPhaseName())); + } + if(phase->getLaueGroup() != referencePhase->getLaueGroup()) + { + return MakeErrorResult(-9590, fmt::format("Phase group {} of scan '{}' in '{}' has Laue group {}, but the same group of scan '{}' has Laue group {}. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, static_cast(phase->getLaueGroup()), firstScanName, static_cast(referencePhase->getLaueGroup()))); + } + if(phase->getSpaceGroup() != referencePhase->getSpaceGroup()) + { + return MakeErrorResult(-9590, fmt::format("Phase group {} of scan '{}' in '{}' has space group {}, but the same group of scan '{}' has space group {}. The selected scans form one 3D " + "microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, phase->getSpaceGroup(), firstScanName, referencePhase->getSpaceGroup())); + } + + const std::vector scanLatticeConstants = phase->getLatticeConstants(); + const std::vector referenceLatticeConstants = referencePhase->getLatticeConstants(); + if(scanLatticeConstants != referenceLatticeConstants) + { + return MakeErrorResult(-9590, + fmt::format("Phase group {} of scan '{}' in '{}' has lattice constants [{}], but the same group of scan '{}' has lattice constants [{}]. The selected scans form " + "one 3D microstructure and must use identical phase definitions.", + phaseIndex, scanName, inputFilePath, fmt::join(scanLatticeConstants, ", "), firstScanName, fmt::join(referenceLatticeConstants, ", "))); + } + } } return {}; }; - if(Result<> phaseCheck = validatePhaseGroups(firstScanName, phases); phaseCheck.invalid()) + if(Result<> phaseCheck = validatePhaseGroups(firstScanName, phases, false); phaseCheck.invalid()) { return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); } @@ -239,6 +297,10 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure return MakePreflightErrorResult( -9582, fmt::format("An error occurred while reading the header of scan '{}' in '{}'.\n Error Code: {}\n Message: {}", scanName, inputFilePath, err, scanCheckReader.getErrorMessage())); } + if(Result<> spacingCheck = validateSpacing(scanName, scanCheckReader); spacingCheck.invalid()) + { + return MakePreflightErrorResult(spacingCheck.errors().front().code, spacingCheck.errors().front().message); + } if(scanCheckReader.getXDimension() != reader.getXDimension() || scanCheckReader.getYDimension() != reader.getYDimension() || scanCheckReader.getXStep() != reader.getXStep() || scanCheckReader.getYStep() != reader.getYStep()) { @@ -248,14 +310,14 @@ IFilter::PreflightResult ReadH5OinaDataFilter::preflightImpl(const DataStructure scanName, inputFilePath, scanCheckReader.getXDimension(), scanCheckReader.getYDimension(), scanCheckReader.getXStep(), scanCheckReader.getYStep(), firstScanName, reader.getXDimension(), reader.getYDimension(), reader.getXStep(), reader.getYStep())); } - if(Result<> phaseCheck = validatePhaseGroups(scanName, scanCheckReader.getPhaseVector()); phaseCheck.invalid()) + if(Result<> phaseCheck = validatePhaseGroups(scanName, scanCheckReader.getPhaseVector(), true); phaseCheck.invalid()) { return MakePreflightErrorResult(phaseCheck.errors().front().code, phaseCheck.errors().front().message); } } } - // create the Image Geometry and it's attribute matrices + // Create the Image Geometry and its attribute matrices. const CreateImageGeometryAction::DimensionType dims = {static_cast(reader.getXDimension()), static_cast(reader.getYDimension()), pSelectedScanNamesValue.scanNames.size()}; const ShapeType tupleDims = {dims[2], dims[1], dims[0]}; { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp index 103700b0f0..690dc5a2b5 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ReadH5OinaDataFilter.hpp @@ -116,4 +116,3 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5OinaDataFilter : public IFilter } // namespace nx::core SIMPLNX_DEF_FILTER_TRAITS(nx::core, ReadH5OinaDataFilter, "fad3d47f-f1e1-4429-bc65-5e021be62ba0"); -/* LEGACY UUID FOR THIS FILTER 3ff4701b-3a0c-52e3-910a-fa927aa6584c */ diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp index 5ad8bf822e..7c6aa4503d 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5OinaDataTest.cpp @@ -6,11 +6,11 @@ * no DREAM3D 6.5.171 H5OINA importer to compare against, so the oracle carries * the whole burden: * - * - .h5oina parsing : Class 2 (EbsdLib reference, trusted & NOT re-tested). - * EbsdLib's H5OINAReader owns HDF5 traversal, header and - * phase parsing, required-dataset enforcement and its own - * error codes. We do not re-test any of that; we do pin - * the codes it hands back through the filter. + * - .h5oina parsing : Part of the system under test. EbsdLib's H5OINAReader + * owns HDF5 traversal, header and phase parsing, required- + * dataset enforcement, and its own error codes. The tests + * verify its user-visible results against analytical fixture + * values and the independent h5py readback. * - SIMPLNX value-add : Class 1 (analytical) + Class 4 (invariant). The filter's * value-add is deterministic plumbing on top of the reader: * geometry construction from the first scan's header, array @@ -86,6 +86,7 @@ #include #include +#include #include #include @@ -542,6 +543,7 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle" // Class 4 invariant: the ensemble matrix always carries exactly one more tuple // than the file has phase groups. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures))); REQUIRE(dataStructure.getDataRefAs(k_EnsembleAMPath.createChildPath(ebsdlib::AngFile::CrystalStructures)).getNumberOfTuples() == 3); UnitTest::CheckArraysInheritTupleDims(dataStructure); @@ -724,6 +726,11 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Parameter Rejections", "[O args = MakeArgs(inputFile, {"1"}, 1.0F, true, true, true); expectedCode = -9583; } + SECTION("Non-finite Z Spacing (-9580)") + { + args = MakeArgs(inputFile, {"1"}, std::numeric_limits::quiet_NaN()); + expectedCode = -9580; + } auto preflightResult = filter.preflight(dataStructure, args); SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); @@ -901,6 +908,32 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Phase Index Out Of Range r UnitTest::CheckArraysInheritTupleDims(dataStructure); } +//------------------------------------------------------------------------------ +// EbsdLib guard: phase group names must be strict positive integers. The +// pre-correction reader passed the name directly to std::stoi(), which threw an +// exception for this file instead of returning a user-facing reader error. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Phase Group Name rejected (-9582)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureBScan1(); + scan.phases[0].groupName = "phase-one"; + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_invalid_phase_group_name.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9582); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find("phase-one") != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + //------------------------------------------------------------------------------ // Value-add guard: the ensemble arrays are sized from the FIRST selected scan's // phase count, and every selected scan fills them from its own phase groups. A @@ -943,6 +976,97 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Phase Count Mismatch UnitTest::CheckArraysInheritTupleDims(dataStructure); } +//------------------------------------------------------------------------------ +// A multi-scan import creates one 3D microstructure with one shared Ensemble +// Attribute Matrix. Therefore, the phase definition at each index must be the +// same in every selected scan. A mismatch cannot be represented correctly. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Scan Phase Definition Mismatch rejected (-9590)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan2 = MakeFixtureBScan2(); + std::string mismatchField; + + SECTION("Material name") + { + scan2.phases[0].name = "Different cubic phase"; + mismatchField = "material name"; + } + SECTION("Laue group") + { + scan2.phases[0].laueGroup = k_LaueHexagonalHigh; + mismatchField = "Laue group"; + } + SECTION("Space group") + { + scan2.phases[0].spaceGroup = 194; + mismatchField = "space group"; + } + SECTION("Lattice dimensions") + { + scan2.phases[0].latticeDimensions[0] = 4.25F; + mismatchField = "lattice constants"; + } + SECTION("Lattice angles") + { + scan2.phases[0].latticeAngles[2] = 2.0943952F; + mismatchField = "lattice constants"; + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_phase_definition_mismatch.h5oina", {MakeFixtureBScan1(), scan2}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1", "2"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9590); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find("'2'") != std::string::npos); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(mismatchField) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +//------------------------------------------------------------------------------ +// The X and Y step values become the Image Geometry spacing. They must be finite +// and positive before the geometry action is created. +//------------------------------------------------------------------------------ +TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Invalid Scan Spacing rejected (-9591)", "[OrientationAnalysis][ReadH5OinaDataFilter]") +{ + UnitTest::LoadPlugins(); + + ScanSpec scan = MakeFixtureAScan(); + + SECTION("Zero X Step") + { + scan.xStep = 0.0F; + } + SECTION("Negative Y Step") + { + scan.yStep = -0.5F; + } + SECTION("Non-finite X Step") + { + scan.xStep = std::numeric_limits::quiet_NaN(); + } + + const fs::path inputFile = WriteH5OinaFixture("read_h5oina_vv_invalid_spacing.h5oina", {scan}); + + ReadH5OinaDataFilter filter; + DataStructure dataStructure; + const Arguments args = MakeArgs(inputFile, {"1"}); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_INVALID(preflightResult.outputActions); + REQUIRE(preflightResult.outputActions.errors()[0].code == -9591); + REQUIRE(preflightResult.outputActions.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + //------------------------------------------------------------------------------ // Value-add guard: EbsdLib sizes its data buffers to the ACTUAL dataset extent // while the geometry is sized from the header's cell counts. A file whose Data @@ -1007,12 +1131,13 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: Out-of-Range Phase Value r auto executeResult = filter.execute(dataStructure, args); SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); REQUIRE(executeResult.result.errors()[0].code == -34972); + REQUIRE(executeResult.result.errors()[0].message.find(inputFile.filename().string()) != std::string::npos); UnitTest::CheckArraysInheritTupleDims(dataStructure); } //------------------------------------------------------------------------------ -// Error propagation from the trusted EbsdLib boundary. A Data group missing one +// Error propagation from the EbsdLib boundary. A Data group missing one // of the nine required datasets is fatal inside H5OINAReader::readData; the // filter surfaces it as -8970 with the reader's own code and message attached. //------------------------------------------------------------------------------ @@ -1044,7 +1169,7 @@ TEST_CASE("OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough } //------------------------------------------------------------------------------ -// Error propagation from the trusted EbsdLib boundary at header-read time: a +// Error propagation from the EbsdLib boundary at header-read time: a // phase group missing its Lattice Angles dataset is reported through -9582, and // the message carries the file path and scan name. // diff --git a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md index 700960a735..dc73f17233 100644 --- a/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/ReadH5OinaDataFilter.md @@ -1,184 +1,186 @@ # V&V Report: ReadH5OinaDataFilter -| | | -|--------|--------------| +| | | +|---|---| | Plugin | OrientationAnalysis | | SIMPLNX UUID | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | -| SIMPLNX Human Name | Read Oxford Aztec Data (.h5oina) | -| DREAM3D 6.5.171 equivalent | **None.** DREAM3D 6.5.171 has no H5OINA/AZtec importer of any kind (verified by a case-insensitive search of `DREAM3D/Source`, `SIMPL/Source` and `DREAM3D_Plugins` in the 6.5.171 tree at `/Users/mjackson/Workspace/D3D_v6.5.171`: zero source hits — the one match is a byte sequence inside a ZeissImport sample `.bmp`). The filter was written in SIMPLNX (PR #700, `a51dd5f3d`, 2024-03-25) and carries no `FromSIMPLJson` and no legacy-UUID mapping entry. | +| DREAM.3D 6.5.171 equivalent | **None.** | | Verified commit | ** | | Status | READY FOR REVIEW | -| Sign-off | Michael A. Jackson — 2026-08-24. Second engineer: **. | +| Sign-off | Pending second-engineer PR review. | ## At a glance -| Aspect | Current state | -|------------------------|---------------| -| Algorithm Relationship | **New filter, no legacy equivalent.** Nothing in DREAM3D 6.5.171 imports H5OINA, so there is no port to classify and no legacy behavior to inherit or defend. The filter is one of three siblings (`ReadH5OimData`, `ReadH5EspritData`) built on the shared `IEbsdOemReader` template; EbsdLib's `H5OINAReader` does the parsing. | -| Oracle (confirmed) | **Confirmed. Class 1 (analytical) + Class 4 (invariant), with a Class 2 independent readback for production data.** EbsdLib's `H5OINAReader` is the trusted Class 2 boundary and is not re-tested. Three toy fixture specifications are written by the test itself with H5Lite and materialise into twenty-three `.h5oina` files at run time, and the archived production AZtec file is compared against a readback of its own data sets. Encoded as 17 TEST_CASEs in `test/ReadH5OinaDataTest.cpp`; all pass. | -| Code paths enumerated | 23 of 30 paths exercised (see Code path coverage). The seven gaps are one `-9582` return site needing permission manipulation, the display-only preflight information values, the shared `-8971` empty-phase path (unreachable from this filter), two defensive branches of the data-set extent probe, the cancel-signal early returns, and three EbsdLib phase-read codes with no constructible fixture. | -| Tests today | 17 test cases: the Class 1+4 analytical oracle, a 2×2 conversion-option sweep, two multi-scan cases, a stacking-order case, a three-way Format Version sweep, eight value-add rejection cases with the error code pinned per section, two EbsdLib error passthroughs, and the Class 2 readback of the production AZtec file. Fixtures are written at test time; no new archive. | -| Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained, SHA512 `346573ac…d140ea03`, unchanged.** Its genuine Oxford AZtec `.h5oina` is kept as an irreplaceable production input. Its `H5Oina_Test_Data.dream3d` exemplar is no longer consulted: that file was written by this filter, so comparing against it pinned the filter to itself. Documented in `vv/provenance/ReadH5OinaDataFilter.md`. | -| Legacy comparison | **Not run — no legacy equivalent (verified against the 6.5.171 tree).** Its place is taken by the Class 2 independent readback described under Oracle. | -| Bug flags | D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11 — every one a bug under the root-cause taxonomy, every one present in DREAM3D-NX 7.0.0 through 7.4.1, all resolved. Two of them (D1, D6) change the value of a published output; D6 is labelled a breaking change. | -| V&V phase | Discovery, relationship, oracle, reconciliation, algorithm review, tests, deviations, provenance, docs — **complete**. The EbsdLib-side corrections (D5, D6, D7, D10, D11) reach users only through EbsdLib 3.1.1, which the `vcpkg.json` pin requires. In-core build and tests pass in `NX-Com-Qt69-Vtk96-Rel-EbsdLib`; OOC build skipped — the filter's writes are single-pass forward-sequential into freshly created arrays, and the plan for this batch scopes verification to in-core reader plumbing (approved in the batch plan for this filter). Second-engineer sign-off outstanding. | +| Aspect | Current state | +|---|---| +| Algorithm Relationship | **New filter, no legacy equivalent.** `ReadH5OinaDataFilter` was added in SIMPLNX and first shipped in DREAM3D-NX 7.0.0. | +| Oracle (confirmed) | **Class 1 analytical + Class 4 invariant, with Class 2 independent h5py readback.** The tests create small H5OINA inputs and compare all imported values with independently derived values. | +| Code paths enumerated | **26 of 33 paths exercised.** The uncovered paths require file-permission changes, file replacement during execution, cancel injection, or HDF5 objects that the fixture writer cannot create. | +| Tests today | **20 test cases and 9,943 assertions through ctest.** The tests cover data conversion, multi-scan stacking, phase definitions, malformed inputs, error propagation, and a real AZtec file. | +| Exemplar archive | **`H5Oina_Test_Data.tar.gz` retained as input only.** The vendor `.h5oina` file is used. The generated `.dream3d` output was retired as a circular oracle. | +| Legacy comparison | **Not applicable.** DREAM.3D 6.5.171 has no H5OINA importer. The independent h5py readback replaces the legacy comparison. | +| Bug flags | **Thirteen bugs resolved:** `ReadH5OinaDataFilter-D1` through `ReadH5OinaDataFilter-D13`. The affected released versions are DREAM3D-NX 7.0.0 through 7.4.1. | +| V&V phase | **COMPLETE** | ## Summary -`ReadH5OinaDataFilter` ("Read Oxford Aztec Data (.h5oina)") imports one or more scans from an Oxford Instruments AZtec `.h5oina` file into a single Image Geometry: it builds the geometry from the first selected scan's header, creates the nine cell arrays and the three ensemble arrays, copies each scan's data into its own tuple slab in the order the stacking-order setting asks for, optionally widens the file's `uint8` Phase column to `int32`, and optionally applies the EDAX/TSL hexagonal x-axis alignment to φ2. Verification is Class 1 analytical plus Class 4 invariant on hand-authored `.h5oina` fixtures, with a Class 2 independent readback standing in for the legacy A/B comparison that cannot exist — DREAM3D 6.5.171 has no H5OINA importer. Headline result: eleven defects were found, six in the filter and five in EbsdLib's `H5OINAReader`, including two crashes and a silently wrong orientation for every hexagonal point at the shipped default settings; all are corrected and pinned, and all 17 tests pass. +`ReadH5OinaDataFilter` imports one or more Oxford Instruments AZtec H5OINA scans into one Image Geometry. Verification uses inline Class 1 analytical data, Class 4 invariants, and an independent Class 2 h5py readback of a vendor file. The V&V found and fixed thirteen defects in the filter and EbsdLib H5OINA reader. -## Algorithm Relationship +## Dependency state -*Classification:* **New filter.** +- EbsdLib version 3.1.2 +- Intended DREAM3D-NX release: 7.5.0 -*Evidence:* A case-insensitive search for `oina` across `D3D_v6.5.171/DREAM3D/Source`, `.../SIMPL/Source` and `.../DREAM3D_Plugins` returns zero source hits; the single match is a byte sequence inside `DREAM3D_Plugins/ZeissImport/Data/ZeissImport/SampleMosaic/SampleMosaic_p0.bmp`. The `aztec` hits in that tree are the `H5Aztec` file-version constant belonging to DREAM3D's own HDF5-CTF archive format, which is unrelated to Oxford's H5OINA. The filter has no `FromSIMPLJson`, no `SIMPLConversion` include, no entry in the plugin's legacy-UUID mapping and no SIMPL conversion fixtures — all consistent with a filter that never existed in SIMPL. It was added by PR #700 (`a51dd5f3d`, 2024-03-25). +## Algorithm Relationship -Because there is no legacy equivalent, the same-UUID equivalence claim that drives the Deviations gate does not apply; the Deviations file instead records the differences between what DREAM3D-NX 7.0.0–7.4.1 shipped and the corrected behavior. +**New filter, no legacy equivalent.** -*PRs since introduction that touched these files:* the full list between `a51dd5f3d` and `d65d859b5` is #874, #934, #937, #996, #1088, #1152, #1187, #1238, #1263, #1438, #1439, #1472 and #1576. The ones that changed behavior rather than form are #996 (OEM reader error messages), #1088 (parameter versioning), #1152 (spacing/origin ordering), #1263 (phase info in preflight values), #1472 (EbsdLib 2.0.0 API migration) and #1576 (error-message sweep). None of them touched the hexagonal-alignment constant, the multi-scan offsets or the pattern path; those three carried their defects from the initial import through every subsequent change. +*Evidence:* DREAM.3D 6.5.171 contains no H5OINA importer. The filter was added to SIMPLNX by PR #700 in commit `a51dd5f3d` and has no SIMPL conversion function or legacy UUID mapping. ## Oracle -*Class:* **1 (Analytical) + 4 (Invariant)**, plus **2 (Independent readback)** for the production file; EbsdLib parsing = **Class 2 boundary (trusted, not re-tested)**. - -### The EbsdLib boundary (what we do NOT re-test) - -EbsdLib's `H5OINAReader` owns HDF5 traversal, the header and phase-group parsing, the nine required data-set reads, the Laue-group-to-crystal-structure mapping in `CtfPhase`, and its own error codes. Those behaviors are upstream's to verify, and the tests pin only that their codes and messages reach the user. The filter's value-add — everything this oracle covers — is the deterministic plumbing on top: geometry construction from the first scan's header, array creation and typing, per-scan slab offsets and their stacking order, the verbatim column copies, the `uint8`→`int32` Phase widening, the hexagonal φ2 alignment, ensemble slot-0 defaults, and the value-add rejection paths. - -Five of the eleven deviations sit *inside* that boundary but corrupt user-visible SIMPLNX output, crash the process, or blank out the reason a file was rejected (D5, D6, D7, D10, D11). They are corrected upstream on `topic/3_1_1_staging` rather than worked around in the filter, which is what creates the EbsdLib 3.1.1 release dependency. - -### Applied - -Toy `.h5oina` files are written by the test itself with `H5Support::H5Lite` into the binary test-output directory, from a fixture specification declared as C++ structs at the top of `test/ReadH5OinaDataTest.cpp`. Three fixture specifications materialise into twenty-three files at run time: **eight** that are imported successfully and **fifteen** that back a rejection or passthrough case. The eight that import carry exactly the dataset set `H5OINAReader` requires — the four `Header` scalars, one or more `Phases/` groups with `Phase Name` / `Lattice Dimensions` / `Lattice Angles` / `Laue Group` / `Space Group`, and the nine `Data` datasets — plus the inert root `Manufacturer` / `Software Version` / `Index` datasets for realism, and all but one of them also carry `Format Version`, which is not in the required set (which is why the variant that omits it still imports). Of the fifteen behind the rejection and passthrough cases, three omit a required dataset outright — `Data/Bands` once and `Phases//Lattice Angles` twice — while the other twelve carry the full dataset set and are rejected on a parameter value, on a header or phase value, or on how the scans are selected. - -- **Fixture A** — one scan, 3 × 2 cells, steps 0.25 / 0.5, a hexagonal phase (Laue 9) and a cubic phase (Laue 11), with Phase values `{1, 2, 0, 2, 1, 1}` so an unindexed point is present. Points 0, 1 and 2 all carry φ2 = `0.1F` on a hexagonal, a cubic and an unindexed point respectively, so the three expectations differ only through the alignment branch. The hexagonal phase's lattice angles are 90/90/120 degrees stored as radians, so γ ≠ β. -- **Fixture B** — two scans, 2 × 2 each, cubic only, with every one of the nine columns disjoint between the two scans. With no hexagonal point present the Euler array must be a pure verbatim copy, which isolates slab placement and stacking order. -- **Fixture C** — two scans, 2 × 2 each, every point hexagonal, so a shift applied to the wrong slab or applied twice changes pinned values. -- **Guard fixtures** — zero / negative cell counts, mismatched scan grids, a missing scan name in first and second position, a phase group named outside 1..N in the first and in a later scan, a later scan whose phase-group count differs from the first scan's in both directions, an 8-row data set behind a 16-cell header, an out-of-range phase byte, a missing `Data/Bands`, and a phase group missing `Lattice Angles` in the first and in a later scan. -- **Format Version variants** — `"5.0"`, `"2.0"` and absent, which must all produce identical output. +*Class:* **1 (Analytical) + 4 (Invariant)**, with **2 (Independent readback)** for the vendor file. -Expected values are derived from the fixture specification, not from observed output. Every floating-point value written into a toy fixture is a float32 literal stored as float32, so the file round-trips it bit-for-bit and verbatim copies are asserted with exact equality; the cell counts and the Laue and space group numbers are `int32` scalars and five of the nine data columns are `uint8`, all of which round-trip exactly as well. (The production file's values are whatever AZtec wrote; that test asserts exact equality too, but against an independent readback of the file rather than against literals.) The hexagonal expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, derived independently with IEEE-754 float32/float64 semantics in NumPy and embedded as literals with derivation comments. The derivation script and its recorded output are described in `vv/provenance/ReadH5OinaDataFilter.md`. +*Applied:* The tests write small H5OINA files from explicit fixture specifications. Expected geometry, cell arrays, ensemble arrays, unit conversions, stacking order, and rejection results are derived from those specifications. A separate h5py script reads the vendor file and derives its expected imported values without EbsdLib or SIMPLNX. -**Precision pinning.** φ2 values are chosen where the float32 result *differs* between a double-precision intermediate and a float32 intermediate: `0.1F` and `0.34F` in Fixture A and Fixture C slab 0, and `0.09F`, `0.22F`, `0.35F` in Fixture C slab 1. None of those five is an exactly representable decimal, which is deliberate — a dyadic value has trailing zero mantissa bits, so the sum rounds to the same float32 under either intermediate and cannot separate the two paths. `0.25F`, `0.5F` and `0.75F` are exactly representable and are carried alongside on hexagonal points as controls that round identically either way, so the suite distinguishes magnitude errors from arithmetic-shape errors. The hexagonal fixture phase's third lattice angle is a sixth discriminator of the same kind, on the EbsdLib radians-to-degrees conversion rather than on the alignment: `2.0943952F → 120.0F` under a double intermediate and `120.00000762939453F` under a float32 one. The discriminating values were found by an exhaustive sweep of the float32 values in `[0.25, 6.5)`, of which 8,289,627 of 38,797,312 separate the two paths. +*Encoded:* `test/ReadH5OinaDataTest.cpp` contains 20 test cases. The archived `h5oina_oracle.py` script reproduces its Class 1 values and the vendor-file readback. -Class 4 invariants encoded: the ensemble matrix always carries exactly one more tuple than the file has phase groups; slot 0 always holds `UnknownCrystalStructure` / `"Invalid Phase"` / zeroed lattice constants; the Phase values are unchanged by either conversion option; the hexagonal shift never reaches a cubic or an unindexed point; and the geometry is identical under both stacking orders. +*Second-engineer review:* Pending PR review. -### The Class 2 independent readback (substitute for the legacy A/B) +## Bugs found and fixed -There is no DREAM3D 6.5.171 H5OINA importer, so the comparison that would normally establish behavioral continuity does not exist. Its place is taken by an independent readback of the production AZtec file: +This branch fixes all defects in this table. The fixes are intended for DREAM3D-NX 7.5.0. EbsdLib defects require EbsdLib 3.1.2. -- `test/ReadH5OinaDataTest.cpp::"Real AZtec File Readback"` reads the archived `H5Oina_Test_Data.h5oina`'s own `Data` datasets with `H5Lite` — the file bytes, bypassing `H5OINAReader` entirely — and compares them element-wise against the filter's output. The file is a single 25 × 25 cubic scan with 625 points, both indexed and unindexed, and the comparison covers all nine cell arrays (6,966 assertions in that test case). -- The same readback is performed out of band with h5py, in a second language and a second HDF5 binding, re-deriving the geometry, the ensemble values and the cell arrays from the documented rules. It is the source of the ensemble literals pinned in that test case (`CrystalStructures {999, 1}`, `MaterialName {"Invalid Phase", "Titanium cubic"}`, `LatticeConstants {3.192, 3.192, 3.192, 90, 90, 90}`). The script and its recorded output are described in `vv/provenance/ReadH5OinaDataFilter.md`. - -This file cannot exercise the hexagonal alignment (its only phase is cubic), the multi-scan slab offsets or the stacking order (it holds one scan); the toy fixtures carry those paths. - -*Second-engineer review:* Outstanding — to be recorded at PR review. The oracle design is auditable in the test source: the fixture specification, the derivation of every literal and the Python cross-check are all committed or recorded in the working folder described in the provenance sidecar. - -## Algorithm review - -Line-by-line review of `Algorithms/ReadH5OinaData.cpp` and the filter's `preflightImpl`. - -- **Correctness:** the hexagonal alignment adds 30 degrees expressed in radians on a double intermediate (D1); the Euler slab offset is an element offset rather than a tuple offset (D2); the alignment loop walks the scan's own slab (D3); the scan iteration honors the stacking order (D9). -- **Robustness:** seven malformed-input rejections (`-9584`, `-9585`, `-9586`, `-9587`, `-9589`, `-34971`, `-34972`), each naming the offending value, the scan and the file. The phase-group checks run for every selected scan, not only the first, because the ensemble arrays are sized from the first scan and filled from all of them (D8). -- **Dead code:** the execute-side pattern block was unreachable — preflight always failed first — and internally inconsistent, creating a `uint16` array that execute fetched as `UInt8Array`. It is removed along with its error code `-34970`, and preflight rejects the parameter honestly (D4). -- **Progress and cancel:** the scan loop lives in `ReadH5OinaData::operator()` rather than in the shared `IEbsdOemReader::execute()`, which is what lets the per-scan progress messages, the three cancel checks and the stacking order apply to this filter without changing the sibling filters. The shared `readData()` is still used. -- **Message quality:** the two `-9582` sites that name a scan carry the file path and the scan name. The third, on `readScanNames()`, carries the file path only, because no scan has been selected at that point. `-8970`'s message carries both and is pinned by a test. -- **Preflight cost:** preflight performs one `readScanNames` open, one first-scan header open and one further open per additional selected scan, and preflight fires on every GUI parameter edit. That is `O(S)` file opens per keystroke for a selection of `S` scans. It is the price of validating every selected scan's header rather than only the first, and it is bounded by the number of scans a file contains. -- **Documentation:** the page states that the reader ignores the file's `Format Version` value, names φ2 as the third Euler angle the hexagonal alignment shifts, lists the created outputs with their types and component counts, and recommends thresholding `Phase` > 0 to mask unindexed points. That last point is format-specific: the `.ctf` convention that `Error` = 0 marks a good point does not hold here. In the bundled AZtec export every one of the 587 indexed points carries `Error` = 1 and every one of the 38 un-indexed points carries `Error` = 2, and no point carries 0, so an `Error` = 0 mask selects nothing; the page says so. A Migration Notes section covers the two deviations that change a value a saved pipeline may compare against. -- **Not changed:** `utilities/IEbsdOemReader.hpp` is untouched, so `ReadH5OimData` and `ReadH5EspritData` are unaffected. See Follow-ups. +| Deviation | Defect | Affected released versions | Resolution in this branch | +|---|---|---|---| +| `ReadH5OinaDataFilter-D1` | The hexagonal alignment added 30 radians instead of 30 degrees to `phi2`. | DREAM3D-NX 7.0.0 through 7.4.1. | The filter adds 30 degrees expressed in radians with a double-precision intermediate. | +| `ReadH5OinaDataFilter-D2` | The Euler destination offset used tuples instead of elements. Later scans overwrote part of an earlier scan. | DREAM3D-NX 7.0.0 through 7.4.1. | The Euler copy multiplies the tuple offset by three components. | +| `ReadH5OinaDataFilter-D3` | Hexagonal alignment repeatedly modified the first scan and did not modify later scans. | DREAM3D-NX 7.0.0 through 7.4.1. | The alignment loop uses the current scan slab. | +| `ReadH5OinaDataFilter-D4` | Pattern import was exposed but could not succeed. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight reports that H5OINA pattern import is not supported and gives an alternative. | +| `ReadH5OinaDataFilter-D5` | The gamma lattice angle copied the beta angle. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib copies the third lattice angle into the gamma slot. | +| `ReadH5OinaDataFilter-D6` | H5OINA lattice angles were imported in radians while other EBSD readers use degrees. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib converts the lattice angles to degrees. | +| `ReadH5OinaDataFilter-D7` | A phase without a required lattice dataset caused invalid access and a process crash. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates the required phase datasets before it reads their values. | +| `ReadH5OinaDataFilter-D8` | Multi-scan phase groups could exceed the ensemble bounds or describe different phases at the same index. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight validates every phase index, phase count, and phase definition in every selected scan. | +| `ReadH5OinaDataFilter-D9` | The Stacking Order setting was accepted but ignored. | DREAM3D-NX 7.0.0 through 7.4.1. | The scan order now determines which scan is placed at Z = 0. | +| `ReadH5OinaDataFilter-D10` | H5OINA reader error messages were constructed but stored as empty strings. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib stores and forwards the complete error message. | +| `ReadH5OinaDataFilter-D11` | The reader did not validate both cell counts and returned an error code that differed from its stored code. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates rows and columns and returns the stored error code. | +| `ReadH5OinaDataFilter-D12` | A non-numeric phase-group name caused `std::stoi()` to throw and left an HDF5 group open. | DREAM3D-NX 7.0.0 through 7.4.1. | EbsdLib validates the complete group name before it opens the group. | +| `ReadH5OinaDataFilter-D13` | Non-positive or non-finite scan spacing could create an invalid Image Geometry. | DREAM3D-NX 7.0.0 through 7.4.1. | Preflight requires finite, positive X, Y, and Z spacing. | ## Code path coverage -*23 of 30 enumerated paths exercised. Source: `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5OinaData.cpp` (272 lines) + preflight in `Filters/ReadH5OinaDataFilter.cpp` (328 lines) + the shared read and ensemble fill in `utilities/IEbsdOemReader.hpp` (147 lines), which rows 16–18 live in.* Logical phases: **(a)** preflight, **(b)** execute read + ensemble population (shared `IEbsdOemReader::readData`), **(c)** per-scan cell-data copy. - -| # | Phase | Path | Test case | -|----|-------|------|-----------| -| 1 | (a) | `z_spacing <= 0` → `-9580` | `Parameter Rejections` (section "Non-positive Z Spacing") | -| 2 | (a) | empty scan-name list → `-9581` | `Parameter Rejections` (section "No Scan Names Selected") | -| 3 | (a) | `read_pattern_data` on → `-9583` | `Parameter Rejections` (section "Pattern Import Not Supported") | -| 4 | (a) | `readScanNames` failure → `-9582` | *Not directly tested — the file has to be present but unreadable, which needs permission manipulation. This is a distinct return statement from rows 6 and 9, and it is the one `-9582` site whose message carries the file path without a scan name.* | -| 5 | (a) | selected scan absent from the file → `-9586` | `Missing Scan Name rejected (-9586)` (sections: missing first scan; missing second scan) | -| 6 | (a) | first scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` (section "First selected scan") | -| 7 | (a) | `X Cells`/`Y Cells` < 1 → `-9584` | `Invalid Cell Counts rejected (-9584)` (sections: zero X; zero Y; negative X) | -| 8 | (a) | phase group of any selected scan named outside `[1, first scan's phase count]` → `-9587` | `Phase Index Out Of Range rejected (-9587)` (sections: first selected scan; later selected scan) | -| 9 | (a) | later scan `readHeaderOnly` failure → `-9582` | `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` (section "Later selected scan"; asserts the scan name this site injects) | -| 10 | (a) | later scan grid differs from the first → `-9585` | `Scan Header Mismatch rejected (-9585)` (sections: differing cell counts; differing step size) | -| 11 | (a) | later scan phase-group count differs from the first → `-9589` | `Scan Phase Count Mismatch rejected (-9589)` (sections: more phases; fewer phases) | -| 12 | (a) | geometry action: dims (X, Y, scan count), spacing (X Step, Y Step, z_spacing), user origin | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement`; `Multi-Scan Hexagonal Alignment` | -| 13 | (a) | ensemble matrix sized phase count + 1; three ensemble array actions | `Class 1 Analytical Oracle` (3 tuples, 2 phases) | -| 14 | (a) | nine cell-array actions with the Phase type chosen by `convert_phase_to_int32` | `Conversion Option Combinations` (both branches assert the Phase array's type) | -| 15 | (a) | preflight scan/phase information values | *Not directly tested — display-only values, exercised implicitly by every preflight and asserted by none.* | -| 16 | (b) | `readFile()` failure → `-8970` | `EbsdLib Error Passthrough - Missing Data Column (-8970)` (code and message content pinned) | -| 17 | (b) | empty phase vector → `-8971` | *Not directly tested — `H5OINAReader` rejects a file with no phase groups at header-read time with `-90009`, so preflight `-9582` fires first and this shared-code path is unreachable from this filter.* | -| 18 | (b) | ensemble slot-0 defaults and per-phase fill (Laue mapping, name, lattice constants) | `Class 1 Analytical Oracle`; `Real AZtec File Readback` | -| 19 | (b)/(c) | stacking order: Low-to-High reads the selection in list order, High-to-Low in reverse | `Stacking Order` (both sections; each pins values only its own order can produce) | -| 20 | (c) | the file cannot be reopened for the extent probe → `-34971` | *Not directly tested — the file was opened successfully moments earlier by the reader, so reaching this needs the file to be deleted or its permissions changed mid-execute.* | -| 21 | (c) | `getDatasetInfo` fails for a `Data` dataset → skip that dataset | *Not directly tested — a missing `Data` dataset is fatal inside `H5OINAReader`, which has already run, so `-8970` fires first (row 16) and this branch is defensive only.* | -| 22 | (c) | data-set extent disagrees with the header, in either direction → `-34971` | `Dataset Extent Mismatch rejected (-34971)` | -| 23 | (c) | four `uint8` verbatim copies into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | -| 24 | (c) | Euler copy at three times the tuple offset | `Multi-Scan Slab Placement` (24 values across two slabs) | -| 25 | (c) | phase value outside `[0, phase count]` → `-34972` | `Out-of-Range Phase Value rejected (-34972)` | -| 26 | (c) | Phase widened to `int32` / copied verbatim as `uint8` | `Conversion Option Combinations` (both branches) | -| 27 | (c) | three `float32` verbatim copies (MAD, X, Y) into the scan's slab | `Class 1 Analytical Oracle`; `Multi-Scan Slab Placement` | -| 28 | (c) | hexagonal alignment on the scan's own slab, Hexagonal-High points only | `Class 1 Analytical Oracle`; `Conversion Option Combinations`; `Multi-Scan Hexagonal Alignment` | -| 29 | (b)/(c) | cancel checks (3 sites) | *Not directly tested. Requires cancel-signal injection; standard early-return pattern. Excluded from scope by direction.* | -| 30 | (a) | EbsdLib phase-read failures `-90030` (unopenable phase group), `-90031` (`Lattice Dimensions`) and `-90033` (`Laue Group`) surfaced through `-9582` | *Not directly tested — the fixture writer emits every phase dataset or omits `Lattice Angles`, which is `-90032` and is covered by rows 6 and 9. `-90030` needs an HDF5 object that cannot be opened, which `H5Lite` cannot write; the other two need fixture switches this suite does not carry.* | +26 of 33 paths are exercised. + +Source: `Filters/ReadH5OinaDataFilter.cpp`, `Filters/Algorithms/ReadH5OinaData.cpp`, and the shared `utilities/IEbsdOemReader.hpp` read and ensemble-fill path. + +| # | Phase | Path | Test case | +|---|---|---|---| +| 1 | Preflight | Reject non-positive or non-finite Z spacing (`-9580`) | `Parameter Rejections` | +| 2 | Preflight | Reject an empty scan selection (`-9581`) | `Parameter Rejections` | +| 3 | Preflight | Reject pattern import (`-9583`) | `Parameter Rejections` | +| 4 | Preflight | Fail while listing scans (`-9582`) | *Not directly tested. Requires a present file that cannot be read.* | +| 5 | Preflight | Reject a selected scan that is not in the file (`-9586`) | `Missing Scan Name rejected` | +| 6 | Preflight | Fail while reading the first scan header (`-9582`) | `Missing Lattice Angles`; `Invalid Phase Group Name` | +| 7 | Preflight | Reject cell counts below one (`-9584`) | `Invalid Cell Counts rejected` | +| 8 | Preflight | Reject non-positive or non-finite X or Y spacing (`-9591`) | `Invalid Scan Spacing rejected` | +| 9 | Preflight | Reject a phase index outside the ensemble bounds (`-9587`) | `Phase Index Out Of Range rejected` | +| 10 | Preflight | Fail while reading a later scan header (`-9582`) | `Missing Lattice Angles` | +| 11 | Preflight | Reject a later scan with a different grid (`-9585`) | `Scan Header Mismatch rejected` | +| 12 | Preflight | Reject a later scan with a different phase count (`-9589`) | `Scan Phase Count Mismatch rejected` | +| 13 | Preflight | Reject a later scan with different phase definitions (`-9590`) | `Scan Phase Definition Mismatch rejected` | +| 14 | Preflight | Create geometry dimensions, spacing, origin, and cell matrix | Class 1 oracle; multi-scan tests | +| 15 | Preflight | Create the ensemble matrix and three ensemble arrays | Class 1 oracle | +| 16 | Preflight | Create nine cell arrays and select the Phase type | `Conversion Option Combinations` | +| 17 | Preflight | Generate display-only scan and phase information | *Not directly tested. The values are shown only in preflight.* | +| 18 | Execute | Propagate an EbsdLib data-read failure (`-8970`) | `Missing Data Column` | +| 19 | Execute | Reject an empty phase vector (`-8971`) | *Not directly tested. H5OINAReader rejects this file during preflight first.* | +| 20 | Execute | Initialize ensemble tuple 0 and fill phase tuples | Class 1 oracle; real-file readback | +| 21 | Execute | Apply Low-to-High or High-to-Low stacking | `Stacking Order` | +| 22 | Copy | Fail when the file cannot be reopened (`-34971`) | *Not directly tested. Requires file replacement during execution.* | +| 23 | Copy | Skip an extent probe that cannot inspect a dataset | *Not directly tested. The reader rejects a missing required dataset first.* | +| 24 | Copy | Reject a dataset extent that differs from the header (`-34971`) | `Dataset Extent Mismatch rejected` | +| 25 | Copy | Copy the four `uint8` arrays into the current scan slab | Class 1 oracle; multi-scan placement | +| 26 | Copy | Copy Euler values with a three-component offset | `Multi-Scan Slab Placement` | +| 27 | Copy | Reject a Phase value outside the ensemble bounds (`-34972`) | `Out-of-Range Phase Value rejected` | +| 28 | Copy | Widen Phase to `int32` or retain `uint8` | `Conversion Option Combinations` | +| 29 | Copy | Copy MAD, X, and Y into the current scan slab | Class 1 oracle; multi-scan placement | +| 30 | Copy | Apply hexagonal alignment only to Hexagonal-High points in the current slab | Class 1 oracle; option sweep; multi-scan alignment | +| 31 | Execute/Copy | Return at the three cancellation checks | *Not directly tested. Requires cancel-signal injection.* | +| 32 | EbsdLib | Reject a noncanonical phase-group name (`-90034`, surfaced as `-9582`) | `Invalid Phase Group Name rejected` | +| 33 | EbsdLib | Reject an unreadable phase group, lattice dimensions, or Laue group | *Not directly tested. The fixture writer cannot create the unreadable-group case; the other required-dataset cases use the same checked path as lattice angles.* | ## Test inventory | Test case | Status | Notes | -|-----------|--------|-------| -| `OrientationAnalysis::ReadH5OinaDataFilter: Class 1 Analytical Oracle` | new-for-V&V | Class 1 + 4 over Fixture A. Geometry, all nine cell arrays and all three ensemble arrays asserted element-wise; the ensemble tuple-count invariant. | -| `…: Conversion Option Combinations` | new-for-V&V | Class 1 + 4. `GENERATE` over the 2 × 2 hexagonal-alignment × phase-conversion grid; pins the Phase array's type in each branch and that the alignment never reaches cubic or unindexed points. | -| `…: Multi-Scan Slab Placement` | new-for-V&V | Class 1 over Fixture B (cubic, two scans). 24 Euler values plus eight other arrays across two slabs; regression pin for D2. | -| `…: Multi-Scan Hexagonal Alignment` | new-for-V&V | Class 1 over Fixture C (hexagonal, two scans). Regression pin for D3; 24 Euler values, five of which carry the precision discrimination and three of which are the dyadic controls. | -| `…: Format Version Variants` | new-for-V&V | `GENERATE` over `"5.0"` / `"2.0"` / absent; pins that the reader does not gate on the version and that the dataset is optional. | -| `…: Parameter Rejections` | new-for-V&V | Three sections pinning `-9580`, `-9581`, `-9583`. Replaces the previous invalid-execution test, whose sections asserted only "some error" and whose input file was in neither extracted archive. | -| `…: Invalid Cell Counts rejected (-9584)` | new-for-V&V | Three sections: zero X Cells, zero Y Cells, negative X Cells. | -| `…: Scan Header Mismatch rejected (-9585)` | new-for-V&V | Two sections: differing cell counts, differing step size. | -| `…: Missing Scan Name rejected (-9586)` | new-for-V&V | Two sections: the missing name in first and in second position. | -| `…: Phase Index Out Of Range rejected (-9587)` | new-for-V&V | Two sections: a phase group named `7` in the first selected scan, and the same in a later selected scan. The second fixture passes preflight and crashes the process against the pre-correction filter; regression pin for D8. | -| `…: Scan Phase Count Mismatch rejected (-9589)` | new-for-V&V | Two sections: a later scan declaring more phase groups than the first, and one declaring fewer. The first fixture passes preflight and crashes the process against the pre-correction filter; regression pin for D8. | -| `…: Dataset Extent Mismatch rejected (-34971)` | new-for-V&V | A 4 × 4 header over 8-row data sets; deterministic, no file-mutation injection needed. | -| `…: Out-of-Range Phase Value rejected (-34972)` | new-for-V&V | A one-phase file with a Phase byte of 5. | -| `…: EbsdLib Error Passthrough - Missing Data Column (-8970)` | new-for-V&V | Pins the code and that the message names the scan and the file. | -| `…: EbsdLib Error Passthrough - Missing Lattice Angles (-9582)` | new-for-V&V | Two sections, one per `-9582` return site, each asserting the file path and the scan name that site injects. This fixture crashed the process before D7 was corrected. | -| `…: Stacking Order` | new-for-V&V | Two sections over Fixture B, one per order, each pinning Euler, Band Contrast and Phase values that only that order can produce; regression pin for D9. Replaces `Stacking Order Warning (-9588)`, which pinned a warning that no longer exists. | -| `…: Real AZtec File Readback` | kept | Class 2. Was `Valid Filter Execution`, which compared against the archive's `.dream3d` exemplar — a file this filter had written. It now compares all nine cell arrays element-wise against an `H5Lite` readback of the `.h5oina`'s own data sets, with the geometry and the ensemble values pinned from the h5py derivation; 6,966 assertions. | -| *(retired)* `…: InValid Filter Execution` | retired | Its sentinel extracted `6_6_ImportH5Data.tar.gz` while its paths pointed into `H5Oina_Test_Data/`, and the file it named exists in neither archive, so the "incompatible manufacturer" section passed on file-not-found. Replaced by `Parameter Rejections` and the eight code-pinned rejection cases plus the two EbsdLib passthroughs. | - -All 17 pass in the EbsdLib preset build `NX-Com-Qt69-Vtk96-Rel-EbsdLib`, built and run at the tree state one commit before the `vcpkg.json` pin is raised to EbsdLib 3.1.1 — the pin makes the head unconfigurable until that release exists, so every measurement here is from the pre-pin tree, whose sources are otherwise identical. The whole `OrientationAnalysis::` suite is 308 tests with one failure at that state, `ComputeSchmidsFilter`, which is the known EbsdLib-staging numerical drift and is analysed in `vv/provenance/ReadH5OinaDataFilter.md`. The filter suite reports **9,374 assertions** when the 17 test cases run in one process. A `ctest -V` run sums to **9,390** instead, because `UnitTest::LoadPlugins()` guards itself per process, so its single assertion is counted once in each of ctest's 17 processes rather than once overall; both numbers are correct for how they were measured. OOC build skipped, for the reason recorded in the V&V phase row. +|---|---|---| +| `Class 1 Analytical Oracle` | new-for-V&V | Verifies geometry, all cell arrays, all ensemble arrays, alignment, and invariants. | +| `Conversion Option Combinations` | new-for-V&V | Covers the 2 x 2 alignment and Phase-type option matrix. | +| `Multi-Scan Slab Placement` | new-for-V&V | Verifies all scan slabs with disjoint values. | +| `Multi-Scan Hexagonal Alignment` | new-for-V&V | Verifies alignment on each scan slab. | +| `Format Version Variants` | new-for-V&V | Covers version 5.0, version 2.0, and no version dataset. | +| `Parameter Rejections` | new-for-V&V | Covers Z spacing, empty selection, and unsupported pattern import. | +| `Invalid Cell Counts rejected` | new-for-V&V | Covers zero and negative cell counts. | +| `Scan Header Mismatch rejected` | new-for-V&V | Covers grid dimensions and step-size differences. | +| `Missing Scan Name rejected` | new-for-V&V | Covers missing first and later scan names. | +| `Phase Index Out Of Range rejected` | new-for-V&V | Covers invalid phase indices in the first and later scans. | +| `Invalid Phase Group Name rejected` | new-for-V&V | Verifies that EbsdLib reports a non-numeric group name without throwing. | +| `Scan Phase Count Mismatch rejected` | new-for-V&V | Covers later scans with more or fewer phase groups. | +| `Scan Phase Definition Mismatch rejected` | new-for-V&V | Covers material, Laue, space-group, lattice-dimension, and lattice-angle differences. | +| `Invalid Scan Spacing rejected` | new-for-V&V | Covers zero, negative, and non-finite scan spacing. | +| `Dataset Extent Mismatch rejected` | new-for-V&V | Verifies header and dataset extent agreement. | +| `Out-of-Range Phase Value rejected` | new-for-V&V | Verifies Phase values before ensemble indexing. | +| `Missing Data Column` | new-for-V&V | Verifies EbsdLib execute error propagation. | +| `Missing Lattice Angles` | new-for-V&V | Verifies EbsdLib preflight error propagation for first and later scans. | +| `Stacking Order` | new-for-V&V | Verifies both scan orders. | +| `Real AZtec File Readback` | kept, modified | Uses the vendor H5OINA input and independent readback. The generated DREAM3D exemplar is not used. | +| `InValid Filter Execution` | retired | The old test used a missing file and did not verify the intended error. | + +All 20 test cases pass. The ctest run reports 9,943 assertions. + +## Test sensitivity verification + +Test sensitivity verification introduces one temporary defect at a time and confirms that the applicable test fails. + +Eleven temporary defects were evaluated. Each defect caused the expected V&V test to fail. + +| Temporary defect | Test that detected the defect | Result | +|---|---|---| +| Restore the 30-radian alignment value. | Class 1 oracle; option sweep; multi-scan alignment; format variants | Detected | +| Use a float32 intermediate for alignment. | Class 1 oracle; option sweep; multi-scan alignment; format variants | Detected | +| Use the tuple offset for the Euler element offset. | Multi-scan slab placement; multi-scan alignment | Detected | +| Apply alignment from tuple 0 for every scan. | Multi-scan alignment | Detected | +| Copy beta into the gamma lattice slot. | Class 1 oracle | Detected | +| Leave lattice angles in radians. | Class 1 oracle; real-file readback | Detected | +| Remove the dataset-extent validation. | Dataset extent mismatch | Detected | +| Pass an invalid phase-group name to `std::stoi()`. | Invalid phase group name | Detected | +| Disable phase-definition validation. | Scan phase definition mismatch | Detected | +| Disable spacing validation. | Parameter rejections; invalid scan spacing | Detected | +| Remove the file path from the Phase-range error. | Out-of-range Phase value | Detected | ## Exemplar archive -- **Archive:** `H5Oina_Test_Data.tar.gz`, SHA512 `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03`, matching the `download_test_data()` entry at `test/CMakeLists.txt:146`. Unchanged — no re-upload. -- **Retained:** `H5Oina_Test_Data.h5oina`, a genuine Oxford AZtec export (Format Version 5.0, 25 × 25, 4 µm steps, one cubic titanium phase). Irreplaceable production realism, used as the input to the Class 2 readback. -- **No longer consulted:** `H5Oina_Test_Data.dream3d`. It was produced by running this filter on the sibling `.h5oina`, so it is a self-oracle. It also had the pre-correction radian lattice angles baked into `LatticeConstants`, which means the previous test actively enforced D6. -- **Provenance:** `src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md`. - -## Deviations from DREAM3D 6.5.171 - -**Not applicable — DREAM3D 6.5.171 has no H5OINA importer** (evidence in Algorithm Relationship). No legacy comparison was run and none is possible. +`H5Oina_Test_Data.tar.gz` is retained because it contains a vendor H5OINA input. The generated DREAM3D output was a circular oracle and was retired for this filter. The replacement uses inline Class 1 analytical data, Class 4 invariants, and the Class 2 h5py readback. -`vv/deviations/ReadH5OinaDataFilter.md` instead records, in the same structured form, the eleven differences between what DREAM3D-NX 7.0.0 through 7.4.1 shipped and the corrected behavior: +SHA512: `346573ac6b96983680078e8b0a401aa25bd9302dff382ca86ae4e503ded6db3947c4c5611ee603db519d8a8dc6ed35b044a7bfea9880fade5ab54479d140ea03`. -- `ReadH5OinaDataFilter-D1` — the hexagonal alignment added 30 radians to a radian-valued φ2 instead of 30 degrees; the option ships ON. -- `ReadH5OinaDataFilter-D2` — in a multi-scan import the Euler block of scan 2 onward landed a third of the way into its slab. -- `ReadH5OinaDataFilter-D3` — the hexagonal alignment was applied to the first scan once per scan and never to the others. -- `ReadH5OinaDataFilter-D4` — "Import Pattern Data" could never succeed and reported a misleading reason. -- `ReadH5OinaDataFilter-D5` — the third lattice angle was discarded and γ echoed β. -- `ReadH5OinaDataFilter-D6` — lattice angles were reported in radians while every other importer reports degrees. **Breaking change**; see its release note and migration section. -- `ReadH5OinaDataFilter-D7` — a phase group missing `Lattice Angles` crashed the process. -- `ReadH5OinaDataFilter-D8` — a multi-scan selection whose scans declared different phase groups wrote past the end of the ensemble arrays and crashed the process. -- `ReadH5OinaDataFilter-D9` — the Stacking Order setting was accepted and never applied. -- `ReadH5OinaDataFilter-D10` — every error message `H5OINAReader` composed was discarded, so failures reached the user with a blank reason. -- `ReadH5OinaDataFilter-D11` — `H5OINAReader::readData()` returned a code that did not match the one it set, and never validated its column count. +## Deviations from DREAM.3D 6.5.171 -D5, D6, D7, D10 and D11 are corrected in EbsdLib and reach users only through EbsdLib 3.1.1. +DREAM.3D 6.5.171 has no H5OINA importer. Therefore, no legacy comparison is possible. -## Follow-ups for the engineering team +The deviations document compares this verified version with `ReadH5OinaDataFilter` as shipped in DREAM3D-NX 7.0.0 through 7.4.1. See `vv/deviations/ReadH5OinaDataFilter.md` for the root cause, affected users, and recommendation for each defect. -1. **Sibling exposure.** `ReadH5OimData` and `ReadH5EspritData` share `utilities/IEbsdOemReader.hpp` and several of the same defect shapes. That header is unchanged, so the two siblings behave exactly as they did. Specifically: `ReadH5OimData::copyRawEbsdData` has the same pattern-array tuple-offset bug in its pattern copy loop; neither sibling validates that every selected scan exists or that later scans match the first scan's grid; neither range-checks the phase column; neither honors the Stacking Order setting the scan-selection parameter carries (D9's shape); and the ensemble fill in the shared `IEbsdOemReader::readData` writes `crystalStructures[phaseId]` with no bounds check, which this filter now prevents from its own preflight but the siblings do not. -2. **`-9587` and `-9589` are preflight checks over a shared write.** Both guard an out-of-range write that happens in the shared `IEbsdOemReader::readData` at execute. A file modified between preflight and execute would still reach that write. Closing it at the point of the write means bounds-checking the shared header, which changes all three filters. -3. **EbsdLib 3.1.1.** The corrections behind D5, D6, D7, D10 and D11 live on `topic/3_1_1_staging` and reach users only through that release; the `vcpkg.json` pin requires it. -4. **`H5OINAReader::getPatternDims(std::array)` takes its argument by value** and `getPatternData()` returns `nullptr`. Implementing pattern import for H5OINA is a feature, not a fix. +| Deviation | Observed difference | +|---|---| +| `ReadH5OinaDataFilter-D1` | Hexagonal `phi2` alignment uses 30 degrees instead of 30 radians. | +| `ReadH5OinaDataFilter-D2` | Every scan writes its Euler values to the correct tuple slab. | +| `ReadH5OinaDataFilter-D3` | Hexagonal alignment modifies each scan exactly once. | +| `ReadH5OinaDataFilter-D4` | Unsupported pattern import reports a clear preflight error. | +| `ReadH5OinaDataFilter-D5` | The gamma lattice slot contains gamma instead of beta. | +| `ReadH5OinaDataFilter-D6` | Lattice angles use degrees instead of radians. | +| `ReadH5OinaDataFilter-D7` | Missing phase datasets produce an error instead of invalid access. | +| `ReadH5OinaDataFilter-D8` | Multi-scan inputs must have safe phase indices and identical phase definitions. | +| `ReadH5OinaDataFilter-D9` | Stacking Order controls the scan order. | +| `ReadH5OinaDataFilter-D10` | Reader error messages reach the user. | +| `ReadH5OinaDataFilter-D11` | Both cell counts are validated and the returned error code is consistent. | +| `ReadH5OinaDataFilter-D12` | Invalid phase-group names produce an error instead of an exception. | +| `ReadH5OinaDataFilter-D13` | Geometry spacing must be finite and positive. | diff --git a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md index 32bc44e711..261898ce13 100644 --- a/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/deviations/ReadH5OinaDataFilter.md @@ -4,8 +4,7 @@ This file therefore records, in the same structured form, every difference between the behavior DREAM3D-NX shipped and the correct behavior. Entries are referenced by stable ID (`ReadH5OinaDataFilter-D`) from the V&V report and from public migration guidance. The ID is stable across renames; the Filter UUID field is the permanent cross-reference anchor. -Affected releases are named from `docs/dream3d_nx_release_dates.md`. The filter first shipped in DREAM3D-NX 7.0.0 (2024-11-11); D1 through D11 are present in every release from 7.0.0 through 7.4.1 (2026-03-23). -EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINAReader` and every release of DREAM3D-NX up through and including 7.4.1 will have these bugs. +The filter first shipped in DREAM3D-NX 7.0.0. D1 through D13 affect every released version from 7.0.0 through 7.4.1. D5, D6, D7, D10, D11, and D12 are corrections to EbsdLib's `H5OINAReader` and require EbsdLib 3.1.2. --- @@ -100,7 +99,7 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Deviation ID** | `ReadH5OinaDataFilter-D5` | | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | -| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | **Symptom:** The `LatticeConstants` ensemble array reported each phase's γ angle as a copy of its β angle. The file's third lattice angle was never read. For a cubic phase, where α = β = γ, the error is invisible; for a hexagonal phase, whose angles are 90/90/120, the reported γ was 90 instead of 120, and the same applies to any monoclinic, triclinic, trigonal or rhombohedral cell whose γ differs from its β. @@ -108,9 +107,9 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA **Affected users:** Anyone importing an `.h5oina` file whose phases are not cubic or tetragonal, and who reads `LatticeConstants` downstream. It does not affect orientations, phase indices or crystal-structure symmetry, which come from the Laue group and not from the lattice angles. -**Correct behavior:** The gamma slot receives the third angle. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Store the third H5OINA lattice angle in the gamma slot`). Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, whose hexagonal fixture phase has γ ≠ β specifically so that a gamma slot echoing beta is visible. +**Correct behavior:** The gamma slot receives the third angle. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"`, whose hexagonal fixture phase has γ ≠ β specifically so that a gamma slot echoing beta is visible. -**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still reports the duplicated angle. +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. --- @@ -121,7 +120,7 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Deviation ID** | `ReadH5OinaDataFilter-D6` | | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | -| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | | **Breaking change** | **Yes.** The value of a published output array changes for every H5OINA import. | **Symptom:** The three angle slots of `LatticeConstants` were reported in radians for H5OINA imports and in degrees for every other EBSD importer. A cubic phase imported from an `.h5oina` file reported `1.5707964, 1.5707964, 1.5707964`, while the same phase imported from a `.ctf` or `.ang` file reported `90, 90, 90`. The array's meaning therefore depended on which file format the phase happened to come from, with nothing in the data to say which. @@ -130,9 +129,9 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA **Affected users:** Anyone comparing or combining phase information across file formats, and anyone reading `LatticeConstants` from an H5OINA import while assuming the degrees convention that the rest of DREAM3D-NX uses. -**Correct behavior:** `H5OINAReader` converts the angles from radians to degrees on import, on a double-precision intermediate, so the array carries the same unit no matter which format the phase came from. Corrected in EbsdLib on `topic/3_1_1_staging` (`ENH: Convert H5OINA lattice angles to degrees on import`). Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"` (90, 90, 120 for the hexagonal fixture phase) and `…::"Real AZtec File Readback"` (90, 90, 90 for the production file's titanium-cubic phase). +**Correct behavior:** `H5OINAReader` converts the angles from radians to degrees on import, on a double-precision intermediate, so the array carries the same unit no matter which format the phase came from. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"Class 1 Analytical Oracle"` (90, 90, 120 for the hexagonal fixture phase) and `…::"Real AZtec File Readback"` (90, 90, 90 for the production file's titanium-cubic phase). -**Release note and migration:** This is a **breaking change to a published output**, and the first release that carries it is the first DREAM3D-NX release built against EbsdLib 3.1.1; no released version carries it today. Every `.dream3d` file written by 7.0.0 through 7.4.1 has radians in components 3, 4 and 5 of `LatticeConstants`, so any saved exemplar, regression baseline or pipeline comparison that reads those components changes value on upgrade. To compare a stored radian value against a new import, multiply it by 180/π. Consumers that compensated by converting H5OINA lattice angles themselves must stop doing so. Nothing else in the import changes unit: `Euler` is still radians and the three lattice dimensions are still unconverted. The user-facing migration note is in `docs/ReadH5OinaDataFilter.md` under "Migration Notes". The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). +**Release note and migration:** This is a **breaking change to a published output**. The first release that carries it is DREAM3D-NX 7.5.0 with EbsdLib 3.1.2. Every `.dream3d` file written by 7.0.0 through 7.4.1 has radians in components 3, 4 and 5 of `LatticeConstants`, so any saved exemplar, regression baseline or pipeline comparison that reads those components changes value on upgrade. To compare a stored radian value against a new import, multiply it by 180/π. Consumers that compensated by converting H5OINA lattice angles themselves must stop doing so. Nothing else in the import changes unit: `Euler` is still radians and the three lattice dimensions are still unconverted. The user-facing migration note is in `docs/ReadH5OinaDataFilter.md` under "Migration Notes". The archived `H5Oina_Test_Data.dream3d` exemplar has the pre-correction radian values baked in, which is one of the reasons it is no longer used as a comparison target (see `vv/provenance/ReadH5OinaDataFilter.md`). **Recommendation:** Trust the corrected behavior — the degrees convention is the one the rest of the toolkit uses and the one the other importers already produced. @@ -145,7 +144,7 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Deviation ID** | `ReadH5OinaDataFilter-D7` | | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | -| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | **Symptom:** An `.h5oina` file whose phase group was missing its `Lattice Dimensions` or `Lattice Angles` dataset **crashed the process**. This is empirically demonstrated, not inferred: running the fixture against a build with the corrected filter sources but the pre-correction `H5OINAReader` reports the regression test as `OrientationAnalysis::ReadH5OinaDataFilter: EbsdLib Error Passthrough - Missing Lattice Angles (-9582) (SEGFAULT)` rather than as a failure. The suite log recording that run is listed in `vv/provenance/ReadH5OinaDataFilter.md`. @@ -153,9 +152,9 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA **Affected users:** Anyone opening a truncated, partially written or otherwise malformed `.h5oina` file. The crash occurs during preflight, so it takes down the application as the file is selected, before any pipeline runs. -**Correct behavior:** The four required phase reads are checked and reported with EbsdLib error codes `-90030` through `-90033`, each naming the phase and the dataset; `Space Group` remains optional because the reader only passes it through. The filter surfaces the failure as `-9582` with the file path and the scan name. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Check the phase dataset reads in H5OINAReader::readHeader()`). Pinned by `test/ReadH5OinaDataTest.cpp::"EbsdLib Error Passthrough - Missing Lattice Angles (-9582)"`, which is the fixture that used to crash. +**Correct behavior:** The four required phase reads are checked and reported with EbsdLib error codes `-90030` through `-90033`, each naming the phase and the dataset; `Space Group` remains optional because the reader only passes it through. The filter surfaces the failure as `-9582` with the file path and the scan name. Corrected in EbsdLib 3.1.2. Pinned by `test/ReadH5OinaDataTest.cpp::"EbsdLib Error Passthrough - Missing Lattice Angles (-9582)"`, which is the fixture that used to crash. -**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still crashes on such a file. +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. --- @@ -168,15 +167,15 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Affected releases** | 7.0.0 through 7.4.1 | | **Status** | resolved | -**Symptom:** Selecting two or more scans from a file whose scans do not all declare the same phase groups **crashed the process** at execute, with no preflight error and no warning. No malformed input is involved: an AZtec file may legitimately carry scans with different phase lists, and a scan that declares more phases than the *first* selected scan is enough. +**Symptom:** A later scan with a phase index above the first scan's phase count wrote past the ensemble arrays and could crash the process. Scans with the same indices but different phase definitions silently used one shared definition for all scan slices. **Root cause:** Bug. The single Ensemble Attribute Matrix that all of the stacked scans share is sized from the first selected scan's phase count, `phases.size() + 1`. The shared ensemble fill in `utilities/IEbsdOemReader.hpp` then runs once per selected scan and writes `crystalStructures[phaseId]`, `materialNames[phaseId]` and `latticeConstants` component `phaseId`, where `phaseId` is the integer in that scan's HDF5 phase group name. Any later scan whose phase index exceeds the first scan's phase count writes past the end of all three arrays. Preflight range-checked those indices for the first selected scan only, and the execute-side `-34972` check inspects the phase *column values*, not the phase *group names*, and runs after the ensemble fill has already happened. **Affected users:** Anyone selecting more than one scan from an `.h5oina` file whose scans differ in their phase lists. Single-scan imports and multi-scan imports of files with one uniform phase list — which is every file in the shipped test data — were never affected, which is why the pre-existing suite could not see it. -**Correct behavior:** Preflight applies the phase-index range check to **every** selected scan, with the bound taken from the first scan's phase count (`-9587`), and rejects a selected scan whose phase-group count differs from the first scan's (`-9589`), telling the user to import scans with differing phase lists separately. Pinned by `test/ReadH5OinaDataTest.cpp::"Phase Index Out Of Range rejected (-9587)"` section "Later selected scan" and `…::"Scan Phase Count Mismatch rejected (-9589)"`, whose fixtures both pass preflight and crash the process against the pre-correction filter. +**Correct behavior:** Preflight validates the phase indices (`-9587`), phase count (`-9589`), and phase definitions (`-9590`) of every selected scan. The definitions must match because the scans become slices of one 3D microstructure and share one Ensemble Attribute Matrix. -**Recommendation:** Trust the corrected behavior. Every reproduction of the defect crashed before any output was written, so a multi-scan import that *completed* under 7.0.0 through 7.4.1 is very unlikely to have been affected — but the underlying write is undefined behavior, so silent corruption cannot be strictly excluded. Residual restriction: the guards require every selected scan to declare the same phase-group *names and count*; the group *contents* (material name, Laue class, lattice constants) are not compared across scans — the shared ensemble arrays keep the values of the last scan read, and the preflight phase display always reflects the first-listed scan. Scans whose phase definitions differ in content should be imported separately. +**Recommendation:** Trust the corrected behavior. Re-import any multi-scan file whose phase definitions differ between scans. Import those scans separately if they describe different microstructures. --- @@ -208,7 +207,7 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Deviation ID** | `ReadH5OinaDataFilter-D10` | | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | -| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | **Symptom:** Every failure reported out of `H5OINAReader` reached the user with a blank `Message:` field. A malformed `.h5oina` produced an error code and an empty explanation, so nothing in the message said which dataset, phase or scan was at fault. @@ -216,9 +215,9 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA **Affected users:** Anyone who hit any `H5OINAReader` error — a malformed file, a missing dataset, an absent scan. The filter's own `-8970` and `-9582` messages name the file and the scan, so the file was identifiable; the reason was not. -**Correct behavior:** Each of the ten sites composes into a stream of its own and reports that stream's contents, and `readFile()`'s wrappers name the scan and append the inner message rather than replacing it. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Report the error messages H5OINAReader builds`). +**Correct behavior:** Each of the ten sites composes into a stream of its own and reports that stream's contents, and `readFile()`'s wrappers name the scan and append the inner message rather than replacing it. Corrected in EbsdLib 3.1.2. -**Recommendation:** Trust the corrected behavior. Note that the correction ships only with EbsdLib 3.1.1; a DREAM3D-NX build against EbsdLib 3.1.0 or earlier still reports the blank message. No test asserts the text `H5OINAReader` composes — the filter-side assertions match the scan name and file path that the DREAM3D-NX format strings inject, which are independent of the reader's message. +**Recommendation:** Trust the corrected behavior. The correction requires EbsdLib 3.1.2. --- @@ -229,7 +228,7 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA | **Deviation ID** | `ReadH5OinaDataFilter-D11` | | **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | | **Affected releases** | 7.0.0 through 7.4.1 | -| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | **Symptom:** Two defects in one code path. A scan whose header declared zero rows was rejected with error code `-90301` recorded on the reader but `-301` handed back as the return value, so a caller reporting the return value and a caller reading `getErrorCode()` disagreed about what had happened. A scan whose header declared a **negative** column count was not rejected at all: the count was widened to `size_t`, producing an enormous allocation request. @@ -237,12 +236,54 @@ EbsdLib Deviations: D5, D6, D7, D10 and D11 are corrections to EbsdLib's `H5OINA **Affected users:** Anyone opening a truncated or otherwise malformed `.h5oina` file. DREAM3D-NX reports the code it receives, so the mismatched value was the one the user saw. -**Correct behavior:** Both the row and the column count are validated and the rejection returns the code it sets. Corrected in EbsdLib on `topic/3_1_1_staging` (`BUG: Validate H5OINAReader::readData()'s row and column counts`). The filter rejects the same shape earlier and independently: `-9584` rejects `X Cells` or `Y Cells` below 1 at preflight, which is pinned by `test/ReadH5OinaDataTest.cpp::"Invalid Cell Counts rejected (-9584)"`, including the negative case. +**Correct behavior:** Both the row and the column count are validated and the rejection returns the code it sets. Corrected in EbsdLib 3.1.2. The filter rejects the same shape earlier and independently: `-9584` rejects `X Cells` or `Y Cells` below 1 at preflight, which is pinned by `test/ReadH5OinaDataTest.cpp::"Invalid Cell Counts rejected (-9584)"`, including the negative case. **Recommendation:** Trust the corrected behavior. The filter's `-9584` guard fires first for a file selected through DREAM3D-NX, so this entry matters to other `H5OINAReader` callers. --- +## ReadH5OinaDataFilter-D12 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D12` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved in EbsdLib — requires EbsdLib 3.1.2 | + +**Symptom:** A non-numeric or noncanonical phase-group name caused `std::stoi()` to throw during preflight. The exception stopped the import and left the opened HDF5 phase group unclosed. + +**Root cause:** Bug. `H5OINAReader::readHeader()` passed each HDF5 group name directly to `std::stoi()` after it opened the group. The code did not validate that the complete name was a positive integer. + +**Affected users:** Users with a malformed or manually edited H5OINA file whose phase group is not named with a canonical positive integer. + +**Correct behavior:** EbsdLib parses the complete group name before it opens the group. It returns `-90034` for a noncanonical name. The filter reports the error through `-9582` and identifies the file, scan, and group name. + +**Recommendation:** Trust the corrected behavior. Correct the phase-group names or export the file again from AZtec. + +--- + +## ReadH5OinaDataFilter-D13 + +| Field | Value | +|---|---| +| **Deviation ID** | `ReadH5OinaDataFilter-D13` | +| **Filter UUID** | `fad3d47f-f1e1-4429-bc65-5e021be62ba0` | +| **Affected releases** | 7.0.0 through 7.4.1 | +| **Status** | resolved | + +**Symptom:** The filter accepted non-positive or non-finite X, Y, or Z spacing and created an invalid Image Geometry. + +**Root cause:** Bug. Preflight checked only whether Z spacing was less than or equal to zero. It did not reject NaN, and it did not validate the X and Y step values from the file. + +**Affected users:** Users with malformed H5OINA step values or a non-finite Z Spacing parameter. + +**Correct behavior:** Preflight requires finite, positive X, Y, and Z spacing. It reports `-9580` for Z spacing and `-9591` for scan spacing, with the actual values and file context. + +**Recommendation:** Trust the corrected behavior. Correct the spacing values or export the file again from AZtec. + +--- + ## Malformed-input rejections These are not behavioral deviations on well-formed files — every one of them is a rejection path that stands where 7.0.0 through 7.4.1 performed an out-of-range read, an out-of-range write, or produced a silently useless output. They are listed here so a reader auditing the error-code series has one place to find them. The multi-scan phase-group cases are the exception and are recorded as D8, because they are reachable from a well-formed file. @@ -254,6 +295,8 @@ These are not behavioral deviations on well-formed files — every one of them i | `-9586` | A selected scan name that is not in the file | Only the first name was checked, at preflight; a bad later name failed part-way through execute with the earlier scans already written into the output arrays and no rollback | | `-9587` | A phase group of **any** selected scan named outside 1 through N, where N is the first selected scan's phase count | The check covered the first selected scan only, so a group named `7` in a later scan wrote past the end of the ensemble arrays — see D8 | | `-9589` | A selected scan whose phase-group count differs from the first selected scan's | The ensemble arrays were sized from the first scan alone and filled from every scan — see D8 | +| `-9590` | A selected scan whose phase definitions differ from the first selected scan's | One shared Ensemble Attribute Matrix silently kept one scan's phase definitions for every scan slice — see D8 | +| `-9591` | Non-positive or non-finite X or Y spacing | The invalid values were assigned to the Image Geometry — see D13 | | `-34971` | A `Data` dataset whose extent disagrees with the header's cell counts, in either direction | The reader sizes its buffers to the actual extent while the copy spans the header's point count, reading past the end of those buffers when the dataset is short and silently dropping the surplus rows when it is long | | `-34972` | A phase value outside `[0, phase count]` | The value indexed `CrystalStructures` unchecked in the alignment loop, reading past the end of the ensemble array | diff --git a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md index cfb5613b43..fabbb108d3 100644 --- a/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md +++ b/src/Plugins/OrientationAnalysis/vv/provenance/ReadH5OinaDataFilter.md @@ -65,7 +65,7 @@ It also had the pre-correction behavior baked in: its `LatticeConstants` reads D6. A test comparing against it did not merely fail to detect D6 — it actively enforced it. Recorded measurement: `exemplar_selforacle_check.txt`. -Its discriminating power against the eleven deviations is near zero: the file has +Its discriminating power against the thirteen deviations is limited: the file has a single cubic phase, so the hexagonal alignment (D1, D3) never executes; it holds one scan, so the slab offsets (D2), the stacking order (D9) and the multi-scan ensemble write (D8) are never reached; pattern import was off, so D4 and its latent type mismatch were never @@ -77,14 +77,10 @@ defaults against a real-world file — is preserved and strengthened by its repl **Toy fixtures, written by the test at run time.** `test/ReadH5OinaDataTest.cpp` declares a fixture specification as C++ structs and writes `.h5oina` files with `H5Support::H5Lite` -into the binary test-output directory. Three fixture specifications materialise into -twenty-three files at run time: eight that are imported successfully and fifteen that back a -rejection or passthrough case. Nothing is committed as binary test data and no archive -upload was needed. The eight that import carry exactly the dataset set `H5OINAReader` -requires, plus the inert root datasets for realism. Of the fifteen behind the rejection and -passthrough cases, three omit a required dataset outright — `Data/Bands` once and -`Phases//Lattice Angles` twice — while the other twelve carry the full set and are -rejected on a parameter value, on a header or phase value, or on how the scans are selected. +into the binary test-output directory. The test cases create valid inputs and focused guard +inputs at run time. Nothing is committed as binary test data and no archive upload was +needed. The valid files carry the complete dataset set that `H5OINAReader` requires. The +guard files omit a required dataset or contain a value that the filter must reject. Every expected value is derived from the fixture specification. The hexagonal-alignment expectations are the correctly-rounded float32 results of `double(φ2) + 30 × (π/180)`, @@ -114,11 +110,11 @@ the ensemble slot-0 defaults, the radians-to-degrees lattice-angle conversion an hexagonal φ2 alignment — and applies them to the fixture specification or to a file's bytes. It never runs the filter. -## Trusted-boundary versions +## Oracle and dependency versions | Boundary | Version | |---|---| -| **EbsdLib** | `topic/3_1_1_staging`, the branch that becomes EbsdLib 3.1.1. This is the primary Class 2 trusted boundary: it owns HDF5 traversal, header and phase parsing, the required-dataset reads and the error codes. Five of this filter's eleven deviations are corrections inside it. | +| **EbsdLib** | 3.1.2. EbsdLib is part of the system under test. Six of the thirteen deviations are corrections in `H5OINAReader`. | | h5py / NumPy | 3.16.0 / 2.5.2, under `/opt/local/anaconda3/envs/dream3d/bin/python` | | HDF5 | as vendored by the `NX-Com-Qt69-Vtk96-Rel-EbsdLib` preset's vcpkg manifest | @@ -130,7 +126,7 @@ It never runs the filter. | `probe_real_file.txt` | The h5py structure dump of the production `.h5oina` | | `error_column_check.txt` | The `Error` column census of the production file — 587 points at 1, 38 at 2, none at 0 — behind the documentation's masking correction | | `exemplar_selforacle_check.txt` | The measurement behind the exemplar's retirement: `Euler` bit-identity and the radian `LatticeConstants` | -| `mutation_table.md` | Seven mutations, each killing exactly its claimed test cases, each reverted to an empty diff | +| `mutation_table.md` | Test-sensitivity evidence for the original seven temporary defects. Internal review added four detected defects for phase-group parsing, phase-definition validation, spacing validation, and error context. | ## Suite logs @@ -178,7 +174,7 @@ comparison establishes is that rolling EbsdLib back changes the non-H5OINA failu at all. The three extra failures at `539ddfc` are the three ReadH5Oina tests that the EbsdLib corrections are required for -- `Class 1 Analytical Oracle`, `Real AZtec File Readback` and `EbsdLib Error Passthrough - Missing Lattice Angles (-9582)`, the last as a -SEGFAULT -- which is the evidence the `vcpkg.json` pin to EbsdLib 3.1.1 rests on. The +SEGFAULT -- which is evidence for the EbsdLib dependency. The structural argument agrees: `git diff --stat 539ddfc..HEAD` on EbsdLib is one file, `Source/EbsdLib/IO/HKL/H5OINAReader.cpp`, with no header and no API change, so nothing outside the H5OINA read path can be reached. @@ -192,6 +188,11 @@ pipeline test of exactly the shape described above. sentinel extraction, not on any assertion; a failure of that shape clears on an immediate re-run with no source change. +The final internal-review run uses EbsdLib 3.1.2. The results are 20 of 20 H5OINA tests, +408 of 408 EbsdLib tests, and 996 of 996 SimplnxCore tests. The OrientationAnalysis suite +passes 281 of 282 tests. The only failure is the unrelated ComputeSchmids archive +comparison against values from before its precision correction. + ## Oracle-before-comparison ordering There was no comparison to order against: DREAM3D 6.5.171 has no H5OINA importer. The oracle