Repository navigation
[ESSreduce] Add TransformationOverrides for NeXus transformation chains - #858
Merged
Merged
Conversation
jl-wynen
reviewed
Nov 9, 2026
| """Values replacing entries of the transformation chains of a component type. | ||
|
|
||
| Maps the absolute NeXus path of a chain entry to the value to use for it, e.g., | ||
| ``{'/entry/instrument/detector_carriage/value': sc.scalar(1.5, unit='m')}``. |
Member
There was a problem hiding this comment.
Looks good overall. But I am a little skeptical about using the absolute path here. We often look up components via NeXus classes and group names, not absolute paths. Did you consider alternatives here that don't rely on the absolute path?
Does livedata compute the paths by searching for Kafka topics in the file?
Member
Author
There was a problem hiding this comment.
Livedata looks for topic and source attributes that indicate that a value comes from a stream. It thus naturally has an absolute path, and uses those as dict keys:
jl-wynen
approved these changes
Nov 9, 2026
Add `TransformationOverrides[Component, RunType]`, which maps the absolute NeXus path of a chain entry to a value. Both `get_transformation_chain` and `get_multi_transformation_chain` apply it, so a motor value set for a component type reaches single and multi chains of that type. The workflows default to no overrides. The key includes the component type so that a job depends only on the motors of the components it uses. The cost is that a motor shared by several types must be set once per type. A path that is not in the chain raises KeyError. For multi chains this applies only if no chain of the type contains the path, since a motor may carry only some of the components. `extract_instrument_angle` now wraps a scalar value in a DataArray, as declared by `InstrumentAngle`. Otherwise `group_by_rotation` fails on a scalar override or a fixed angle in a file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State which providers overrides do not reach, and that pipelines mapping over banks must map the overrides if a motor does not move every bank. The BIFROST test now compares with the expected rotated positions, so that a wrong axis or a doubly applied rotation fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jl-wynen
force-pushed
the
transformation-overrides
branch
from
November 9, 2026 13:36
830060a to
f1708b3
Compare
jl-wynen
enabled auto-merge
November 9, 2026 13:36
SimonHeybrock
added a commit
to scipp/esslivedata
that referenced
this pull request
Nov 10, 2026
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which the job-level motor gating imports. essspectroscopy 26.11.1 includes the scipp/ess#802 changes this branch adapts to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
SimonHeybrock
added a commit
to scipp/esslivedata
that referenced
this pull request
Nov 12, 2026
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which the job-level motor gating imports. essspectroscopy 26.11.1 includes the scipp/ess#802 changes this branch adapts to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the review of #802, which left the chain overrides out of that PR.
What changes
TransformationOverrides[Component, RunType]maps the absolute NeXus path of a chain entry to a value.get_transformation_chainandget_multi_transformation_chainboth apply it, on a copy of the chain. All three generic workflows default to no overrides.esslivedata needs this to feed motor readbacks into chains. It currently replaces
get_transformation_chainwith its own provider, which does not reach multi chains. The esslivedata side is scipp/esslivedata#1444.Choices
NXdetector,NXcrystalandElasticMonitor.KeyError. Otherwise a typo leaves the placeholder in place and fails later with a less helpful error. Multi chains raise only if no chain of the type contains the path. Single chains always raise, so a pipeline mapping over banks has to map the overrides too if a motor does not move every bank. This follows the [ESSspectroscopy] Bifrost: New workflow without map-reduce #802 review; see theTransformationOverridesdocstring.extract_instrument_anglewraps a scalar in aDataArray, asInstrumentAngledeclares. Otherwisegroup_by_rotationfails on a scalar override or a fixed angle in a file.Limits
Overrides reach only providers that use the chain types. These read the stored value directly and ignore overrides:
parse_raw_choppers)load_detector_rotation)load_sample_angle, see [ESSSPECTROSCOPY] BIFROST: derive SampleAngle (a3) from the sample transformation chain #829)Relation to #828: overrides also reach mapped branches when set after
.map, since the overrides node does not depend on the mapped name. This addresses the point that live data cannot replace a shared link after the map. The chain is still loaded once per branch, so #828 stays open.Follow-up
essspectroscopydeclaresessreduce>=26.8.0, butess/spectroscopy/types.pyneeds the next essreduce release at import. That has been the case since #802, becauseessreduce/26.10.1has noNeXusMulti*types. The bound can be raised once that release exists.Test plan
tests/nexus: 274 passedtest_simulation_workflow_produces_the_same_data_as_beforecould not download its file becausepublic.esss.dkwas unreachable, so this regression test has not been run against the change.packages/essreduce/tests/streaming_test.py::test_StreamProcessor_overall_behaviordepends on test order and also fails onmainunderpytest-randomly. It is unrelated to this PR.🤖 Generated with Claude Code