Skip to content

[ESSreduce] Add TransformationOverrides for NeXus transformation chains - #858

Merged
jl-wynen merged 2 commits into
mainfrom
transformation-overrides
Nov 9, 2026
Merged

jl-wynen merged 2 commits into
mainfrom
transformation-overrides

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Follow-up to the review of #802, which left the chain overrides out of that PR.

What changes

TransformationOverrides[Component, RunType] maps the absolute NeXus path of a chain entry to a value. get_transformation_chain and get_multi_transformation_chain both apply it, on a copy of the chain. All three generic workflows default to no overrides.

wf[TransformationOverrides[snx.NXdetector, SampleRun]] = {
    '/entry/instrument/detector_carriage/value': sc.scalar(5.0, unit='m')
}

esslivedata needs this to feed motor readbacks into chains. It currently replaces get_transformation_chain with its own provider, which does not reach multi chains. The esslivedata side is scipp/esslivedata#1444.

Choices

  • Per component type, not per run. With one dict per run, every motor becomes an ancestor of every chain, so a live job would wait for motors it never reads. The cost is that a motor shared by several types has to be set once per type. On BIFROST the tank angle needs NXdetector, NXcrystal and ElasticMonitor.
  • Unknown paths raise KeyError. Otherwise a typo leaves the placeholder in place and fails later with a less helpful error. Multi chains raise only if no chain of the type contains the path. Single chains always raise, so a pipeline mapping over banks has to map the overrides too if a motor does not move every bank. This follows the [ESSspectroscopy] Bifrost: New workflow without map-reduce #802 review; see the TransformationOverrides docstring.
  • extract_instrument_angle wraps a scalar in a DataArray, as InstrumentAngle declares. Otherwise group_by_rotation fails on a scalar override or a fixed angle in a file.

Limits

Overrides reach only providers that use the chain types. These read the stored value directly and ignore overrides:

Relation to #828: overrides also reach mapped branches when set after .map, since the overrides node does not depend on the mapped name. This addresses the point that live data cannot replace a shared link after the map. The chain is still loaded once per branch, so #828 stays open.

Follow-up

essspectroscopy declares essreduce>=26.8.0, but ess/spectroscopy/types.py needs the next essreduce release at import. That has been the case since #802, because essreduce/26.10.1 has no NeXusMulti* types. The bound can be raised once that release exists.

Test plan

  • essreduce tests/nexus: 274 passed
  • essspectroscopy tests: 27 passed, 1 not run. test_simulation_workflow_produces_the_same_data_as_before could not download its file because public.esss.dk was unreachable, so this regression test has not been run against the change.
  • New BIFROST test on the CODA file: a tank angle override rotates detectors and analyzers by the expected angle and sets a4
  • essreflectometry, essimaging, esssans, essnmx: the same tests fail with and without this change. These failures come from the local environment, such as missing downloads and optional packages. essdiffraction passes.

packages/essreduce/tests/streaming_test.py::test_StreamProcessor_overall_behavior depends on test order and also fails on main under pytest-randomly. It is unrelated to this PR.

🤖 Generated with Claude Code

"""Values replacing entries of the transformation chains of a component type.

Maps the absolute NeXus path of a chain entry to the value to use for it, e.g.,
``{'/entry/instrument/detector_carriage/value': sc.scalar(1.5, unit='m')}``.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. But I am a little skeptical about using the absolute path here. We often look up components via NeXus classes and group names, not absolute paths. Did you consider alternatives here that don't rely on the absolute path?

Does livedata compute the paths by searching for Kafka topics in the file?

@SimonHeybrock SimonHeybrock Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Livedata looks for topic and source attributes that indicate that a value comes from a stream. It thus naturally has an absolute path, and uses those as dict keys:

https://github.com/scipp/esslivedata/blob/18d6040421f592a1fad45b3a57bae113da677779/src/ess/livedata/config/instruments/bifrost/streams_parsed.py#L13-L19

SimonHeybrock and others added 2 commits November 9, 2026 14:35
Add `TransformationOverrides[Component, RunType]`, which maps the absolute
NeXus path of a chain entry to a value. Both `get_transformation_chain` and
`get_multi_transformation_chain` apply it, so a motor value set for a
component type reaches single and multi chains of that type. The workflows
default to no overrides.

The key includes the component type so that a job depends only on the motors
of the components it uses. The cost is that a motor shared by several types
must be set once per type.

A path that is not in the chain raises KeyError. For multi chains this applies
only if no chain of the type contains the path, since a motor may carry only
some of the components.

`extract_instrument_angle` now wraps a scalar value in a DataArray, as declared
by `InstrumentAngle`. Otherwise `group_by_rotation` fails on a scalar override
or a fixed angle in a file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State which providers overrides do not reach, and that pipelines mapping over
banks must map the overrides if a motor does not move every bank.

The BIFROST test now compares with the expected rotated positions, so that a
wrong axis or a doubly applied rotation fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jl-wynen
jl-wynen force-pushed the transformation-overrides branch from 830060a to f1708b3 Compare November 9, 2026 13:36
@jl-wynen
jl-wynen enabled auto-merge November 9, 2026 13:36
@jl-wynen
jl-wynen added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 5d57f90 Nov 9, 2026
24 checks passed
@jl-wynen
jl-wynen deleted the transformation-overrides branch November 9, 2026 14:01
SimonHeybrock added a commit to scipp/esslivedata that referenced this pull request Nov 10, 2026
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which
the job-level motor gating imports. essspectroscopy 26.11.1 includes the
scipp/ess#802 changes this branch adapts to.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
SimonHeybrock added a commit to scipp/esslivedata that referenced this pull request Nov 12, 2026
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which
the job-level motor gating imports. essspectroscopy 26.11.1 includes the
scipp/ess#802 changes this branch adapts to.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

essreduce Issues for essreduce. essspectroscopy Issues for essspectroscopy.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants