Repository navigation
[ESSspectroscopy] Bifrost: New workflow without map-reduce - #802
Conversation
| " # TODO We need to associate data with banks so that assemble_detector_data\n", | ||
| " # can put it all together. But the events loaded here lack the required\n", |
There was a problem hiding this comment.
Can you clarify the problem? event_id is unique across banks, so in Livedata I just concat and then binning into the concatenated empty detector just works.
b69cd24 to
135a0fb
Compare
|
@SimonHeybrock This should be working now. We can add transform chain patching in |
| triplets form a regular rectangular subset of the full 5x9 detector array, | ||
| they will be folded into those dimensions. Otherwise, they are concatenated | ||
| along a 'triplet' dimension. | ||
| def assemble_bifrost_detector_data( |
There was a problem hiding this comment.
I benchmarked this on my laptop and assemble_bifrost_detector_data takes by far the longest (~900ms vs ~600ms for the next slowest provider). Is there a better way of merging the events?
There was a problem hiding this comment.
Yes, I think we can bin once instead of once per bank. event_id is unique across banks, so we can concatenate first, as concat_event_lists did before this PR:
def concat_detector_events(
events: NeXusManyData[snx.NXdetector, RunType],
) -> NeXusData[snx.NXdetector, RunType]:
return NeXusData[snx.NXdetector, RunType](
sc.concat(list(events.values()), dim='event_time_zero')
)The generic assemble_detector_data then does the rest, so assemble_bifrost_detector_data can go and io/mcstas.py can take NeXusData again. I have not benchmarked this.
It also matters for Livedata. We merge all banks into one event stream before binning and pass the result as the dynamic key NeXusData[NXdetector, SampleRun]. In this PR that key is no longer an ancestor of RawDetector, and StreamProcessor does not complain about a dynamic key that is not in the graph. This provider would keep our key working.
There was a problem hiding this comment.
Thanks! This works and shaves off 1s and a bit from the reduction workflow.
135a0fb to
36cc505
Compare
36cc505 to
aa870fb
Compare
| triplets form a regular rectangular subset of the full 5x9 detector array, | ||
| they will be folded into those dimensions. Otherwise, they are concatenated | ||
| along a 'triplet' dimension. | ||
| def assemble_bifrost_detector_data( |
There was a problem hiding this comment.
Yes, I think we can bin once instead of once per bank. event_id is unique across banks, so we can concatenate first, as concat_event_lists did before this PR:
def concat_detector_events(
events: NeXusManyData[snx.NXdetector, RunType],
) -> NeXusData[snx.NXdetector, RunType]:
return NeXusData[snx.NXdetector, RunType](
sc.concat(list(events.values()), dim='event_time_zero')
)The generic assemble_detector_data then does the rest, so assemble_bifrost_detector_data can go and io/mcstas.py can take NeXusData again. I have not benchmarked this.
It also matters for Livedata. We merge all banks into one event stream before binning and pass the result as the dynamic key NeXusData[NXdetector, SampleRun]. In this PR that key is no longer an ancestor of RawDetector, and StreamProcessor does not complain about a dynamic key that is not in the graph. This provider would keep our key working.
| ) | ||
|
|
||
|
|
||
| def get_transformation_chain( |
There was a problem hiding this comment.
Re your question: yes, chain patching would work for Livedata. I would put it in the generic chain provider so that all NeXus workflows get it:
TransformationOverrides[RunType] # absolute NeXus path of a chain entry -> value
def get_transformation_chain(
component: NeXusComponent[Component, RunType],
overrides: TransformationOverrides[RunType],
) -> NeXusTransformationChain[Component, RunType]:
chain = deepcopy(component['depends_on']) # the component is a cached node
for path, value in overrides.items():
if path in chain.transformations:
chain.transformations[path].value = value
return NeXusTransformationChain[Component, RunType](chain)The overrides are per run because the same path appears in several component types. In the CODA Bifrost file from 2026-11-03.nxs, the triplet, analyzer and elastic_monitor chains all pass through /entry/instrument/detector_tank_angle/value. Each component loads its own copy of the chain, so a patch only affects that copy. But this provider runs per component with the same dict, so one override reaches every chain that goes through it. The elastic monitor chain also feeds extract_instrument_angle, so geometry and a4 stay consistent, provided that no provider reads depends_on directly (see the comment in detector.py).
The default {} is needed in each of the three workflows that list TransformationTimeFilter.
There was a problem hiding this comment.
This doesn't quite work because we would need providers and types with 'multi' chains. But I can create those.
There was a problem hiding this comment.
Good point, I missed that. Yes please, if you can add those. get_calibrated_detector_bifrost and get_calibrated_analyzers can then take the chains as input, as you said. See my reply in the detector.py thread: the overrides should be keyed by component as well, so please use that for the multi providers too.
| for analyzer in analyzers.values(): | ||
| detector = detector_components[analyzer['detector_name']] | ||
| transform = to_transformation( | ||
| chain=detector['depends_on'], |
There was a problem hiding this comment.
This reads depends_on directly, so overrides in get_transformation_chain would not reach it. The same goes for get_calibrated_analyzers. How about a generic provider instead of patching in both places?
NeXusManyComponents[Component, RunType], TransformationOverrides[RunType]
-> NeXusManyTransformationChains[Component, RunType]These two providers would then take the chains as input and would not need to know about overrides.
There was a problem hiding this comment.
How would TransformationOverrides[RunType] work when there is only one chain vs multi? Do we need to override separately for each chain or use the same override for all?
I guess for bifrost, using the same for all works because the tank rotation is shared.
There was a problem hiding this comment.
Same override for all chains of a component type, so one dict per (Component, RunType). That means changing my proposal, though: the key should be TransformationOverrides[Component, RunType], not [RunType]. A Livedata job waits for every motor its graph depends on before it starts. With a single per-run dict, every motor becomes an ancestor of every chain. I tried both on the LOKI configs: a front-bank detector view would wait for up to 10 motors it never reads, instead of 0, and a sample motor update would also recompute the detector positions. The per-component key gave the same set of motors as our current wiring in all six jobs I tried.
def get_transformation_chain(
component: NeXusComponent[Component, RunType],
overrides: TransformationOverrides[Component, RunType],
) -> NeXusTransformationChain[Component, RunType]: ...
def get_multi_transformation_chains(
components: NeXusMultiComponents[Component, RunType],
overrides: TransformationOverrides[Component, RunType],
) -> NeXusMultiTransformationChains[Component, RunType]: ...The cost is that a shared motor has to be set once per type. For the Bifrost tank angle that's NXdetector, NXcrystal and ElasticMonitor. Livedata already declares which components each motor moves, so that's fine for us.
The default can be TransformationOverrides: {} in the workflow params, next to TransformationTimeFilter. I checked that with Sciline: the bare key fills in every specialization, and a specific param or provider overrides it.
Since the dict is now per type, could a path that isn't in the chain raise instead of being skipped? For multi chains, raise only if no chain of that type contains it. Otherwise a typo in a path just leaves the placeholder in place and fails later with a less helpful error.
There was a problem hiding this comment.
OK. I am going to extract the chain providers like in other workflows. But I am not going to add the overrides in this PR. That should be separate.
| child = group[name] | ||
| found = snx.Field if isinstance(child, snx.Field) else child.nx_class | ||
| if found != nx_class: | ||
| warnings.warn( |
There was a problem hiding this comment.
Should this raise instead? A name that resolves to the wrong class looks like a configuration error, and with a warning we continue with the wrong component.
There was a problem hiding this comment.
Or a faulty file. I wanted to still allow reading files in case the classes are not set correctly. But, AFAIK, we don't support these kinds of fallbacks elsewhere. So should I turn this into a hard error?
There was a problem hiding this comment.
I'd prefer not having such fallbacks/workarounds creep into workflows. #839 would be my preferred solution.
This speeds up the workflow significantly because it avoids repeated `.bins.concatenate` calls.
scipp/ess#802 removed the argument of `BifrostQCutWorkflow`, the detector names are now a parameter. It also renamed `load_analyzer_for_detector` to `load_bifrost_analyzer_for_detector`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which the job-level motor gating imports. essspectroscopy 26.11.1 includes the scipp/ess#802 changes this branch adapts to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
scipp/ess#802 removed the argument of `BifrostQCutWorkflow`, the detector names are now a parameter. It also renamed `load_analyzer_for_detector` to `load_bifrost_analyzer_for_detector`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
essreduce 26.11.0 provides TransformationOverrides (scipp/ess#858), which the job-level motor gating imports. essspectroscopy 26.11.1 includes the scipp/ess#802 changes this branch adapts to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
EDIT
This refactors the BIFROST workflow to avoid Sciline's map-reduce.
There are some new types and providers in ess.reduce for handling multiple instances of the same nexus class, all names something with 'many'. The BFIROST workflow uses those to load all (or a subset of) detectors and analyzers. This puts the old loops into the providers.