Repository navigation
Conversation
| - Names of `NXdetector` groups are the names of the groups in the NeXus files of the runs. | ||
| A name that is not a bank of the instrument is an error, since a misspelled name would otherwise mask nothing. | ||
| - The name of a dataset in `masks` is the name of the mask in the reduced data. | ||
| Datasets with the same name in several banks form one mask. |
There was a problem hiding this comment.
Copilot decided this should go. I also fail to see how they could form just one mask, since detector banks are loaded separately in the nexus workflow. It would imply that the mask has a name and then lists the detector banks inside it, which leads to the previous data structure.
I think we are still checking that masks with the same name in different detector banks should have the same dims.
| Readers accept any integer type, and the boolean enumeration that h5py writes for numpy booleans. | ||
| - Dimensions of a mask are named with HDF5 dimension labels. | ||
| They are a subset of the logical dimensions of the bank, in any order, with matching sizes. | ||
| Masks with the same name in different banks use the same dimension labels, though their sizes may differ. |
There was a problem hiding this comment.
Why was this added? There now is a new function to enforce this requirement. But do we actually need it?
There was a problem hiding this comment.
Also, this is not possible, e.g., for DREAM, right?
There was a problem hiding this comment.
In which case is it now possible for Dream? Do you mean because some dimensions do not exist in all banks?
I guess if you are in bank 1 and have a mask named mask1, and also a mask1 in bank 2, you don't need to enforce anything.
I guess I was just thinking about mask names like bad_tubes, but that is just a user-defined name. We shouldn't deduce any logic from the names of the masks.
But it's still important to raise if there is a mask for the same bank, with the same name, in two different mask files that are both being loaded.
There was a problem hiding this comment.
Do you mean because some dimensions do not exist in all banks?
Yes, that is what I mean.
| for mask_name, mask in content.items(): | ||
| if mask_name in masks: | ||
| raise ValueError(f"Several masks have the name '{mask_name}'.") | ||
| masks[mask_name] = _align_bank_mask(mask, detector.sizes, mask_name) |
There was a problem hiding this comment.
This entire function looks fishy. name is a bank name, so why does the first branch blindly assign that to masks? And why does the last branch check the name? And why does this loop at all instead of calling pixel_masks.get(bank)? The bank is None case doesn't seem to be supported by PixelMasks.
There was a problem hiding this comment.
As I understand it, this function takes in a list of masks for all detectors, and tries to extract the masks for the requested detector.
First branch blindly assigns to masks
The isinstance(content, sc.Variable): case is for the ISIS xml masks case, where the loaded masks look like:
{'mask_new_July2022.xml': <scipp.Variable> (detector_id: 249022) int32 <no unit> [1, 2, ..., 458751, 458752]}
I believe this is also why we can't just pixel_masks.get(bank). In the case above, if we try to load some xml masks for Loki, it would fail because bank which is loki_detector_0 would not be in the dict of masks.
Instead, we have to lookup the detector ids and determine if they are in the requested bank or not.
So the asumption is that if the value is a Variable, it must be a mask from an xml file (which only contains det ids, has no concept of bank names).
I guess this is quite a bold assumption, but maybe ok? I fail to see how you could save masks such as
{'bad_tubes': sc.zeros(sizes={'layer':4, 'tube':32}),
'bad_straws': sc.zeros(sizes={'layer':4, 'tube':32, 'straw':7}),}that would be for multiple detector banks, as each bank usually has its own number of layers, tubes, straws. And you don't want to have a mechanism to identify the right shape, as it can be ambiguous.
You can of course save a mask in a scipp hdf file, that would hold detector ids, like the xml mask, but then we would just treat it as the xml case above.
The bank is None case
I don't understand fully this case. For isis xml masks, this part of the code is not used.
For sans (Loki), the bank name is always specified.
Powder diffraction is the only use case I found where bank is None, but couldn't really wrap my head around how the pixel_masks were defined (they are just empty dicts in the docs examples).
if known_banks is not None and name not in known_banks:
I guess that's to protect against bank names that are misspelled, so that we raise an error instead of sliently dropping the masks?
There was a problem hiding this comment.
That seems too complicated. I would normalise when loading instead of having a complex extraction logic because we might look at the masks dict manually or using other tools as well. So I'd rather have a consistent in-memory structure.
There was a problem hiding this comment.
I also thought that would be better, but I think it would mean you need to know the detector name to be able to load the masks, meaning that you can't just load one nexus file with masks all in one go, you just load the part for the detector that you're interested in.
Which in turn makes it more difficult to just load the masks and inspect them?
There was a problem hiding this comment.
Is this only a concern for XML mask files where we don't have a bank name in the file? I see two options for adjusting the representation:
- Use a catch-all bank name like
"__all__". But that would mean checking for it in the extraction function. - Use a dedicated type that allows us to check whether we have this case or the ESS case of one entry per bank. E.g., two separate classes or a class with an attribute (returning an enum) to distinguish the cases.
I just don't like having to inspect the contents of a dict and guessing that if you see a variable, the dict key is a mask name, otherwise it is a bank name.
There was a problem hiding this comment.
Could we have a special provider for reading xml masks?
It could take in the detector numbers and the detector name as inputs, to output a mask just for the required bank, as for the others.
But it means it needs to load the detector numbers before it can read the masks (but maybe that's fine?).
Do we expect to need the detector id based masks for non-xml mask files?
@SimonHeybrock I could use some input here 🙏
In #874 we fixed the order in which masks were written to file.
We apparently forgot to fix the reader as well, which we do here.
Most importantly, we add a round-trip test which verifies that masks saved to disk and loaded again match the original masks.