Test JF16M

[phyy-nx]

I'd like to add https://zenodo.org/record/3352358#.XYJrbtNKj2I for regular testing and I think dials-data is the way to do it but I don't know how. It's a 4 GB dataset.

Specifically I want to ensure that multi-module NeXus datasets are tested. See #92.

[graeme-winter]

Is it possible to test with a smaller data set? This means every developer needs to get the data locally

[Anthchirp]

You could make the test conditional on some flag, then only those who use the flag would need to download it. Within the DLS network there would be no need to download it either as dials-data is loaded from a shared location.

[rjgildea]

@phyy-nx I've begun adding tests for some of the nexus.py Factory classes in https://github.com/cctbx/dxtbx/commits/nexus-refactor. It should be possible to test some of this functionality by mocking.

[rjgildea]

Regarding adding datasets to dials-data - you just need to copy one of the existing .yml files in https://github.com/dials/data/tree/master/dials_data/definitions, edit the various fields appropriately, and then submit a pull request with your changes.

[phyy-nx]

@graeme-winter I guess, though I would have to re-write the h5 file myself which I haven't done much of. Is https://zenodo.org/record/3385862#.XYKBo9NKj2I part of dials-data and used in a test? I see it's quite a bit smaller.

@Anthchirp I'm trying to figure out dials-data works. I see it has pointers to dials/data-files, a repository with a few image examples and it has pointers to zenodo and elsewhere. Can you point me to the test that uses, say, https://github.com/dials/data/blob/master/dials_data/definitions/thaumatin_eiger_screen.yml?

@rjgildea mocking sounds interesting, but I'd like to start by testing the raw data itself. Making my own yml makes sense, I just need to know how to use it.

[rjgildea]

@graeme-winter I guess, though I would have to re-write the h5 file myself which I haven't done much of. Is https://zenodo.org/record/3385862#.XYKBo9NKj2I part of dials-data and used in a test? I see it's quite a bit smaller.

I don't believe so - at least there isn't a reference to zenodo dataset 3385862 in any of the definition .yml files.

@Anthchirp I'm trying to figure out dials-data works. I see it has pointers to dials/data-files, a repository with a few image examples and it has pointers to zenodo and elsewhere. Can you point me to the test that uses, say, https://github.com/dials/data/blob/master/dials_data/definitions/thaumatin_eiger_screen.yml?

https://github.com/dials/dxtbx/blob/92f4eb4a92333d19943623085a78f853828cd4b2/tests/format/test_FormatNexusEigerDLS16M.py#L142

[phyy-nx]

👍

[graeme-winter]

@phyy-nx downloading the data now - for 68 images is a lot of bytes. My best guess here is that this is completely uncompressed...

cs03r-sc-serv-16 jungfrau :( $ h5ls -r lyso009a_0087.JF07T32V01.h5 
/                        Group
/data                    Group
/data/data               Dataset {68, 16448, 1030}
cs03r-sc-serv-16 jungfrau :) $ ls -l 
total 4500084
-rw-rw-r--. 1 gw56 dls-softinst 4608074112 Sep 14 16:28 lyso009a_0087.JF07T32V01.h5

That would be yes:

cs03r-sc-serv-16 jungfrau :) $ python
Python 2.7.5 (default, Jun 11 2019, 12:19:05) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 68*16448*1030
1152017920
>>> 4608074112./_
4.000002111078272

i.e. exactly 4 bytes / pixel.

[graeme-winter]

Is there any reason (probably something to discuss with the PSI) not to build a VDS over this to make it look like a single 16M array rather than array of 32 panels? Would save a bunch of development effort and is exactly what hdf5 has VDS for. That said this is not a review of this data set.

I've tried h5repack to compress with gzip / SHUF and it really does not. This is very noisy data?

[phyy-nx]

It's not that it's noisy (though it somewhat noisy), but it's more likely to do with the fact that each pixel is auto-swapped into one of three gain modes, with each gain mode being calibrated per-pixel, along with 3 possible pedestals per pixel. So I imagine these corrections add extra error making compression more difficult. The shape of the array in the data file is 68*16448*1030. Is that not already the 16M block you suggest? The NeXus master file (created using https://github.com/cctbx/cctbx_project/blob/master/xfel/swissfel/jf16m_cxigeom2nexus.py) uses metadata to split the panels virtually, but the data stays in its block.

…

On Wed, Sep 18, 2019 at 10:59 PM Graeme Winter ***@***.***> wrote: Is there any reason (probably something to discuss with the PSI) not to build a VDS over this to make it look like a single 16M array rather than array of 32 panels? Would save a bunch of development effort and is exactly what hdf5 has VDS for. That said this is not a review of this data set. I've tried h5repack to compress with gzip / SHUF and it really does not. This is very noisy data? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93?email_source=notifications&email_token=ADGY5SQOYMTEXOD5RK5RSWDQKMIKDA5CNFSM4IYCFIX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7CJ5MI#issuecomment-532979377>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGY5SU5WXV4ZTCJV3PXYG3QKMIKDANCNFSM4IYCFIXQ> .

[biochem-fan]

Oh, the data type is H5T_IEEE_F32LE! This is extremely difficult to compress...

The same thing happened in the CryoEM field before and the solution was to save gain-nonnormalised raw data in integers (which compress nicely) and apply corrections (i.e. multiplication by gain) on the fly during processing.

In SACLA, there is no API to obtain such raw data, so I ended up rounding floats into integers after scaling them such that one photon corresponds to 10 arbitrary unit. This leads to loss of precision, but the gain correction itself is not very accurate, and the result was indistinguishable.

[phyy-nx]

Excellent, thanks. I happened to have just sent a separate email to dials-support about on the fly processing! Very interested in this :)

…

On Thu, Sep 19, 2019 at 10:14 AM biochem_fan ***@***.***> wrote: Oh, the data type is H5T_IEEE_F32LE! This is extremely difficult to compress... The same thing happened in the CryoEM field before and the solution was to save gain-*non*normalised raw data in integers (which compress nicely) and apply corrections (i.e. multiplication by gain) on the fly during processing. In SACLA, there is no API to obtain such raw data, so I ended up rounding floats into integers after scaling them such that one photon corresponds to 10 arbitrary unit. This leads to loss of precision, but the gain correction itself is not very accurate, and the result was indistinguishable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93?email_source=notifications&email_token=ADGY5SW5HI2ZJWHZUJIVM4LQKOXO5A5CNFSM4IYCFIX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7EGCBQ#issuecomment-533225734>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADGY5SQF5WBQCWOOXY3UPXLQKOXO5ANCNFSM4IYCFIXQ> .

[phyy-nx]

(We'll want to use @yayahjb's revised version here: https://zenodo.org/record/3526738)