The lz4 data in Stream2 is different from Stream

[MarkRivers]

The data returned over the Stream2 interface appears to be the same as that returned by the Stream interface if compression=None or bslz4. The data sizes are the same, and the bslz4 decompressor that ADEiger uses for Stream works fine for Stream2.

However, the lz4 data are different. The following are the lz4 compressed sizes for a particular detector configuration with a radioactive source and 10 second acquire time:

Stream:  lz4 compressed size = 771KB
Stream2: lz4 compressed size = 1,119 KB

The lz4 size with Stream 2 is 45% larger than with Stream.

When I call the lz4 decompressor that is used with Stream it crashes when passed the lz4 data received by Stream2.

This seems strange. I have asked Dectris whether indeed the lz4 compressed data is different for Stream and Stream2.

[MarkRivers]

I have been unable to decompress the lz4 data from the Stream2 interface by directly calling the lz4 library the way I did with the Stream interface.

However, I have been able to decompress it by calling the functions in the file compression.c supplied by Dectris. For now I have added this file in eigerApp/compressionSrc.

I would like to avoid using that file if possible, but it is not high priority, since there is a working solution.

[MarkRivers]

It appears to be necessary to use the compression.c with Stream2. This is implemented, so closing this issue.

[MarkRivers]

This issue is actually important. If the LZ4 data is passed uncompressed to the NDPluginCode to decompress it crashes the IOC. If it is passed uncompressed to NDPluginHDF5 it does not crash the IOC, but it crashes h5dump when attempting to read the file.

[MarkRivers]

This is a message I just sent to the Dectris staff in the US about this issue:

---

Hi Melanie and Zack,

I need to come back to this issue with lz4. Let me explain what our needs are and what I observe.

In the EPICS ADEiger driver as images arrive on the Stream or Stream2 interface we have the option to decompress them in the driver, or to pass the compressed data directly to the areaDetector plugins.

Only 3 of the areaDetector plugins can handle compressed data. These are:

• NDFileHDF5, which writes HDF5 files. If it receives uncompressed data it can compress it using HDF libraries functions. If it receives compressed data it writes it directly, using what is called Direct Chunk Write. This method is highly desirable for the Eiger, because it lets us add our own metadata to the HDF5 file, and structure the file as the users want, without needing to decompress and then recompress the data.
• NDPluginPva. This plugin sends the images to clients using the EPICS pvAccess protocol. It knows how to handle compressed images. We have written plugins for ImageJ and the EPICS CSS/Phoebus display program that can decompress the images and display them. This allows us to send compressed images across the network to client display computers, drastically reducing the network bandwidth, particularly for large detectors like the 9M and 16M.
• NDPluginCodec. This plugin decompresses the images from the ADEiger driver so that other plugins like statistics, ROIs, etc. can process them.

The architecture I describe above works perfectly for the Stream interface. It also works perfectly for the Stream2 interface when using BSLZ4 compression. This is because the images can be decompressed using the standard bitshuffle and lz4 library functions. HDF5 files written using the Direct Chunk Write of compressed data are fine.

There is, however, a major problem with lz4 compression on the Stream2 interface.

• I cannot decompress the data using the standard lz4 library.
• I can decompress the images using the lz4 decompression code that Dectris distributes. Thus, ADEiger driver can decompress the images and pass them to plugins. However, as described above this is very undesirable, we need to be able to pass compressed data to the plugins.
• The standard lz4 decompressor which is used in NDPluginCodec, ImageJ, CSS/Phoebus, and the HDF5 library cannot decompress the images, and trying to do so typically crashes the applications.
• The lz4 compression in the Stream2 interface produces files that are often 45% larger than the files produced in using the Stream interface.

This is a listing of LZ4 and BSLZ4 HDF5 files written by NDPFileHDF5 using Direct Chunk Write of the compressed data directly from the Eiger 9M. Each file contains 10 images.

-rw-rw-r--   1 epics     domain users   7463742 Jan 24 17:31  Stream_BSLZ4_001.h5
-rw-rw-r--   1 epics     domain users   7465572 Jan 24 17:36  Stream2_BSLZ4_001.h5
-rw-rw-r--   1 epics     domain users   2653958 Jan 24 17:34  Stream_LZ4_001.h5
-rw-rw-r--   1 epics     domain users   3861409 Jan 24 17:39  Stream2_LZ4_001.h5

Note that the BSLZ4 files collected with Stream and Stream2 are nearly identical in size. The Stream LZ4 file is almost 3 times smaller than the BSLZ4 files, which shows that LZ4 is better than BSLZ4 in some cases. However, the LZ4 file collected with Stream2 is 45% larger than the LZ4 file collected with Stream. This does not make sense to me, LZ4 compression should produce the same size file.

This is what I see when I run the Linux utility h5dump on these files:

$ h5dump --dataset /entry/data/data Stream_BSLZ4_001.h5 | more
HDF5 "Stream_BSLZ4_001.h5" {
DATASET "/entry/data/data" {
   DATATYPE  H5T_STD_I32LE
   DATASPACE  SIMPLE { ( 10, 3262, 3108 ) / ( 10, 3262, 3108 ) }
   DATA {
   (0,0,0): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
   (0,0,22): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,


$ h5dump --dataset /entry/data/data Stream2_BSLZ4_001.h5 | more
HDF5 "Stream2_BSLZ4_001.h5" {
DATASET "/entry/data/data" {
   DATATYPE  H5T_STD_I32LE
   DATASPACE  SIMPLE { ( 10, 3262, 3108 ) / ( 10, 3262, 3108 ) }
   DATA {
   (0,0,0): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
   (0,0,22): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,


$ h5dump --dataset /entry/data/data Stream_LZ4_001.h5 | more
HDF5 "Stream_LZ4_001.h5" {
DATASET "/entry/data/data" {
   DATATYPE  H5T_STD_I32LE
   DATASPACE  SIMPLE { ( 10, 3262, 3108 ) / ( 10, 3262, 3108 ) }
   DATA {
   (0,0,0): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
   (0,0,22): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

$ h5dump --dataset /entry/data/data Stream2_LZ4_001.h5
HDF5 "Stream2_LZ4_001.h5" {
DATASET "/entry/data/data" {
   DATATYPE  H5T_STD_I32LE
   DATASPACE  SIMPLE { ( 10, 3262, 3108 ) / ( 10, 3262, 3108 ) }
Segmentation fault (core dumped)

Note that h5dump displays all files are fine except the one with LZ4 data from Stream2. The file was written successfully, but the h5dump program crashes when it tries to decompress the lz4 compressed data.

I am quite certain this is because the LZ4 compressed data from Stream2 is not actually valid LZ4 compression. No clients that know how to decompress LZ4 data will be able to handle it.

I think the Stream2 interface needs to be fixed so that it uses the same LZ4 compressor as the Stream interface.

Thanks,
Mark