Add Rice (and/or GZIP) compression to FITS export
Issue #1074
closed
Hello.
Following the issue opened by Han (from ASTAP) here: https://bitbucket.org/Isbeorn/nina/issues/1069/alternative-export-option-add-16-bit
I think it could be great to implement the RICE compression in the FITS export. Indeed, a lot of open-source software are using it (like Siril) and it works pretty well, especially for 16bits images.
No special file extension would be required even if sometime we found the fz one.
Here the cfitsio reference about FITS compression: https://fits.gsfc.nasa.gov/registry/tilecompression.html
Comments (8)
-
-
reporter
- changed title to Add Rice (and/or GZIP) compression to FITS export
-
I have done a compression test of the different algorithms for 95 files collected from nova.astrometry.net. all in 16 bit format. Total file size was 2190 mbytes.
Siril with Rice compression is lightning fast. Even good old LZW in TIFF is doing well. The Pixinsight implementation is disappointing in speed. I can not do this test with Nina since there is no batch conversion available as I know
Han
I gave the following hints in PI:compression-codec lz4hc+sh
compression-codec zlib+sh
-
The “sh” is byte shuffling. This is a pre-compression (and post-compression) action where the high and low order bytes are rearranged in the array (“shuffled”) to optimize the data more to obtain a better compression ratio. If you want to compare compression speed, and only compression speed, you will want to turn byte shuffling off, as reordering an entire array of pixels can be quite time consuming. The benefit of byte shuffling on the compression ratio will of course vary from example to example. In XISF, the entire data block, which is usually the entire image array, is compressed as a single unit.
FITS compression happens on a “tile” basis. I think in fpack, the default tile size is 1 row of pixels. So this means the selected compression algorithm is compressing 1 row of pixel data at a time. An image with 2000 rows will have 2000 individually compressed tiles. The tile size can be larger, to compress more rows as a single unit. There can be benefit to this, or it can hurt the overall compression ratio; it depends on the data being presented. I think it is largely a trade-off between compression time and ratio.
-
Without sh it is worse in compression and speed. For XISF with “compression-codec zlib+sh” you achieve the best compression but with a massive penalty of almost 4 seconds per file. Something is very inefficient or this XISF format is not good for images.
I can share the images for testing and maybe the Nina implementation is more efficient?
-
Correction ASTAP TIFF routine is using Deflate for compression rather then LZW.
-
repo owner
- changed status to closed
-
reporter
Glad to hear read that. Thanks a lot !
- Log in to comment
I can only support the wider use/availability of the compression as documented in the FITS standard 4.0. The Rice algorithm seems to provide the best lossless compression for images. Better then general purpose compression algorithms. E.g. documented here https://heasarc.gsfc.nasa.gov/fitsio/fpack/ And it is adapted by the proffesional astro community.
Personally I would prefer the fit.fz extension as used in the fpack, funpack utilities for better detection. This could avoid programs without the decompression capabilities to even try.