Alternative export option. Add 16 bit compressed TIFF as output option (substitute for compressed XISF)
Issue #1069
closed
Nina can create compressed XISF files. This format is still rarely used.
I got a request to add compressed XISF functionality to ASTAP to save disk space. Adding XISF to ASTAP is not easy since the Pleiades Astrophoto library is written in C and can not be easily be used in ASTAP since ASTAP is written in Pascal programming language.
As an alternative I would suggest to add 16 bit compressed TIFF as one of the Nina output options. The FITS header can be stored inside the TIFF
Advantages:
- Standardized 16 bit format with compression. Almost any program can read it.
- Full header information can be stored in TIFF (TiffImageDescription)
This is already implemented in ASTAP. Exporting a FITS file to 16 bit TIFF and backwards restores the original file. The FITS header is stored inside the TiffImageDescription.
TIFF also supports 32 bit integer or 32 bit float.point images and 64 bit.
Attached an example of a TIFF file written by ASTAP where the FITS header is stored in the TIFF. ASTAP will read the files like a FITS file.
Han
author of ASTAP
Comments (29)
-
repo owner
-
reporter
- edited description
-
reporter
Hi Stefan,
Would it be possible to add the header like ASTAP. That would create a new astro standard.
-
repo owner
I’ll check out how this header is written by ASTAP and see if I can adapt our TIFF writer to that.
-
reporter
This is the way it looks in Photoshop. See field description:
-
We did a native implementation of XISF in C#, so I would suppose the same would be possible in Pascal. The compression algorithms are popular and I would imagine there are implementations of them for Pascal. There is no need to create a new “standard” because of that. And, stuffing unstructured data into the TIFF description field is more of an agreement than any kind of standard.
-
reporter
It is not unstructured stuff. It is a copy of the FITS header following the FITS standard.
-
repo owner
Implementing such a thing seems to be quite trivial. I have checked the code and did a small adjustment to the TIFF writer and this is the result file:
https://www.dropbox.com/s/mxmlrdcwczxv8lv/_2022-02-09_13-48-06__30.00_5.00s_0000.tif?dl=0I agree however, that this is not the cleanest approach, as the Title of a TIFF can contain anything (unstructured data) and if you want to read it, you need to check first if this is really a fits header structure or not.
It’s still better than no description at all, so maybe we can add this. -
reporter
The file including header is full readable by ASTAP. No problem. This is quick implemeted.
The text start with SIMPLE and that is the standard FITS check, so no mistakes can be made interpretation of the description field.
If have to tweak the ASTAP code that it is maintains the file as TIF. In case of stacking it currently converts to FITS but that is an easy tweak.
I you can release this officially that will make me a happy man
Implementing compressed XISF in a Pascal language program is a too large project, I really don’t want to do that.
-
repo owner
I’ll do some more tweaks to the code and also maybe add some capabilities to read out the header when loading from tiff files again. (useful for example in framing)
Won’t have much time today to do this, so maybe in the next couple of days I can finalize it and discuss if we add this to the beta. -
The problem with this is that you cannot look at any given TIFF file and know that it contains a certain blob of information in a certain field that has no formal definition beyond containing free-form text. Merely transplanting a FITS header out of a FITS file and into that field does not create a standard, and there is the additional work of now having to try to determine if the text found in that field looks and smells like a FITS header. The FITS data is being crammed into a field that gives no forward indication that it contains information of that format, and FITS data itself is unstructured because it cannot be readily serialized or deserialized without first going through a series of tests to identify it. This is, plainly, a hack that attempts to sidestep a larger issue.
Like ASTAP, NINA started with support only for uncompressed XISF and, like ASTAP, we have our own implementation of XISF creation and parsing - we also do not use the Pleiades C library. Adding support for reading and writing compressed XISF took a little work, but the work involved can be divided into two realms:
- Rewriting the existing XISF header parser and writer to understand the presence of a compressed XISF data block
- Uncompressing or compressing that data block using any one of the 2 defined algorithms (LZ4 and LibZ)
The easy part is reading and writing the header to know that the data block is compressed and what kind of compression algorithm is used. The harder part is implementing the compression algorithm, but it appears that FreePascal makes that easy via the Parallel Compression Library, which supports a wide range of old and modern compression algorithms.
So you might want to take a closer look at compressed XISF. For those with high-megapixel cameras and work in PixInsight (NOT a rare combination), compressed XISF is a way of life and it’s the main reason why I implemented it in NINA.
-
I want to also note that TIFF compression algorithms are -not- CPU friendly. They are set up for maximal compression in mind and are not designed for speed. After all, the origin of the TIFF format was for sending faxes over phone modems in the 1980s. The LZ4 algo that XISF defines is a best of both worlds algorithm - it is not as strong as LibZ or LZW, but it gets close for not much of a CPU hit. We use the same compression algo in the storage appliance world when streaming data too and from disk - it is computationally inexpensive as far as compression algorithms go and is an established and well-maintained format.
-
reporter
Dale, I’m looking for a practical solution. It is not about which format is better. Implementing compressed XISF is a huge complicated project for me and I’m not sure if I can technically do it. There is nothing wrong with using the description field of a TIFF file. That is the intention of the description field. That will work for users who want to stack files with ASTAP. Pixel insight users can user XISF. And hopefully everybody is enjoying astro imaging.
Han
-
The want for “practical” solutions is what fostered bad design decisions such as the Alpaca ImageArray fiasco. I understand that you had a hand in correcting that huge mistake, so you might appreciate the issues the original implementation created and how it fell short in practice because no forethought was given to it when it was originally designed. I see this proposal in the same light - it’s a hack that coopts a random TIFF field for purposes it wasn’t intended for; the only attraction is that it’s convenient. Convenient does not always mean correct or better, and everyone involved now runs a high risk of assuming technical debt that must be paid in unexpected ways later.
-
repo owner
Dale I agree that solely depending on this Description field is quite arbitrary and should not be seen as a standard.
However an empty TIFF description field serves even less purpose and there is no harm at least from N.I.N.A. side to add this simple part into the TIFF header without heavily relying on it for anything. Then users who are utilizing TIFF have at least some form of readable description inside their image headers. The code will just utilize the already existing FITS functionality of N.I.N.A. and no complex utility is added, but rather existing things slightly adapted.
-
It’s a big mistake to expect that this arrangement will be contained to only NINA and ASTAP. People will eventually want these files to work in other apps throughout the entire pipeline, turning an ad hoc agreement between two apps into a decision that others must make. And when an application maintainer says “no, I will not support that”, then you have created the same situation as XISF compression. Standards are standards because they are clearly and deliberately defined. They tend to work because consideration and role are part of the input and they are optimized for solving the problem at hand.
-
repo owner
I have adapted the latest beta to write this into the description. However this should not be seen as any standard, as it is an arbitrary decision to put this in this particular format into the file. It is merely a convenience.
Maybe we can collaborate with you Han to bring XISF compression capabilities to the ASTAP code.
-
reporter
Thanks Stefan. Then we can get some field experience with it.
Not everyone is on the same boat. For example the stack program Siril is embracing the standardized FITS compression (typically RICE compression) as available in the package CFITSIO
https://heasarc.gsfc.nasa.gov/docs/software/fitsio/compression.html
ASTAP and CCDciel program support this using a cfitsio ccommandline program.
But the pros and cons are better discussed in an other forum/media.
-
reporter
In cooperation with Cyril (stack program Siril) and Patrick (CCDCiel aquisition program) I have written an online specification:
http://www.hnsky.org/astro-tiff.htm
-
There exists TWO other astro image standards that have compression and, of course, contain astronomy-related metadata in far more formal facilities. Why do we need a third?
-
reporter
Fact: TIFF is an exisiting feature in any astronomical application because TIFF support 32 bit images.
The specification only specifies what to do with the existing base line tag Description. The alternative is to keep it empthy which would be a waste.
I create the following benefits:
- The TIFF compression algorithm makes the files smaller then FITS but contains the same information
- The files are readable by almost any image viewer.
- Easier to implement in many software applications due to the availability of TIFF libraries.
This all by a few lines of code. So why not? Benefit 2) is unique.
-
FITS is also an existing feature of any astronomical application. It also supports 32bit images, and even larger data types. Again, why create a third major standard that is less capable than either of the two existing other standards?
FITS is the original, and offers optional compression (albeit hampered by the fact that it’s implemented in the reference CFITSIO code as an executable, not a library)
XISF was invented to improve the shortcomings and oversights of FITS and bring the metadata facility into the modern era. XISF is, objectively, an improvement upon FITS.
I don’t see how “Astro TIFF” contributes anything to the existing environment and why it is necessary.
-
XISF is, objectively, an improvement upon FITS.
For me, this is not the case. XISF is a format created by PixInsight dev for PixInsight users. There is no interest for people using other software to use it. Therefore this is not a standard as FITS file are.
The fact that TIFF files can be read by all software is a good point for Astro-TIFF, even FITS files are not easily readable by common visualization software. Of course, this does not create a third major standard and this is not the purpose, but it is a good hack for astro images that does not hurt.
-
repo owner
- changed status to closed
Closing this issue, as latest beta already puts the fits like header into the files. For compression refer to
#1074 -
For me, this is not the case. XISF is a format created by PixInsight dev for PixInsight users. There is no interest for people using other software to use it. Therefore this is not a standard as FITS file are.
XISF is an open standard that has a full and patent-unencumbered definition. Anyone can implement it, as we have in NINA. XISF’s goal is to correct oversights and outdated concepts in the FITS format. Limitations in metadata length, arbitrary keyword use and adoption, defining an origin of the image array, and metadata that can be readily serialized and deserialized are part of why it exists. XISF would not have to exist if the FITS standards committee existed as a functional entity and worked to evolve the standard.
From this viewpoint, FITS is a “dead” specification that lacks evolution or forward momentum. You can take the community-organized support for the ROWORDER keyword as an example. It is a sign of the community attempting to paper over at least one glaring shortcoming of FITS that will likely never be addressed in the core specification. That happening, along with alternative formats such as XISF popping up, are a sign that FITS is not suitable for today’s demands.
Astro-TIFF is another attempt to put a band-aid on the situation, but it attempts to replace one design antipattern with an objectively worse one. It also makes no attempt to address the metadata issues that we have to live with in FITS and still invent hacks to work around. These issues are really only addressable by a new format, and we have XISF for that. If there are any lackings in XISF, such as compression types, then the standard can be evolved to incorporate them.
And who is the target audience of Astro-TIFF, anyway? People who wish to go straight from imaging app to… PhotoShop? MS Paint? These are not applications that are well-suited for processing astro images, and they won’t care about metadata anyhow. As elitist as it might sound, I don't think I need to be concerned with people who do that with their images. So who will this really benefit?
-
And the ROWORDER keyword brings up something about Astro-TIFF. The orientation of a TIFF image array is defined in the image’s baseline Orientation field. The Astro-TIFF description does not define what one should do if the TIFF orientation conflicts with any included ROWORDER keyword.
-
From this viewpoint, FITS is a “dead” specification that lacks evolution or forward momentum. You can take the community-organized support for the ROWORDER keyword as an example. It is a sign of the community attempting to paper over at least one glaring shortcoming of FITS that will likely never be addressed in the core specification. That happening, along with alternative formats such as XISF popping up, are a sign that FITS is not suitable for today’s demands.
I do not agree, this ROWORDER keyword was created because a lot of software did not follow the original bottom-top convention. But this could happen with a lof of format. And in fact, it did happen with SER file that the spec was not followed: https://free-astro.org/index.php?title=SER#Specification_issue_with_endianness In this case we must adapt things. While Pixinsight finds the flexibility of the FITS file a weakness, I think the opposite. And I did used FITS files a lot, even in my work with files from professional telescope, with no issue at all.
XISF is an open standard that has a full and patent-unencumbered definition
Writing a specification does not make the format a Standard. A standard must be widely used too. It must be created in response to the community needs. Here it was created in response to the needs of PixInsight developers.
Of course, for a software like NINA, a data producer, it is very useful to produce XISF file for people that use PixInsight. No problem for that. But for data consumer software like ASTAP, Siril, APP or others, there is no need, I repeat. However, I don’t want to flood this topic, and I really appreciate the huge effort you put in NINA development (even if I’m a Linux User). Thanks for adding the Astro-Tiff feature too.
Regards, -
I do not agree, this ROWORDER keyword was created because a lot of software did not follow the original bottom-top convention. But this could happen with a lof of format.
“A lot of formats” do in fact define row order or the origin pixel of the 2D array. FITS, conspicuously, does not. It is up to each implementor’s wishes as to how they interpret row order. Obviously, this particular example has not been conducive to interoperability and the user observing consistent results across applications. Conventions regarding it are not part of any standard. Conventions are, at the absolute best, informal agreements and are the equivalent of “empty calories” when it comes to ensuring interoperability. In fact, they can have a detrimental effect on interoperability in some cases. They might feel nice but, in the end, they mean nothing - the specification and standard still reign. The introduction of ROWORDER attempts to bring some civility to the vacuum formed by the lack of a formal definition in FITS but, again, this is an informal agreement.
The correct route would be to advocate for a revision to the FITS standard that formally defines a row order or a way to indicate it, as it can be argued that this issue is structural in nature and there is a demonstrated need for it. It is never too late to do this but the FITS standards committee does not appear to have any lights on in their house, so the community is left to create their own informal extension which no one is obliged to observe.
And in fact, it did happen with SER file that the spec was not followed: https://free-astro.org/index.php?title=SER#Specification_issue_with_endianness In this case we must adapt things. While Pixinsight finds the flexibility of the FITS file a weakness, I think the opposite. And I did used FITS files a lot, even in my work with files from professional telescope, with no issue at all.
The SER format is an even more tragic situation. Here, there isn’t a full definition of it publicly available anywhere and no addressable organization responsible for it. I’ve looked; there just isn’t one. If such a document does exist, it’s likely accessible only by Teledyne/Lumenera customers. The SER format is the worst case in that it is a “standard” that is replicated only through mimicking other implementations - and usually these are closed-source implementations (SharpCap, Firecapture.) Given that, it is absolutely no surprise to me that interoperability issues exist between different applications. Honestly, no one can be blamed for anything other than attempting to utilize a closed format and making mistakes as they attempt to feel their way around in the dark.
Writing a specification does not make the format a Standard. A standard must be widely used too. It must be created in response to the community needs. Here it was created in response to the needs of PixInsight developers.
If you want to say this about XISF then you might as well say it as well for Astro-TIFF. In fact, the stated reason of existence for Astro-TIFF is “implementing XISF compression is too hard.” Ok whatever, but standards don’t come about because one developer finds the existing suitable alternatives to be outside their own coding ability. They come to fill a need for users and application developers, and the Astro-TIFF concept strains to do both and does not accomplish anything better than the existing alternative. And that’s an extremely charitable assessment. It also has some glaring oversights, one of which I already mentioned.
A “standard” does not exist because some arbitrary critical mass of people happen to use it. A standard is more than just a document. It it a considered and defined design that was thought out, debated, and addresses specific goals that existing standards do not. An open standard is one that invites input and governance by more than one person or entity, among other aspects. Every standard starts with a small number of implementations and grows from there. Adopting it means putting aside any notion of “not invented here” and working to improve it further if improvements can be articulated.
-
repo owner
- removed version
Removing version: 2.0 Beta (automated comment)
- Log in to comment
Nina can already output compressed tiff.
It just doesn't include the fits header