Sequence stops downloading images from ASI553MC and saving to network drive

[Mark Aston]

I have a problem in that NINA (V1.10 HF3) stops displaying and saving images from my new ZWO ASI553MC Pro to a network drive after the first image (or couple of images) have been saved. I’ve tried both native and ASCOM driver for this in NINA and I repeatedly get this issue. I don’t know if this is a bug or a problem caused by needing to delay start of next image acquisition to allow previous image to be saved to the network drive. Has anyone else had this problem?

[Dale Ghent]

Would need to see a log file from NINA, preferable at the debug level of output, to try to get a sense of what is going on. The different mechanisms in the app that you cover in the description are too varied and unrelated to be able to form a clear understanding of what might be happening.

‌

[Mark Aston]

Hi Dale, I have just run a sequence of 30s exposures, saving to a test directory. Only the first image saved. The Debug log file is just about to be attached.

[Mark Aston]

Hi Dale, I have just attached a run using the native NINA ASCOM driver for the ASI533MC, saving to the local observatory PC HDD rather than a NAS. Same issue with only the first acquired image being saved, the rest not. It also appears that the only image displayed by NINA is the first image, so I suspect the rest of the sequence is not being downloaded for some reason. The end of sequence resulted in an error displayed as in the latest Debug log. I will try a different camera just to see if it’s something ASI533 specific.

[Mark Aston]

Just to confirm using a 314L, NINA works fine and saves files to HDD and NAS

[Mark Aston]

One final report for today - I tried a sequence of 1 frame, reset and then take a sequence of 1 frame again around 30 seconds later. This resulted in an error as well, and I attach the debug file.

[Mark Aston]

Just returned to this issue for another play. So, I updated to Nightly 1.11 (issue as of today) and still a problem with saving images in sequence to local PC drive. However, I then tried turning off the de-Bayer option in Settings and the sequence worked fine with images saved to HDD without issue. Hopefully this helps narrow any issue down.

[Stefan B]

That sounds odd. Can you also attach the logs of the 1.11 session, they are much more detailed and hopefully reveal the issue

[Dale Ghent]

Hi Mark, I was able to take a look at your logs and the failure in the imaging pipline with Debayering on is evident with the following: System.OutOfMemoryException

Basically, there is not enough memory available to NINA to debayer the image; a process that requires much more memory over monochrome. This is borne out by your discovery that turning debayering off solves the issue. Additionally, having unlinked stretch on further increases memory requirements as additional copies of each color channel must be made in memory.

It does appear that you are running a 32bit NINA, and this would be the likely cause. High pixel count sensors can push process memory requirements beyond the level which 32bit programs can allocate memory, and problems like this are the result. I can look at adding some notifications in this area to assist users in the future in realizing this particular problem.

In the meantime, I advise that you install the 64bit version of NINA.

[Mark Aston]

Thanks Both. I will upload the 1.11 log later just in case it’s of use, but I’ll have to re-do the sequence as I didn’t have it in Debug mode.

I think you might be right about it being a 32-bit issue, but there is still a puzzle in that my QHY8 runs with sequences in NINA without any issue (albeit it a few megapixel less image size), and Maxim DL V4 has no problems running a full sequence with the new ASI533MC that <with my plugin> completes sequences with dark & bias subtraction, flats correction, debayers and saves to a NAS HDD.

There’s one further piece of observed program behaviour that doesn’t potentially scan with it just being a 32-bit memory issue - When the first image in a sequence is acquired from the ASI533MC, the temp image is saved, the final named image is saved and NINA displays the debayered image (I can follow the workflow NINA displays whilst the net image is downloaded). It’s the 2nd image onwards that does not replicate this behaviour - note that the processing and final display of image does happen whilst the next frame is being acquired - that is different behaviour to Maxim and ASI Suite, both of which do not fail on sequences (the ASI suite also debayers each image…).

If it was just a memory issue per frame, surely the first frame would run out of memory if the subsequent ones do? Maxim seems to show the PC can hold multiple ASI frames in memory without issue, so is there anything in the way the ASI is downloaded and held in processing memory in NINA that could be an issue?

I noticed this afternoon, NINA appears to do some star detection/mapping as part of the process - is there anything memory intensive there that I can turn off?

The reason I was exploring using NINA is that Maxim V4 is getting old and I liked the look of NINA and how you are developing it proactively. Many thanks for your thoughts and help on this.

[Dale Ghent]

The QHY8 has a 6MP sensor, and the IMX533 is a 9MP sensor. It could very well be that 6MP worth of debayered image data just fits inside memory, and 9MP worth of data pushes it over the edge. Memory limits are a very binary thing - you either have enough or you don’t. It could be that you’re already operating close enough to the edge that the 9MP sensor data puts you over that threshold.

The memory issue is real, after all, it’s an out-of-memory exception that is being generated by the .NET framework during the debayering process. That’s the OS complaining; not NINA’s own assessment of the situation. I’m not sure what MDL does with its frame data, but NINA does make copies of it for various operations:

• Stretching creates a copy of the data for stretching and display of that stretched data
• Unlinked stretching creates one additional copy of each R, G, and B color channel so that each may be stretched individually
• A copy of the image data is made for the image statistics thread to complete
• If debayered HFR calculations are on, a copy of the debayered data is created for the autofocus analysis
• Of course, a copy of the undebayerd image data is used to write out a TIFF/FITS/XISF file

So, there are a lot of cases where copies of an image are made in NINA to do certain tasks, and a lot of these processes happen in parallel. Obviously, running NINA with color cameras tends to apply the largest amount of memory pressure due to debayering and unlinked stretching. This is why switches exist to deactivate those processes. As for how much memory NINA uses at any given moment, that is left largely up to the .NET garbage collection strategies.

My main question here is: is there a reason why you’re running a 32bit NINA? 32bit NIN is a vanishingly small minority of installations and is required only in the very occasional case of a person running an ancient 32bit-only PC. ASCOM drivers that are 32bit-only are quite the exception these days and workarounds exist to make them usable by a 64bit app.

[Mark Aston]

OK, thanks Dale. 32-bit is used for historic reasons - this is an observatory with limited resources.

My only comment is to re-emphasise the point that with all ‘switches’ on, the first frame acquired processes out fine and saves to disk. The subsequent frames do not. Following the logic of your description of process, the first image should not work if it is a per-frame memory issue.

I fully accept the point about 32-bit aging quickly, but I implore that you do not fall into the trap of making NINA memory inefficient just because it matters less with 64-bit memory addressing.

As you feel there is no underlying issue here, I’ll close the Issue.

Best regards

Mark

[Stefan B]

Mark i do get your concerns about the first image working but not the second one.

But this is a major design difference from N.I.N.A. to other applications. MaximDL and others are very sequential in their acquisition process. They take an image, download the image, process the image, save the image and only after that take a new image.

N.I.N.A. on the other hand takes an image, downloads an image and then starts to parallely take a new image, process the image and save the image all in parallel. So you will have not only one image in memory but multiple. This drastically improves performance, while having a bigger hit on memory. In addition to that, having a process run in x64 has already a more efficient memory management in general, as it will work completely different in the background of the .NET framework compared to an x86 process.

To reduce the memory footprint, disable unlinked stretch (this is by far the most memory intensive one) as well as debayering. I am also thinking about making these options completely unavailable for x86, as this doesn’t work well there.

[Dale Ghent]

The first frame being OK but subsequent ones failing does actually make sense. It really comes down to the memory management that Windows and the .NET framework impose. Like I said, you might be right on the precipice of the 32bit process memory limits and the first frame from either camera fits neatly into the available space. However with unlinked stretch and all the other processes going on inside the app, then we can have at times two frames' worth of data consuming memory in some cases, even if only for a brief moment. We can look at ways to make Windows be more aggressive about reclaiming memory in certain areas, but a more hands-on approach to memory management is something that Windows advises strongly against doing as it can have unintended consequences.

As far as 32bit goes, I think it would be correct to say that the 32bit platform is supported under a “best effort” status. 32bit-only Intel platforms ceased being commonly sold roughly a decade ago and the newest version of Windows does not support running on 32bit-only hardware. The writing is clearly on the wall and it’s time to sunset that architecture and not allow it to hold back progress that 64bit systems can handily accommodate. A lot of NINA’s speed and parallelism of its internal processes is made possible by embracing 64bit platforms. This does come at the cost of increased memory requirements, but such is life. Memory is there to be used, not conserved.