Remove overscan area while using QHY native driver

[Anne van Houwelingen]

Discussed in #support, but I do not see a formal issue, so I will post it here.
Please provide an option of removing the overscan area when using the native QHY driver.

[Dale Ghent]

I have some code for this in the works that I hope to have functioning in a few days. Since I do not have a QHY camera that has overscan, would you be willing to test it out?

[Anne van Houwelingen]

That is good news! Would be happy to test this.

[Dale Ghent]

Hello, Anne. I’m sorry it took a little while, but I had time recently and I think I have a solution for this. Are you still interested in testing this, and would you prefer a 32bit or 64bit NINA installation to try?

[Anne van Houwelingen]

Sure, Dale. Please send me a 64bit NINA installation.

[Dale Ghent]

Hi Anne,

You can retrieve a 64bit installer here: https://daleghent.com/misc/nina/NINASetupBundle64.exe

It is based on the current development branch of NINA, version 1.10.

In the camera control screen, where you select the camera to connect to, you will notice that the settings icon (the gears next to the drop-down box) will now be available to click on once you connect to a QHY camera. Clicking on that will produce a window where you may choose to include or exclude the overscan area in the image. The default is to not include it. Please try testing both options.

[Anne van Houwelingen]

Hello Dale,

I did the test. It works, but there is work to be done.

1. Choosing the native QHY driver (for the 168C) with the option no overscan gives an image with dimensions 4952x3288. This is exactly the dimension resulting from the no overscan option in the ASCOM driver.
2. Then I did not disconnect, but changed to overscan. When imaging this resulted in “Error retrieving image data from camera”
3. I disconnected the camera and connected again, chosing overscan. The equipment panel now says: 5040x3346 pixels. But when you image, the resulting image has no visible overscan and the dimensions from the statistics tab are 4952x3288.
4. I closed NINA and started new, but the results are as in 3.

I hope this gives enough clues to give an indication where to look in the code

Regards

Anne

‌

[Dale Ghent]

Hi Anne,

Thank you for trying this and providing the detailed feedback. I am feeling around in the dark on this feature since I do not have a camera that has an overscan area, so such details are helpful.

I reviewed the logic in light of your feeback and believe I have found the deficiency. If you could, please download an updated installer at the same URL above and give it another try? Thank you very much.

[Anne van Houwelingen]

Hi Dale,

I am aware that you must rely on what I report, so I try to be detailed. This time also with screenshots.

• When I connected the camera the default this time is overscan on (but this could be because I ended there in the previous version and it is stored in the profile?). The image dimensions in the settings were that of the non-overscan btw.
• I have found that changing between overscan and no overscan requires not only a settings change, but also a re-connect of the camera. Otherwise strange things happen (like the “error receiving image data from camera” msgs).
• Whatever I did. Settings, reconnect camera, restarting NINA, I could get the settings panel to report the right dimensions, but the image taken was always the one without the overscan area. See screenshots.
• I also noticed cascading corruption. With that I mean that when subsequently changing settings back an forth, I got NINA to crash and I got a corrupted image with a reported dimension of 1972x6560.

To me, it looks a lot like the previous version…..and in effect, the creation date of this particular version (from the original link, as instructed) is 12-09-2019 0.59. It is now 14-9-2019. Are you sure, I tested the latest version?

*

‌

[Dale Ghent]

Hi Anne,

When you start NINA, immediately go to Options → General and make sure that Log Level is set to Debug.

Once you have set that, then connect the camera. Try to take 1 exposure. Switch the overscan setting to the opposite of what it was and try to take an exposure again. When you have done what you could there will be a detailed log file in %LOCALAPPDATA%\NINA\Logs. You can copy and paste that location into Windows file explorer to get there. Could you please attach the log file to this ticket?

Also, the need to reconnect the camera is strange! If this is a requirement, is not documented by QHY. I hope that is not the case because it increases the complexity greatly. I will verify with them.

‌

[Anne van Houwelingen]

Hi Dale,

See attachment.

The action is all in the first 5 minutes…

[Dale Ghent]

It seems that you were still using the original version that I posted. There are some log messages that I added in the second version that should be there, but are not. But aside from that, the log messages reveal a few things, and it looks like we are being bitten by a bug in the QHY SDK that might be specific to the 168C.

If you also look at the log file at lines 21, 29, and 33, you will note the following messages, trimmed here for brevity:

QHYCCD: Chip Info: ChipX=24.192mm, ChipY=16.0608mm, ImageX=5040, ImageY=3346, PixelX=4.8um, PixelY=4.8um, bpp=16
QHYCCD: Effective Area: StartX=4, StartY=58, SizeX=4952, SizeY=3288
QHYCCD: Overscan Area: StartX=0, StartY=0, SizeX=0, SizeY=0

These messages are logged when we connect to the camera and discover its capabilities.

The first line are the results of when we ask the camera for its sensor properties. It gives us the sensor’s physical dimensions, pixel array dimensions, pixel size, and supported bits per pixel output. So it tells us that the maximum resolution of the chip is 5040x3346.

The second line logs the results of when we ask the camera for its “Effective Area” - the area of the sensor that is used for exposing an image and excludes any overscan area. Here it tells us that resolution is 4952x3288. It also tells us that the origin pixel of the 2-dimensional array is 4,58.

The third line is important. It is supposed to tell us the origin pixel, and the X and Y size of the overscan area. Here, is where we start getting confused - it is all zeros! The dimensions of the Effective Area plus the dimensions of the Overscan Area are supposed to match the dimensions given to us in the first line, but because the Overscan Area is said to not exist, we run into problems. This is why when you try to expose an image with Include Overscan = On, it tells you that the expected dimensions of the image we retrieved from the camera do not match what we calculated.

But this is not all that is going on with the camera.

I have some experience with other models, and there are sometimes SDK bugs that pop up with them that I have gotten QHY to fix. But the 168C seems to still have them. The first one is a fake gain bug, where the SDK claims that we can adjust the gain of the camera, but it always returns a gain of 1 no matter what we set it to. The second issue is that the SDK claims that we can change the USB speed of the camera, but I test to see if we can really do that, and it does not seem to be the case.

I will report all these issues to QHY and see what they say.

[Anne van Houwelingen]

Thanks for the detailed write-up, Dale.

For completeness sake, I did these tests with QHY5IIISeriesDriver190327.

Interesting stuff. Up untill now, I have always used the ASCOM driver. In another acquisition program that does not support native qhy. The ascom driver has another trick…it reports the camera as being monochrome. So, in that other program there simply is a “color” checkbox for the user to indicate that this is a color camera and there is a bayer pattern to choose from (in the case of 168C it is RGGB). Of course it is a kind of patch on top of the ASCOM driver. But it is fairly simple and it works.

[Dale Ghent]

Hi Anne,

Another try here. I also named the installer files so that they can be easily identified. the “os3” part of the name means “Overscan 3”, for our third attempt

32 bit Installer: https://daleghent.com/misc/nina/NINASetupBundle-os3-32.exe

64 bit Installer: https://daleghent.com/misc/nina/NINASetupBundle-os3-64.exe

If you can try the one of your choice and let me know how it goes with the native driver, that would be great. As always, logs would be helpful.

[Anne van Houwelingen]

Hi Dale,

Attached 2 logs from OS3. The first one is the one after the install and the second one is after a necessary restart of NINA (as detailed below).

I had a quick look through the log and there is at least one interesting phenomenon. At exposuretime, the toggle IncludeOverscan does not seem to have any influence on the image area.

‌

[2019-09-16T11:44:04]    [DEBUG]     [MemberName] StartExposure
[2019-09-16T11:44:04]    [DEBUG]     [FileName] C:\Users\daleg\source\repos\nina\NINA\Model\MyCamera\QHYCamera.cs
[2019-09-16T11:44:04]    [DEBUG]     [Message] QHYCCD: Setting image resolution: startx=4, starty=58, sizex=4952, sizey=3288, IncludeOverscan=False

[2019-09-16T11:45:27]    [DEBUG]     [MemberName] StartExposure
[2019-09-16T11:45:27]    [DEBUG]     [FileName] C:\Users\daleg\source\repos\nina\NINA\Model\MyCamera\QHYCamera.cs
[2019-09-16T11:45:27]    [DEBUG]     [Message] QHYCCD: Setting image resolution: startx=4, starty=58, sizex=4952, sizey=3288, IncludeOverscan=True

This is what happened:

I opened NINA. When connected, the camera was set to No Overscan, with reported image dimensions of 4952x3288. I took an image. Looks fine (apart from the annoying beep after the exposure…)

I went back to setting and changed to inlude overscan. Hit OK. The dimensions in the settings display do not change after that. I image. Gives an OK image.

I disconnect and reconnect. Went back to the settings screen and now it reports an image size of 4972x6568, wich of course is wrong. I go back to imaging and take an image. Now I get the “error retrieving etc message”.

I close NINA and connect again (second log file). And then it behaves again like when I originally started. Images are fine (no overscan), but it will not go into overscan mode….

As for the USB speed issue. I do not know if this is of influence or not, but I operate this camera on a USB2 port. The whole cabling chain is USB 3, but after replacing cables, powered hubs etc, it turned out that the USB3 port on this laptop does not play nice with the camera (intermittent garbled images). I only found out when I connected the camera to the USB3 port of a brand new laptop….. USB2 is a bit slower in downloading the image, but that is not really a problem.

[Dale Ghent]

Okay, after laying in bed and thinking about this some more, I came to a realization: that perhaps the overscan area size is not reported unless we have already chose to ignore it and that only when we ignore overscan, its size will be revealed. The QHY documentation makes no clarifications on when this function will return data, so I’m trying to figure this out.

In this new version, I have made no assumptions: We ask what the overscan area is no matter what, each time we change the option. I am no longer assuming that I can ask only once when we first connect to the camera. I also have the origin pixel (x, y) of the image dynamically calculated, just in case it is also in the same situation.

IF this logic also fails, then I am at a loss and will ask QHY to look at it again… in my opinion, if I ask the camera what its overscan area size and origin pixel is, it should always tell me without any silly conditions. Here is “os4”:

32 bit: https://daleghent.com/misc/nina/NINASetupBundle-os4-32.exe

64 bit: https://daleghent.com/misc/nina/NINASetupBundle-os4-64.exe

[Anne van Houwelingen]

Hi Dale,

No luck, I’m afraid. This feels a bit like night time flying without the instrumentation….

I have attached the logs and also two fits file. The overscan is a normal image (with overscan selected). The image is OK, but the image dimensions are that without the overscan.

No_Overscan has a white maxed out band at the bottom. Approx 60 pixels high. There also is a few pixels on the right hand side that are very dark (but not zero).

If I were you, I would stop the frustration and get QHY to give clarification on how it should work (if it actually works that way is to be demonstrated).

Thanks for your efforts, really appreciated!

‌

Anne

[Dale Ghent]

Hi Anne, I just looked at the latest logs you posted above and it seems that the NINA.exe you are running is not the one delivered by the “os4” installer. I can tell by what’s in the logs. Can you make sure that you are installing and executing the correct one?

[Anne van Houwelingen]

Hi Dale,

That is odd. I assumed the installer would overwrite the previous installation? Apparently not..this time. The version I tested above was 6 hours older than OS4. So, I totally de-installed NINA first and installed OS4.

The behaviour is different this time. The image statistics and setting information still always contain the no-overscan dimensions, but the images now contain overscan. Only at different positions (looks mirrored?). I will attach the fits files and the log….

[Dale Ghent]

Ok, the great news here is that my bedtime hunch was right: The Overscan area on this camera is reported only after we set the camera to ignore overscan. This explains why the function I call to get the overscan area dimensions returns no data (all zeros) instead of origin pixel and x and y size dimensions. This fact is completely missing from the SDK documentation. Frustrating, but that mystery is now solved.

The remaining issue is properly using this data to factor in the new dimensions; and I believe I know why it is not working. I will work on a new version tonight after I get home from work.

[Anne van Houwelingen]

Hi Dale,

Very good. More insight.

I realised that I actually have 2 other acquisition packages that use native QHY drivers. One is QHYCCD EZ-QT. It supports overscan and no overscan, but frankly, I could not detect a difference between the two resulting images.

The other one does not even give me the option of overscan.

Frankly speaking, although QHYCCD EZ-QT and PixInsight support overscan calibration (to correct for BIAS drift), the application I use for calibration (APP) sees that piece of information just as something to crop.

I have never ever used overscan and I think that 99,99% of NINA users ever will. But that is my very personal opinion. We started out with only overscan. If the endresult would be only no-overscan…I am happy.

[Dale Ghent]

You have a good point, and getting this right seems to be a uphill battle with the sparse documentation coupled with testing having to be done by some very patient proxies. I’ve decided to ditch overscan support altogether and now we just use the “Effective Area” dimensions of the sensor, which is the area of the sensor that excludes overscan. I have removed the setup window and the option is no more. If someone wants Overscan support later, maybe I can get my hands on a suitable camera as well. This is also holding up an important project - native support for QHY cameras that have integrated filter wheels, or filter wheels that are connected using the 4-pin cable.

Here are new installers:

32 bit: https://daleghent.com/misc/nina/NINASetupBundle-os5-32.exe

64 bit: https://daleghent.com/misc/nina/NINASetupBundle-os5-64.exe

‌

[Anne van Houwelingen]

Hi Dale,

Looks fine with me. The log file is for the following:

• Connecting the camera
• Cooling it to -10C
• Setting gain to 1
• Setting offset to 35 (these are my broadband settings)
• Taking a picture (attached as well)
• Warming the camera to +10C
• Shutting off the cooler
• Disconnect from camera.

Of course it would be fine if I could select gain/offset combinations from a drop down box with my own favourites (I have 2 with this camera 1, 35 for Broadband and 11, 60 for narrowband.), but that is not the subject of this particular issue.

‌

‌

[Dale Ghent]

Hi, Anne. Thank you for your feedback. I’ve tidied up the patch and have submitted it for review in pull request #321

This will be included in the next 1.10 point release. Once I have native filter wheel working, I will see about backporting the lot to 1.9 as a non-critical hot fix.