Memory Leak in Advanced Sequencer

[Nico Gärtner]

Hi,

‌

I am experiencing a memory leak in advanced sequencer on my observatory PC when switching between two RDP (Windows Remote Desktop) clients (Mobile Phone and Laptop) with different screen resolutions.

To reproduce:

• Install NINA 2.0 Beta 19 (or use existing installation)
• Use plain “default” profile

• Create a new BIG Advanced Sequence

  • add Startup and End from Template
  • add three basic sequence targets

Stay on the advanced sequence screen.
Now switch back and forth between devices via RDP , while staying in the advanced sequencer - every switch will make the memory consumption grow - the grow rate relates to the complexity of the sequence.

It also takes longer and longer to adjust the screen - up to the point where the application is basically unusable.

‌

[Linwood Ferguson]

I can confirm this. I put a large-ish sequence up (three targets with a typical DSO set for each).

I switched back and forth quite a bit with the imaging pane up with no symptoms, but when I shifted to the advanced sequence display, it very quickly degraded. After 3 round trips I was seeing 800+mb memory (up from around 300) and 14 seconds of complete hang. I did not see the sort of un-filled window as above, but the NINA window does go unresponsive and hung on CPU for one processor (it does appear to be a single thread, as it shows up on my quad core as a steady(ish) 25%.

It looks to me like more than a memory leak, because the CPU increases continually with each swap; it is like it is building an ever more complex structure of some sort it must process each time, but not disposing of a prior (or something like that).

I see none of this when on the imaging pane even with the advanced sequencer mini-window displayed.

[Stefan B]

It seems like the RDP connection will cause a rerender of each control, which causes the controls to be recreated, but it seems that the old control is not discarded properly.

We will investigate why this happens, but it is quite difficult to track, as it seems to root from UI itself and not the databound structures in the background. The problem is also existing without RDP when loading in the same complex sequence into the sequencer again and again, as the previous controls are still somewhere in the background and not garbage collected.

I recommend to move to a different tab while using the remote connection in the meantime.

[Nico Gärtner]

Hi Stefan,

I am afraid not staying on the sequencer tab does not help.

I loaded a big sequence into sequencer while being connected from the laptop and then switched to imaging tab.

Then I connected from mobile, then again from the laptop while staying on the imaging tab.

When I switch now to the sequencer tab, the screen freezes for some seconds and the memory consumption doubles.

The more often I switch between the machines the more the memory increases when I enter the sequence tab - until the point NINA gets completely unresponsive.

Best Regards,

Nico

‌

‌

‌

[Stefan B]

Yes of course if you continue to switch back and forth between the tabs you will run into the same problem.
Recommendation i was referring to is setting up the sequence and then staying away from the sequence tab, as the mini sequencer gives you the status too.

[Linwood Ferguson]

As a workaround you might try VNC. I don’t think it changes the resolution (emphasis on thinks, I know it handles it all differently). It might avoid the issue until it can be fixed. I keep TightVNC running as a server all the time, mostly in case RDP goes south .

[Stefan B]

I have worked the last two days exclusively on this problem. It’s quite difficult to trace all the relevant places that cause this particular problem.

However I have made good progress in understanding where this problem comes from and have already some successful changes that will guard against this. While the problem is not completely gone in my current code branch, it is a lot better.

[Stefan B]

Here is a test sample. The top row is the current Beta Build and the bottom row is the version where I have applied some changes. Boths are using the exact same sequence consisting of three targets.
For this test I was staying on the sequencer page and connected to the machine via RDP a dozen of times. Maybe 20-30 times. The current Beta Build was getting much slower and it was taking almost a minute to get back to being responsive, where as the changed build was up again in a few seconds. There is still some issue present, as the memory is still slowly rising on the changed version, but it is a lot more stable already.

‌

‌

[Linwood Ferguson]

Any idea why the CPU usage seems disproportionally large compared to memory increase? That is what was puzzling when I saw it, the memory was significant, but the CPU was huge.

[Stefan B]

From my understanding (and it is hard to pin down exactly) the sequence page gets recreated when connecting via rdp, but each time there were old references left that had to be computed too. So each time you connected again this workload doubled and it was an exponential growth. I hope my adjustments will make this problem less severe, but i will also keep monitoring it and trying to find the remaining puzzle pieces.

[Linwood Ferguson]

Yeah, that makes sense, it just seems like the growth of computational requirements was faster than the growth of memory. But maybe not, that’s just a “feel” i did not try to measure it. In my case my sequences are fairly short, so it doesn’t become debilitating, though I think I was asking about a night when it all hung up while editing, and in retrospect I bet it was this cause. A restart of NINA fixed it entirely.

Anyway, thanks for all the effort chasing it.

[Nico Gärtner]

Thanks Stefan for taking the time to look into this over Christmas Holidays.

Your dedication and hard work is highly appreciated!

[Stefan B]

The improvements are available in Beta 20

[Nico Gärtner]

Hi Stefan,

for my (rather extreme) sequence things have definitely improved! Thanks a lot.

I unfortunately still loose >100MB memory per switch between devices and the NINA still freezes > 10secs.

It will take much more time to make NINA completely unusable now, but on machines without much RAM there’s still some chance that NINA will run in an out of memory situation in the middle of a sequnce.

Looking forward to see more improvements, as I tend to pack lots of targets into a sequence for the rare clear nights that we have now - I usually work with loooong sequences

‌

[Stefan B]

Yea the problem isn’t completely gone, but it is improved by quite a bit. This is however scratching on limitations of the framework, so I’m not too sure how much more we can do there.

[Stefan B]

Changing priority as the problem is not as bad anymore.

[Stefan B]

Removing version: 2.0 Beta (automated comment)

[Stefan B]

Please check with the latest nightly builds if there are still problems. The sequencer UI is utilizing virtualization now and should be much less memory intensive.