Shutter balky to pair after rotator is power-cycled

[ron crouch]

This is beta 3.0. Normally shutter and rotator are left on 24/7. In this state communications are absolutely solid. Have been solid for weeks. If, however, the rotator is powered off for the day on startup shutter does not pair. I watched in a serial monitor for 20 minutes and XB->Online never shows. If I simply power off shutter for a few seconds and then power back on the rotator and shutter pair instantly. This is repeatable. It is also a on-issue for me as I just leave rotator on and if I do a power cycle I will cycle both.

After many confirmations below I checked a couple of things.. Summary is.. if rotator and shutter connected and rotator turned off they will not pair again until shutter is power cycled. If both are off it does not matter in which order they are powered up. Successful pairing occurs if shutter comes on first or rotator comes on first. Seems to confirm that once shutter is paired it is dutifully holding onto the address. This is a very good thing for sites with multiple domes!

[Ron Kramer]

I just posted this as well. Let me power off and on and confirm as well… brb
Negative. powering down the rotator (it still had USB power) did not allow them to connect upon power up. I tried 3 times.

OLD software did this so it became a habit to leave it on 24.7. Solved the problem with the old software. (unless a power down by mistake)
I hoped the new software would solve this XB connection issue. = (

[ron crouch]

I think the issue is the shutter is holding the old address. Thanks Ron!!

[Ron Kramer]

To confirm… Calling the js - I got a error here. (not online)

[ron crouch]

Also!!  remember it is a known issue with the beta that if you do something like play around and connect and client disconnect.. a few times.. each time a ASCOM NexDome server process will remain running.  Eventually it will croak..  My Moonlight focuser ASCOM driver does the same thing sometimes..

So.. if you disconnect find the ASCOM Nexdome server with Task Manager and kill it.  Tim is aware of the issue.. Ron

[Ron Kramer]

It’s been on all night. Still won’t connect. I missed out on imaging last night due to a non open condition.
when it did this before it would take anywhere from 12 hours to 4 days. I guess it stays on 24/7 again…

[ron crouch]

YUP Just leave rotator 12V on. That way the delicate USB stuff can be off during the day. If you do turn it off re-connection is not at all witchy. Simply pull shutter power for a few seconds then reconnect all with rotator on. Connects instantly. 

Imagine you had two domes side by side..  The old Transparent mode could never work but the new communications should be fine.  

Best.

At this point I've had 20 flawless nights in a row with pretty much the beta or it's alpha. Last night included..

[Ron Kramer]

I can’t quite make out what you’re saying. (my nuc is on all the time)
I am connected today. (48 hours later) however, yesterday I was out there and for grins I pulled the shutter power and put it back in since I was in teh dome anyway (likely that solved it). The real problem would be if this happens for people with very remote operations. Unable to get to the shutter power connector.

Are you powering down? 20 flawless nights (powering down?) And you’re saying should it do this again after a power down, I can pulled and replace shutter power plug and it will then boot? hmmm is there any way to program in an option for a reset/reboot of the shutter? So it can be rebooted remotely?

[ron crouch]

I shutdown the NUC every day when it's hot which here means May -> Oct.  In winter I leave it on and in that mode ACP will run things 24/7 with zero input from me other than to keep the targets queue full which is done through ACP's web-server. The rotator USB goes directly to the NUC so when NUC goes down so does the usb power to rotator but I leave the 12V to rotator on which keeps it totally alive and talking to shutter.

In the NUC bios it's set to boot on power on (KEY item!! so I can be in Japan and run things which I have done a couple of times) so after I shutdown I turn off the NUC power with web power switch... Turning NUC power back on boots it then I access from VNC.  I almost NEVER physically go into the dome though it's in the backyard.

Also on the web power switch as separate items are a 12V for USB hub, the 12V for camera (I used to have one 12V for both but it's useful to have them separate); the nexdome rotator which now just stays on is on it's own web power switch as well. I have a 12" fan I can control with the web switch and of course there is the 15V supply I use for the Mach 1 mount.

ACP startup script powers on camera, mount, 12V for hub (which also powers moonlight DRO focuser and if the newt is on mount the mirror fan) in that order. If I have a refractor on the mount it also powers dew which is an isolated 12V power supply.. No dew control needed for newtonian.

Once all is powered up ACP connects the mount (which has Nexdome as Dome) which gets Mach 1 and Dome connected. Then it launches Maxim 6 and Focusmax and connects those devices.. turns on cooler.  Done.  ACP shutdown just does the reverse.

With this setup I've gone as long as 3 months without ever stepping into the dome.  2.11 was completely robust for me but I much prefer TA 3.0 because it's much smoother and faster..  Only thing differ is I must leave 12V to rotator on.

NEVER lost the shutter EVER with either 2.11 or TA 3 once it's connected.. This goes back many months for me.

So today in dome all is off except the 12V for rotator and the little 12" fan.  After 6:30 when shade hits the dome if it will be likely to be clear I'll power up NUC, launch ACP and in scheduler check the box to run the dispatcher.  That's it period done!  If things cloud ACP handles that perfectly.. will just close up and wait.. if it never clears it runs shutdown and end of astronomical twilight.  In a cool time of year I'd do nothing at all and it would fire up again next evening and if clear open up if not clear just wait..  I do turn off NUC in summer because I do not want to ask for trouble... Can get HOT here!  It is located in a weather proof actual PC case along with web switch and power supplies and cooling fans.  It has stayed up just fine 24/7 with dome in high 90's but unless it's gonna stay in 80's I do shutdown the NUC.

Now regarding the shutter...  It would be simple to power cycle remotely with a little pi piggy-backed running a relay but I've never bothered.  I do have a ~$40 Odroid inside doing stuff that could easily do that if I add a wifi dongle.. I also once bought a digital timer switch which could hook into the 12V on shutter now and do something like cycle the shutter power .. off for a couple of minutes then back on.. every day at like 3pm?  The little timer draws it's power from the 12V but an internal button cell would power it for about 6 months or so..The timer is a $4 solution to cycle shutter but why not simply leave the rotator 12V on?

Best, Ron

[Ron Kramer]

Sounds like we have nearly identical setups - but what is ACP?

[ron crouch]

Oh ACP is Bob Denny’s totally awesome imaging automation client. It’s an amazing thing and used by quite a number of professional observatories. Comes in two parts.. ACP which runs the hardware and The Scheduler which if on top will schedule all activities. Look at .. http://acp.dc3.com/index2.html

‌

Not cheap but pretty near perfection plus Bob’s customer support in legendary. He has a big set of utube videos explaining user features. I used ACP alone for about 2 years before adding Scheduler also about 3 years ago now.

‌

Wow your setup looks fabulous!!! I did not get a side bin so it’s pretty full. Lumpy but functional.

Ron

[George Silvis]

Having problems getting the rotator connect to the shutter. Code and drivers are all from the Beta package.

Watching from putty the rotator keeps calling Start, WaitAT, Config, Detect and the shutter just says WaitAT (And BV battery signals)

Also, the NexDome ASCOM Server window is no longer visible. It is in the task list and manager. But I can’t switch to it. I have seen it earlier in my struggles but now it is elusive. I have rebooted and that did not help.

[ron crouch]

Try this. with rotator on unplug shutter for about 10 secs then replug. Watch in terminal and you should soon see it Online. After or before that. VERY important! Look in Task manager - locate and kill all instances of NexDome ASCOM whatever… There could be several if you tried over and over.. Or just reboot PC.. Once you see in terminal it’s Online you will be good to go.

[George Silvis]

Thanks for the help here, Ron. But no joy. The terminal window to the rotator just repeats its quest for a connection (Start, Wait, Config, Detect). When I plug a usb into the shutter and connect via another terminal to it all I see is XB->WaitAT ever couple seconds. Wait a moment: now its kicking out “S0”, about 5 for every WaitAT.

XBee board issue? Should I dismount the shutter motor unit and poke around inside?

[ron crouch]

Hi George. Tim will be a good help when he can get to this. In the meantime only thing in my mind is XBee firmware version. If you while you have usb on shutter use the same firmware uploader you did to install the beta in exactly the same way but rather than saying --HexFile followed by shuttername.hex load the XBeeFactoryReset.hex. After you load in order for it run run you must connect to shutter in a terminal and just watch.. Shortly after you connect it will print a bunch of text part of which is the XBee firmware version. I do not recall what is right or wrong but write it down for use later.

Reload NexDomeShutter.hex and look again.. If no luck look at next paragraph..

In fact in order to return to version 2.11 which I did a number of times during testing the first step is to do that exact item - run XBeeFactoryReset - on both rotator and shutter. After that install the 2.1 firmware on shutter and then rotator. Finsh by connecting to rotator in terminal and send #X to initiallize XBees in old way. At that point 2.11 should be good to go.

‌

Dang,

‌

Ron

[George Silvis]

• I used the arduino ide to load Blink into the shutter, just to make sure the proper bootloader was in place.
• Then I use the FirmwareUpdater to load the XBee hex file.
• Opened a terminal and found:
  XBee Factory Reset Sketch started
  XB->Startupad
  Waiting for XBee to initialize
  XB->Wait for AT Command Mode
  Attempting to put XBee into AT Command Mode...
  XBee did not enter command mode, try again.
  XB->Wait for AT Command Mode
  and it keeps up the last 3 lines forever.
• So, I think the XBee is on the fritz. Yes?

Found the problem! The XBee was not plugged into the Arduino properly: It was plugged in one-off; the pin 1 was hanging over the edge of the connector.

Reset the XBee and tried again. Nothing. I happened to have another XBee unit on the bench. Success! The miss plugging must have the damaged the XBee unit.

Careful inspection through the hole where the usb cable connects to the shutter might reveal this XBee plug in issue. And it is possible to remove the shutter assembly case without dismounting the whole thing.

Thx
George

[ron crouch]

I’m no expert (far from it!) but I would agree.

[Ron Kramer]

I see is still a issue with beta.9. I installed for someone in the group today and hard the same loop issue. I got it to connect after a reset and it dropped a bit later. Then connected again. It was quite spotty. Because it dropped… I wondered if his fairly new system that hasn’t really worked yet might have a bad or wrong xbee?

[ron crouch]

It is probably useful to load the XBeeReset.hex. Then connect in terminal and watch it run. The firmware version of the XBee will be reported. I know some firmware levels are not compatible but do not know the particulars. I did this several times during development getting back to 2.11. Should be run on both rotator and shutter. At least another data point.

‌

Best,

Ron C

[Ron Kramer]

how do I use the xbee.hex I see the file... but the other hex files I uploaded to each adruino board. How to use the xbee.hex?

[Ron Kramer]

At the end is says run on both - so I upload it to each instead of the firmware? - then install the firmware again after I run the xbee/hex?

[Tim Long]

@George Silvis

Watching from putty the rotator keeps calling Start, WaitAT, Config, Detect and the shutter just says WaitAT (And BV battery signals)

That would indicate that the firmware is unable to put the XBee into “command mode”. Essentially, that’s “game over”.

I’ve noticed that the XBees are very unreliable in that respect. They are supposed to work like Hayes modems, you wait 1 second, send “+++” and wait another second, and the device is supposed to respond “OK” and then you're in AT command mode. Well sometimes they’ll enter command mode, and sometimes they won’t. It typically takes 4 or 5 attempts and sometimes they simply refuse to enter command mode at all. There doesn’t seem to be any logic as to why or when. The only way you can fix that is to power cycle the unit. Unfortunately the Arduino doesn’t even connect up the XBee’s reset pin so there’s no way for the firmware to do a hard restart on it.

[Tim Long]

Issue #30 was marked as a duplicate of this issue.

[Ron Kramer]

Babak is sending Sylvain another Xbee - I got this to connect but 2 minutes later it’s searching for it again. Since Sylvain doesn’t speak english - I explained to Babak and he’s sending him a xbee. Sylvain also had a fire in the shutter from the power switch I think and repaired the burnt wire but may have fried the xbee. We’ll see.

[Tim Long]

@Ron Kramer Yeah I found the same thing. There was definitely something “not right” with it when I tried. It had a very flaky connection.

[Tim Long]

This is most likely a hardware issue due to the XBee's reluctance to enter AT command mode. Resolved by power-cycling the unit, not much to be done in firmware unfortunately as there's no reset pin or any way to do a hard reset on the XBee, so it pretty much has to be power-cycled.