mthread transition in mlogger - torture test failures

[dd1]

I see this on the fgdmini, again some kind of race condition between client removal and transition calls? K.O.

Wed Oct 23 10:09:49 2013 [Logger,INFO] Run #46663 stopped Wed Oct 23 10:09:48 2013 [Logger,ERROR] [midas.c:3589:cm_transition_call,ERROR] cannot connect to client "fefgdmscb01" on host ladd10.triumf.ca, port 44175, status 503 Wed Oct 23 10:09:48 2013 [Logger,ERROR] [midas.c:9093:rpc_client_connect,ERROR] timeout on receive remote computer info: Wed Oct 23 10:09:48 2013 [Logger,INFO] Client 'fefgdmscb01' on 'ODB' removed by cm_cleanup (idle 70.5s,TO 2s) Wed Oct 23 10:09:48 2013 [Logger,INFO] Client 'fefgdmscb01' on 'SYSMSG' removed by cm_cleanup (idle 70.5s,TO 2s) Wed Oct 23 10:09:48 2013 [Logger,ERROR] [midas.c:3589:cm_transition_call,ERROR] cannot connect to client "" on host , port 0, status 503 Wed Oct 23 10:09:48 2013 [Logger,ERROR] [midas.c:9048:rpc_client_connect,ERROR] cannot lookup host name '' Wed Oct 23 10:09:48 2013 [Logger,TALK] stopping run after 30 seconds

[dd1]

10:20:20 [Logger,TALK] starting new run 10:20:20 [Logger,ERROR] [midas.c:9048:rpc_client_connect,ERROR] cannot lookup host name '' 10:20:20 [Logger,ERROR] [midas.c:3589:cm_transition_call,ERROR] cannot connect to client "" on host , port 0, status 503 10:20:20 [Logger,ERROR] [midas.c:4341:cm_transition,ERROR] Could not start a run: cm_transition() status 503, message 'Cannot connect to client ''' 10:20:20 [Logger,ERROR] [midas.c:9048:rpc_client_connect,ERROR] cannot lookup host name '2013-10-23 10:20:17.0RhQ' 10:20:20 [Logger,ERROR] [midas.c:3589:cm_transition_call,ERROR] cannot connect to client "" on host 2013-10-23 10:20:17.0RhQ, port 0, statu

[dd1]

notice the junk text in "hostname". K.O.

[dd1]

Wed Oct 23 21:51:02 2013 [Logger,ERROR] [midas.c:10342:rpc_client_call,ERROR] rpc timeout after 21 sec, routine = "rc_transition", host = "ladd10.triumf.ca", connection closed Wed Oct 23 21:50:41 2013 [Logger,ERROR] [midas.c:11784:rpc_execute,ERROR] Invalid rpc ID (1)

[dd1]

Fri Oct 25 04:27:24 2013 [Logger,INFO] Run #48125 start aborted Fri Oct 25 04:27:22 2013 [Logger,ERROR] [midas.c:4347:cm_transition,ERROR] Could not start a run: cm_transition() status 504, message '(null)' Fri Oct 25 04:27:22 2013 [Logger,ERROR] [midas.c:10344:rpc_client_call,ERROR] rpc timeout after 21 sec, routine = "rc_transition", host = "ladd10.triumf.ca", connection closed Fri Oct 25 04:27:02 2013 [Logger,INFO] tr_start status SUCCESS Fri Oct 25 04:27:02 2013 [Logger,INFO] tr_start, start_requested 0, auto_restart 0 Fri Oct 25 04:27:01 2013 [fefgdwiener01,ERROR] [midas.c:11784:rpc_execute,ERROR] Invalid rpc ID (1) Fri Oct 25 04:27:01 2013 [Logger,TALK] starting new run Fri Oct 25 04:27:01 2013 [Logger,INFO] start_the_run, start_requested 0, auto_restart 1382700420 Fri Oct 25 04:27:00 2013 [Logger,INFO] Run #48124 stopped

[dd1]

rpc_client_connect() is not thread-safe - it uses the global _net_send_buffer. K.O.

[dd1]

current midas passes transition torture tests - almost 7 days running on fgdmini, restarting runs every 10 minutes from the mlogger. no errors. K.O.