1. Olivier Sallou Avatar Olivier Sallou
  2. Untitled project
  3. go-docker
  4. Issues

Errors with killing jobs

Issue #75 closed
IT Expert created an issue

Hi, Olivier. Have issue with killing jobs. Time to time have god-watcher errors like this:

2018-06-05 10:00:00,868 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 10:01:31,374 WARNI [godocker-watcher][MainThread] No exit code: 4266
2018-06-05 10:02:24,822 WARNI [godocker-watcher][MainThread] No exit code: 4277
2018-06-05 10:25:35,022 WARNI [godocker-watcher][MainThread] No exit code: 4290
2018-06-05 10:25:35,087 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 10:25:35,455 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 10:28:33,734 WARNI [godocker-watcher][MainThread] No exit code: 4302
2018-06-05 10:28:33,761 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 10:28:33,771 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 10:32:18,455 WARNI [godocker-watcher][MainThread] No exit code: 4316
2018-06-05 10:35:26,762 WARNI [godocker-watcher][MainThread] No exit code: 4327
2018-06-05 10:39:17,272 WARNI [godocker-watcher][MainThread] No exit code: 4341
2018-06-05 10:56:31,316 WARNI [godocker-watcher][MainThread] No exit code: 4357
2018-06-05 10:57:07,426 WARNI [godocker-watcher][MainThread] No exit code: 4369
2018-06-05 10:57:07,470 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 10:57:07,485 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:05:48,156 WARNI [godocker-watcher][MainThread] No exit code: 4382
2018-06-05 11:05:48,245 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 11:05:48,248 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:09:33,035 WARNI [godocker-watcher][MainThread] No exit code: 4393
2018-06-05 11:09:33,062 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 11:09:33,064 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:09:56,224 WARNI [godocker-watcher][MainThread] No exit code: 4404
2018-06-05 11:09:56,246 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 11:09:56,276 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:10:41,531 WARNI [godocker-watcher][MainThread] No exit code: 4425
2018-06-05 11:10:54,861 WARNI [godocker-watcher][MainThread] No exit code: 4415
2018-06-05 11:10:54,880 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 11:10:54,897 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:13:47,330 WARNI [godocker-watcher][MainThread] No exit code: 4436
2018-06-05 11:14:03,130 WARNI [godocker-watcher][MainThread] No exit code: 4447
2018-06-05 11:14:03,155 ERROR [godocker-watcher][MainThread] Watcher:None:argument of type 'NoneType' is not iterable
2018-06-05 11:14:03,172 ERROR [godocker-watcher][MainThread] Traceback (most recent call last):
  File "/opt/go-docker/godocker/godwatcher.py", line 1146, in run
    self.manage_tasks()
  File "/opt/go-docker/godocker/godwatcher.py", line 1109, in manage_tasks
    self.check_running_jobs()
  File "/opt/go-docker/godocker/godwatcher.py", line 999, in check_running_jobs
    if 'Node' in task['container']['meta'] and 'Name' in task['container']['meta']['Node']:
TypeError: argument of type 'NoneType' is not iterable

2018-06-05 11:15:03,665 WARNI [godocker-watcher][MainThread] No exit code: 4458

The result is job can continue working, but godocker think that it killed.

Comments (6)

  1. Olivier Sallou repo owner

    looks like task['container']['meta'] is not set. Do you try to kill job once scheduled and running?

  2. Olivier Sallou repo owner

    meta is set when scheduling the task. Seems you are in case where job is not yet scheduled and trying yo kill it

  5. IT Expert reporter

    I would like to investigate this, but it seems, errors haven't dependency of: does it stopped by itself or were killed by user.

    Also I found that users use job arrays. And may be in that case parent job some task['container'] fields is not set, so this is leads to some of this errors.

  6. Olivier Sallou repo owner

    parent job indeed does not have container info (it is a virtual jobà. In the fix, I "bypass" in case of error

  7. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
closed
Votes
0
Watchers
1