get_memory_usage is possibly unsafe on abel

Issue #19 resolved
Martin Sandve Alnæs created an issue

Since it spawns a process (calls "ps ...") it depends on existence and correct behaviour of ps without corruption of the mpi process which there are warnings about.

I suggest we disable it by default to be on the safe side. The cluster job system reports memory usage anyway.

I got this error at the end of a couple of simulations (most are fine):

Timestep 10000 finished (t=1.00e+00, 100.0%) in 9.4s (solve: 4.0s). Time spent: 11h 55m 39s Time remaining:   0s
Traceback (most recent call last):
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/case_AugustForwardProblem.py", line 59, in <module>
Traceback (most recent call last):
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/case_AugustForwardProblem.py", line 59, in <module>
    sweep.run(problem_configurations)
    sweep.run(problem_configurations)
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 90, in run
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 90, in run
    sys.exit(main(problem_configurations, sys.argv[1:]))
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 86, in main
    namespace = solve_problem(problem)
    sys.exit(main(problem_configurations, sys.argv[1:]))
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 64, in solve_problem
    namespace = solver.solve()
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 86, in main
    namespace = solve_problem(problem)
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/nssolver.py", line 59, in solve
  File "/cluster/home/martinal/studies/flow-reversal-study/cases/sweep.py", line 64, in solve_problem
    namespace = solver.solve()
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/nssolver.py", line 59, in solve
    self._summarize()
    self._summarize()
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/nssolver.py", line 72, in _summarize
    self._final_memory = get_memory_usage()
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/nssolver.py", line 72, in _summarize
    self._final_memory = get_memory_usage()
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/utils.py", line 60, in get_memory_usage
  File "/usit/abel/u1/martinal/nobackup/local/lib/python2.7/site-packages/headflow/core/utils.py", line 60, in get_memory_usage
    mymemory = getoutput("ps -o rss %s" % mypid).split()[1]
IndexError: list index out of range
    mymemory = getoutput("ps -o rss %s" % mypid).split()[1]
IndexError: list index out of range

Comments (1)

  1. Log in to comment