Optimize memory usage to avoid incurring in allocation problems

[Desrever Nu]

This happened recently

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f8880000, 9437184, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 9437184 bytes for committing reserved memory.

relevant : http://www.ibm.com/developerworks/library/j-nativememory-linux/

[Benjamin Cordes]

I'm not sure but it seems to me the spark framework uses lot /too much memory. I'm wondering whether it has to do with all the logging it does.

[Benjamin Cordes]

I found that I did not ignore jetty in the session log. perhaps that was the cause of this

[Desrever Nu]

Happened again and again, especially after a long run:

os::commit_memory(0x00000000f8e80000, 9437184, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 9437184 bytes for committing reserved memory. # An error report file with more information is saved as: # /root/bots/poloniex_nbtbtc/hs_err_pid1235.log 

full error stack -> https://s3.amazonaws.com/uploads.hipchat.com/120658/880550/M7E12LM6ugeIaoN/hs_err_pid1294.log

After monitoring the memory with JProfiler, it looks to me a thread related error rather than heap space and objects eating up space.

This seems to be confirmed by the huge number of new blocked threads and the stacktrace in the log :

Event: 195017.707 Thread 0x00007f5a6c439800 Exception <a 'java/lang/OutOfMemoryError': unable to create new native thread> (0x00000000f6cc31d8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u40/2855/hotspot/src/share/vm/prims/jvm.cpp, line 3016]

[Desrever Nu]

note for self : test whether it was also appening with java1.7 and nubot 0.1.5

[Benjamin Cordes]

This must be a hipchat bug - pretty sure. Usually only 7 threads are running. With Hipchat it's 8 and steadily growing.

without hipchat

Thread[Monitor Ctrl-Break,5,main] id: 9 name: Monitor Ctrl-Break sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[checkConnection,5,main] id: 11 name: checkConnection sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[sendLiquidity,5,main] id: 12 name: sendLiquidity sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[Strategy Fiat Task,5,main] id: 13 name: Strategy Fiat Task sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[Strategy Crypto Task,5,main] id: 14 name: Strategy Crypto Task sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[priceTriggerTask,5,main] id: 15 name: priceTriggerTask sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
Thread[DestroyJavaVM,5,main] id: 18 name: DestroyJavaVM null group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue

[Benjamin Cordes]

With hipchat

21:25:21.938    DEBUG   64  c.n.n.u.Utils   currently active threads: 8
21:25:21.939    DEBUG   70  c.n.n.u.Utils   0: Thread[Monitor Ctrl-Break,5,main] id: 9 name: Monitor Ctrl-Break sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.940    DEBUG   70  c.n.n.u.Utils   1: Thread[checkConnection,5,main] id: 11 name: checkConnection sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.940    DEBUG   70  c.n.n.u.Utils   2: Thread[sendLiquidity,5,main] id: 12 name: sendLiquidity sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.940    DEBUG   70  c.n.n.u.Utils   3: Thread[Strategy Fiat Task,5,main] id: 13 name: Strategy Fiat Task sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.941    DEBUG   70  c.n.n.u.Utils   4: Thread[Strategy Crypto Task,5,main] id: 14 name: Strategy Crypto Task sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.941    DEBUG   70  c.n.n.u.Utils   5: Thread[priceTriggerTask,5,main] id: 15 name: priceTriggerTask sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.942    DEBUG   70  c.n.n.u.Utils   6: Thread[pool-1-thread-1,5,main] id: 20 name: pool-1-thread-1 sun.misc.Launcher$AppClassLoader@7cca494b group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue
21:25:21.942    DEBUG   70  c.n.n.u.Utils   7: Thread[DestroyJavaVM,5,main] id: 21 name: DestroyJavaVM null group: java.lang.ThreadGroup[name=main,maxpri=10] alivetrue