If you've ever come across this error, it can be very misleading.
Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640)
The common suggestion (as the error suggests) is a memory related problem. Some tips but ultimately unhelpful:
http://candrews.integralblue.com/2009/01/preventing-outofmemoryerror-native-thread/
http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
The common theme amongst all of them is the stack size.
Came across this post with same test code:
http://www.odi.ch/weblog/posting.php?posting=411
However the tests were inconsistent, modified it to only go 50 calls deep. Modified code here.
We discovered that changing the options made very little difference to the maximum number of threads.
Originally we thought it was related to changing the linux option for hard/soft maximum number of files that we did the day before:
# vi /etc/security/limits.conf testuser soft nofile 4096 testuser hard nofile 10240 # ulimit -Hn 10240
However that didn't seem to be it.
Then we finally came across max process per user:
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515005 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Which is set to 1024 by default. So adding this to ~/.profile fixed it for us:
ulimit -u 4096
