IOException: error=24, Too many open files


We experienced a couple of jobs that failed today. Looking at the log, it gave us the message of Too Many Open Files.

Error Message Cannot run program "tail": error=24, Too many open files 
[anonymous] ERROR ( - Unhandled exception handling /admin/application-log 
java.lang.RuntimeException: exception on executing command '[tail, -n, 1000, /opt/datameer/<path>/logs/conductor.log]'


If you are interested in knowing what files are currently open you can use the following command.

lsof -p <pid of jvm>

You can also see what the ulimit for open file is using:

ulimit -n


This error is coming from the operating system because of not having enough file descriptors.You might need to increase the max number of open files - global level or per-user limits (or both). 


To raise the global limit for file descriptors you need to edit, depending on your operating system, the sysctl fs.file-max.

Setting file handles

To do this on a user level you need to edit the ulimit.

Setting shell limits

Usually we recommend to set the values mentioned within in given knowledge base articles.