Too many Open Files is a common Java exception, usually caused by system configuration or programs having Too many files open. This problem is often associated with the use of Ulimit. There are some problems with the use of ulimit, and this article will sort them out.

Too many Open files are abnormal

Here is the Java exception stack when the system exceeds the maximum number of open files:

Exception in thread "main" java.io.IOException: Too many open files
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createTempFile(File.java:2024)
	at java.io.File.createTempFile(File.java:2070)
	at com.imshuai.wiki.ulimit.App.main(App.java:16)
Copy the code

If it is not a program problem (program problem, need to see why a lot of files, such as through lsof), usually through ulimit to adjust the number of files open limit to solve, but ulimit itself has many holes, let’s make a conclusion.

What is the ulimit

Refer directly to the ulimit help documentation (note: not man ulimit, but help ulimit, ulimit is a built-in command that provides Ulimit help in C) :

Modify shell resource limits.

Provides control over the resources available to the shell and processes it creates, on systems that allow such control.

As you can see, ulimit provides management of the resources available to the shell (or the process created by the shell). In addition to the number of open files, the resources that can be managed are: maximum write file size, maximum stack size, core dump file size, CPU time limit, maximum virtual memory size, etc. Help ulimit lists the resources for each option limit. Or look at ulimit -a to see:

maoshuai@ms:~/ulimit_test$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) 100
pending signals                 (-i) 15520
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15520
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Copy the code

Understand the ulimit

Before using ulimit, there are a few confusing points:

Management dimensions of Ulimit

The first question to understand ulimit is what dimension the limit is. For example, if nofile is set to 1024, does this mean that the current user can only open 1024 files in total or a single shell session process can only open 1024 files? ** The process is already specified in help ulimit, but we can verify it by using the following method:

Here is a Java program to open 800 files:

class Ulimit{
    public static void main( String[] args ) throws IOException, InterruptedException
    {
    	List<FileInputStream> fileList = new ArrayList<FileInputStream>();
    	for(int i=0; i<800; i++) { File temp = File.createTempFile("ulimit-test".".txt");
    		fileList.add(new FileInputStream(temp));
    		System.out.println("file_seq=" + i + "" + temp.getAbsolutePath());  
    	}
    	// keep it running, so we can inspect it.Thread.sleep(Integer.MAX_VALUE); }}Copy the code

We set nofile to 1024

ulimit -n 1024
Copy the code

Then we run two process instances:

nohup java Ulimit >a.log &
nohup java Ulimit >b.log &
Copy the code

800 files are created in a.log and B. log, and no exception is reported.

If you set the ulimit to 700 and re-test, and find that the Java program reported Too Many Open Files when creating 688 files (not 700 because Java itself opens some files),

file_seq=688 /tmp/ulimit-test7270662509459661456.txt
Exception in thread "main" java.io.IOException: Too many open files
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)
        at java.io.File.createTempFile(File.java:2070)
        at Ulimit.main(Ulimit.java:12)
Copy the code

Although the U of ulimit stands for user, it turns out that the dimension that ulimit controls is the shell session or the process that the shell creates (at least for nofiles). The number of files opened by the current user can far exceed the value of nofile.

So by lsof | wc -l view system, open the file number to determine whether to open the file number is super, is not correct. In addition, lsof | wc -l is also does not reflect the system open file number! (Follow-up weekly supplement)

Soft and hard

The second important aspect of understanding ulimit is the distinction between soft and hard. Ulimit limits resources into soft and hard categories. That is, the same resource (such as nofile) has soft and hard values.

On command, ulimit distinguishes soft from hard with -s and -h. If -s or -h is not specified, the display value refers to soft, and the setting value refers to both soft and hard.

But what is the difference between soft and hard? The following explanation is more accurate (from Man 2 Getrlimit)

The soft limit is the value that the kernel enforces for the corresponding resource. The hard limit acts as a ceiling for the soft limit: an unprivileged process may set only its soft limit to a value in the range from 0 up to the hard limit, And ** (irre‐versibly) ** Lower its hard limit. A Privileged Process (Under Linux: one with the CAP_SYS_RESOURCE capability) may make arbitrary changes to either limit value.

The difference between soft and hard

  1. Soft is always less than or equal to hard
  2. Actions beyond soft or hard are rejected. Combined with the first point, this statement is equivalent to saying that an operation will be rejected if the soft limit is exceeded.
  3. A process can modify the soft or hard of the current process. However, the modification must meet the following rules:
  • After modification, soft cannot exceed hard. That is, as soft increases, it cannot exceed hard. If hard is lower than soft, soft will be lower as well.
  • Any non-root or root process can increase or decrease soft in the range [0-hard].
  • Non-root processes can reduce hard, but not increase it. If nofile is 1000 and changed to 900, it cannot be changed to 1000 again. (This is a one-way, there is no return operation)
  • The root process can change the hard value at will.

There is no difference between soft and hard in terms of control. Both restrict the use of resources, but soft can be modified by processes before they use it.

Ulimit changes and takes effect

It’s good to know ulimit, but more important is how to modify it, which is a common task at work.

Here are a few things about how ULIMIT works:

  1. The value of ulimit is always inherited from the parent process.
  2. The ulimit command is used to modify the Settings of the current shell process. This also means that in order for the changes to work next time, they need to be durable (at least as far as the target process is involved), such as.bashrc, or the process’s startup script.
  3. It also follows from point 2 that a running process is not affected by changes to ulimit.
  4. Increasing the hard value can only be done by root

Here are two examples:

Example 1: A non-root process requires a nofile of 2048. The current soft value is 1024 and the hard value is 4096

You can add ulimit-ns 2048 to the process startup script

Case 2: A non-root process requires nofile 10240. The current soft value is 1024 and the hard value is 4096

Obviously, non-root users can’t break through. Can only be amended by root, generally modify/etc/security/limits file, modify the method in the configuration file has indicated in the comments, the format is:

One record contains 4️ columns, which are scope domains (i.e., valid scopes, which can be user names, group names, or * for all non-root users); T Type: soft, hard, or – Indicates both soft and hard. Item, the resource control item in ULIMIT, whose name enumeration can refer to the comments in the file; And then finally value. For example, set nofile to 100000 for all non-root users

*  hard nofile 10000
*  soft nofile 10000
Copy the code

View limits for running processes

After the ulimit is changed, you can run the ulimit command to view the change. There is a more accurate way to look at running processes (for example, if a process was started before changing its Ulimit value) : look in the limits file in the process directory. For example, the /proc/4660/limits file records all limits values for process 4660:

maoshuai@ms:~/ulimit_test$ cat /proc/4660/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited  bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15520 15520 processes Max open files 2000 2000 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited  bytes Max file locks unlimited unlimited locks Max pending signals 15520 15520 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited usCopy the code

Ulimit takes no parameters

There was a time when you used ulimit to view the file and saw that it printed unlimited. This is clearly not true, as help ulimit explicitly states:

If no option is given, then -f is assumed.

So, ulimit without an argument is equivalent to ulimit -f -s (-s or -h is equivalent to -s without specifying -s or -h), which actually refers to the maximum size of the file that can be written.

other

How do I view the number of open files in the system

Role is losf command while “list open files”, but use lsof | wc -l statistics on the number of open file is very inaccurate. The main reasons are:

  • In some cases, a line might show a thread, not a process, and in the case of multithreading, it might be a mistake to think that a file has been opened multiple times
  • The child process will share file handler if usedlsofStatistics have to use sophisticated filtering conditions. A simpler and more accurate approach is to look through the /proc directory. /proc/sys/file-nr = /proc/sys/file-nr = /proc/sys/file-nrwww.kernel.org/doc/Documen…). To check the number of open files in a process, check the number of open files in /proc/$pid/fd:

Java automatically increases the soft value of nofile to the hard limit

In the course of my research, I found that Java programs seemed unaffected by the soft value of nofile. Limits file (/proc/$pid/limits); limits file (/proc/$pid/limits); How and when and where JVM change the Max Open files value of Linux?

Nofile is not the same for Java launched from Eclipse in Ubuntu as it is for Java launched from the command line

Using PSTREE, it is found that Java for Eclipse is started through gnOMe-shell and the command line is started through gnOMe-Terminal. The gnome – terminal through systemd – user start again, and systemd – the user didn’t seem to read/etc/security/limits the value of the conf. I’ll fill in the instructions for this hole when I get a chance.

File-max controls the total number of files that the kernel can open

In addition to the ulimit control, the /proc/sys/fs/file-max file controls the total number of files that the system kernel can open. So, even if nofile in ulimit is set to uLimited, it is still limited.

Ulimit Common options

Ulimit -ha # ulimit -hn # nofile ulimit -sn 1000 # nofile ulimit -sn 1000 # nofile -n 1000 # Set the hard and soft values of nofiles to 1000Copy the code

reference

  • Session failures with error “Too many open files”
  • ulimit man page
  • lsof | wc -l sums up a lot of duplicated entries
  • How and when and where jvm change the max open files value of Linux?
  • Why file-nr and lsof count on open files differs?

Java and Linux Learning Weekly is published every Friday, updated synchronously on Github, Zhihu, Nuggets