What could be causing this ?

[just]

http://java-monitor.com/postedimages/d57ba1d5-7ba9-4d00-b29c-baa818d80964.png

Hi,

The server deals with a lot of files (GIS stuff) but sometimes I get "too many open files" exceptions. Looks like files are never closed ? There is also native JAI and Image IO. JAVA_OPTS="-server -Xmx2048M -Xms256M -XX:SoftRefLRUPolicyMSPerMB=36000 -XX:MaxPermSize=512m -XX:+UseParallelGC"

Further: Ubuntu 9.10 32b, 12 GB RAM, Sun JDK 1.6_22, native JAI+ImageIO, Tomcat 6.0.29, no Gdal bindings. Any suggestions welcome. thanks,

Just

[kjkoster]

Dear Just,

Well, file descriptors represent many things. Network sockets, files, pipes, JAR files, you name it.

You can scour your code for resource uses that do not properly handle closing. You need to ensure that *all* resources are cleared in a finally block, like so:


SomeResourceClass resource = null;
try {
resource = new SomethingOrOther();
...
} finally {
if (resource != null) {
resource.close();
}
}


You can have findbugs (http://java-monitor.com/forum/showthread.php?t=200) help you with searching out unclosed resources.

Since your code uses file handling in native code you need to verify also that you are using those libraries in 100 percent the correct way. I am not familiar with those, so you need to check with their respective authors.

Typically, I find code that closes the resource for the happy case, but fails to close it on exceptions that are thrown. Usually the "too many open files" exception is preceded by a bunch of other exceptions that show you where the offending code might be. Are there any exception being thrown 'close' to the "too many open files"?

To find out more about what is happening, start your Tomcat clean and use some kind of fd tool from the command line to dump the file descriptor use. For example, you could use lsof(8) (http://www.manpagez.com/man/8/lsof/) to dump the fd's in use. Then wait for the number of fd's to rise sharply (as seen from this graph) and redo the dump. Finally, diff(1) (http://www.freebsd.org/cgi/man.cgi?query=diff) the two dumps to see what these fd's are used for. This should point you to the right code.

You can make your Tomcat live longer by increasing the number of fd's available to your application. Use ulimit(1) (http://www.freebsd.org/cgi/man.cgi?query=ulimit) or limits(1) (http://www.freebsd.org/cgi/man.cgi?query=limits) to check and increase this value. Linux is very conservative with its default of 1024. I usually run with about 10.000 and Solaris defaults to something like 60.000. It won't solve the problem, but may give you time to solve it.

Or maybe, if your system typically deals with a few hundred open files at any given time, this is perfectly normal behaviour and all you need to do is increase the fd limit. :-)

Kees Jan

[kjkoster]

Dear Just,

By the way, can you please elaborate on why you use "-XX:SoftRefLRUPolicyMSPerMB=36000" as a start-up option. Soft references make my hair itch.

What does this do for you?

Kees Jan

[just]

Dear Kees Jan,

Thanks! I learned a lot. The application is not written by me (offcourse I would always close resources in a finally block :-)). There were two problems causing the issues and both have been resolved now. See also this message thread http://sourceforge.net/mailarchive/message.php?msg_id=26919791 .

The first problem was solved by reconfiguring GIS layers. The second problem was that indeed the max open files was too conservative (1024). I increased to 20480 (which took some searching where/how, but did this in /etc/security/limits.conf and adapted /etc/pam.d/* with "session required pam_limits.so".

I monitored the number of open files lsof | grep <GISdir> | wc -l and these indeed increased far over 1024 but then settled back to a few 100. These are layers of historic maps covering NL with aligned GeoTIFF files using native JAI and optimizing/caching in-memory within GeoServer (www.geoserver.org, FOSS, recommended for geodata publishing/serving).

best regards,

Just

[just]

This was recommended within the GeoServer manual:
http://docs.geoserver.org/stable/en/user/production/container.html#optimize-your-jvm

best,

Just

[kjkoster]

Dear Just,

Great to hear you solved the file descriptor problems.

I never really thought you'd forget to close resources in a finally{} block, but mistakes happen. ;)

I forgot to mention that setting these limits is very OS-dependent. Thanks for letting me know what the settings are for Linux.

Meh. I see that GeoServer uses soft references for caching. I've never really seen the need for that. It makes your cache strategy harder to control or even predict. A big spike in memory use elsewhere in your system can cause your cache to be purged. That ties different parts of the system into each other in unexpected ways. *shudder*

Kees Jan