Java Memory Tuning Quick-Start

[kjkoster]

Dear All,

Here is a quick-starter guide for system administrators who run into Java's java.lang.OutOfMemoryError.

1. You're probably being leaned upon to FIX IT NOW, so here is some quick-fix memory tuning advise that Eugene Dzhurinsky suggests on the FreeBSD Java mailing list. This should buy you some time.
   

   Please find out where the java process is started in scripts, then search for lines like "-Xms" (minimal memory size) "-Xmx" (maximal memory size) and tune it. For example, to grant 256MB of heap to Java use -Xmx256m. If you will not find any of these options, just find where the java executable is defined in scripts and then append the -Xmx right after the executable name to tell Java the new value for heap size.

2. If you are happy with this solution, you're done now. Java VM's tend to have very tight memory defaults and just giving it a bit more breathing room may resolve the OutOfMemoryError's and leave you free to attend to more pressing matters. On the other hand, you may find that even with the raised memory limits, you application still bombs out with the same error message (albeit after a longer uptime). In that case, read on.
   
3. Bone up on Java's memory model and garbage collectors. Here is a little article I wrote about Java's garbage collectors and memory pools. This article specifically talks about the things a system administrator needs to know and leaves the developer-type stuff out. :-) http://www.kjkoster.org/zapcat/Zabbi..._Template.html (scroll down past the configuration items and start to read at "Memory Pools and Garbage Collectors". It's about a fifteen minute read.
   
4. Hook up JConsole to your application. Here are is a collection of links to JConsole howto's for several application servers.
   
5. Find the stricken memory pool. Open JConsole and connect to your application. Then open the "Memory" tab. Check each of the memory pools in use (the bars in the bottom-right of the screen). You can safely ignore the Eden, Young and Survivor pools. They'll cycle through all possible fill rates under normal circumstances. Do check the Old, Tenured and Perm generations and the Code cache. They should be under 70% full at all times. Put some load on the application and watch the pools fill up.
   
6. Tune it. Give the overflowing memory pool some extra memory to work with. Here is a list of command line switches for the JVM, including ones to tune memory pool sizes. If you have a server that is dedicated to run your application you can freely assign huge slabs of memory to your application.
   
7. GOTO 5.

While JConsole is excellent for looking into a running JVM, it has the definite disadvantage that you cannot see how your memory use develops over time. If you install Java-monitor's monitoring tool, you can. Check out the live demo to see what its looks like: http://java-monitor.com/forum/livedemo.php. If you'd like to start using Java-monitor, read these installation instructions. An added benefit is that it will send you an e-mail if your server goes down.

Hope this helps.

Is this what you guys do too, or do you have a different sequence of steps?

Kees Jan

[zwets]

Excellent tips, thanks!

[kjkoster]

Dear All,

For some *really* interesting discussion of Java's garbage collectors, have a look at Peter Johnson's presentation at EUROcmg 2007.

He really digs in, even suggesting ways to do a detailed analysis of garbage collector behaviour, plus a discussion on how to interpret the results.

Grab a coffee, switch off the phone and read it.

Kees Jan

[Barry]

Maybe a belated reply to this thread, but also important to take into consideration.
Not all memory problems can be solved by tuning VM settings.

For example, somewhere I worked I saw some "creative" code that would thumbnail all the images per page at request time. The images were megabyte size scans and archeological pictures and would consistently plow through 64mb, 512mb, 2GB heapspace until we sat down and actually looked why this was happening.

While this example is a tad extreme, it does show that you should look for VM/GC tune-ables first and then persue going through the code (using a profiler for example)

Ah, that reminds me that I still need to do an intro on VisualVM

[kjkoster]

Dear All,

Of course, with the introduction of monitoring right here on this forum you can also download your Java-monitor probe and see the memory stats as you browse the forum. :-)

http://java-monitor.com/install.html

Kees Jan

[kjkoster]

Dear Barry,

Your example is indeed extreme. However, it does illustrate that in most cases OOME's should be solved by the developers, not by administrators. So unless the JVM got too little memory, you will always need code changes (and full test cycle, and release) to resolve OOME's.

All an admin can do is give the JVM extra memory to delay the next crash and buy the developers some time.

Kees Jan