Slow loadtimes

[Pate]

Hey,

Found this forum by visiting #tomcat on freenode network yesterday. Immediatly signed up and so far its looking really promising.

The problem we have is slow loadtimes during heavier loads (not that much really). We know our application demands alot of resources, but the weird thing is that neither the CPU or Memory is peaked during the load.

It's a 64-bit linux system running on 10 GB of RAM. Its using Apache 2.2 with mod_jk (ajpv13). Tomcat 6.0.18, java 1.6 (Java version 10.0-b23).

System seems fast if not many visitors on it. This makes me think its related to the tomcat configuration or maybe MySQL. Need any advice I can get.

What values can I present or should I look on to find what can be set to strict. I might start with saying I don't made that much changes to the default configuration.

[kjkoster]

Dear Pate,

Hmm. We'd need a lot more detail to be able to help you here.

You installed Java-monitor, yeah? If so, how much heap memory is being used and what are your peak garbage collect times?

Low memory and low CPU tends to suggest that you are either disk bound, or limited by an external web service. Does your system depend on external web services?

You can get a feel for what your I/O subsystem is doing with iostat(8) (http://www.freebsd.org/cgi/man.cgi?query=iostat) or comparable tools. You'd need to run iostat when the system is quiet and when it is loaded. It will tell you what disk is taking the load. The load that is easy for your machine to handle is very much dependent on your setup, so you really should get a good feel for what your disks can do before you can use iostat effectively.

If either your app or mysql is the hog, have a look at P6Spy (http://www.p6spy.com/). That is a JDBC driver that collects statistics. It sits between your app and the real MySQL driver. It will tell you what numbers of queries you hammer MySQL with and what queries are slow.

Do you have MySQL's slow query log (http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html) enabled? In Tomcat, enable the response time monito (http://java-monitor.com/forum/showthread.php?t=5)r, so you can see what pages take a long time to load.

Please feel free to post your test results here, so we can discuss them.

Kees Jan

[Pate]

Here's some 5-min poll from last day from MRTG. It gets slow during day when more preasure on it, but fine if very little load.

2 Xeon processors (Quadcore) with 10GB ram.

Memory:
Max Average Current
Used: 4396.1 MB (660.5%) 4396.1 MB (660.5%) 4396.1 MB (660.5%)
Buffers + Cache: 9043.3 MB (1358.8%) 9043.3 MB (1358.8%) 9043.3 MB (1358.8%)

CPU:

Max Average Current
user: 672.0 %CPU 128.0 %CPU 24.0 %CPU
total: 680.0 %CPU 128.0 %CPU 24.0 %CPU


Memory:
Max Average Current
Used: 4396.1 MB (660.5%) 4396.1 MB (660.5%) 4396.1 MB (660.5%)
Buffers + Cache: 9043.3 MB (1358.8%) 9043.3 MB (1358.8%) 9043.3 MB (1358.8%)


From my Java-monitor, the only thing I see changing is the Garbage Collection. Is this normal or high?

Garbage Collection
Collection count Accumulated collection time
PS MarkSweep 7993 +6 in the last minute 3 hours and 17 minutes
PS Scavenge 52 0 seconds and 984 milliseconds

[kjkoster]

Dear Pate,

That is a nice machine. I ordered one just like that for Java-monitor.

Your garbage collectors are in what I call 'fire fighting' mode.

Here is what a normal, quiet machine looks like:

Collection count Accumulated collection time
PS MarkSweep 63 10 seconds and 817 milliseconds
PS Scavenge 44788 +24 in the last minute 1 minutes and 11 seconds

Notice how on my machine the GC "PS Scavenge" seems to be doing all the work and on your machine "PS MarkSweep" is working its furry little butt off.

The fact that only one GC is working is not strange. Java picks two garbage collectors. One is a quick-n-dirty one that loosely rakes together some free memory when called upon. That is the "PS Scavenge" GC in both our cases.

When the scavenger fails to find enough memory for more allocations, the JVM ditches it and brings in the stop-the-world garbage collector. It will find more memory than the scavenger, but at a price.

In your machine, the scavenger is ignored entirely (only 52 collections at the sample time) and the stop-the-world GC is doing all the work (close to 8000 collections).

The solution for your configuration is to tune the memory for your JVM. What is the value of your -Xmx and other memory-related settings?

If you have no -Xmx specified, Java defaults to some small value of a few MB. It is OS dependent how much, but it ranges from 64MB to 256MB I believe. In that case, you will have to experiment with how much RAM you give Java. 1GB is a good starting point. Then compare the GC time graph, the heap mem graph and the GC counts before and after.

Before you change anything, please post the GC time graph (the one you posted was the GC count, which is not as interesting) and the heap memory graph (topmost). Also tell us the value of you -Xmx setting.

Kees Jan