Tomcat freezing

[wakeboarder]

http://java-monitor.com/postedimages/5f91fd7b-e57f-49f4-a70e-e8497df15d35.png

Is this GC graph normal? I've been trying to tweek the settings prior to this, but I think it is hanging. :confused:


-XX:MaxPermSize=256m -Xmx2048m -Xms256m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSInitiatingOccupancyFraction=80

Thanks for any guidance!

[kjkoster]

Dear wakeboarder,

When you say 'hang' what do you mean? Your entire Tomcat server just stops responding, or individual page requests respond slowly?

When it hangs, do you have to restart it to get running again, or are the hangs intermittent and seem to resolve themselves?

9 seconds is certainly a long time for the GC, but you may have quite a big new generation to work through.

Kees Jan

[wakeboarder]

I believe the entire VM waits for the GC. So Tomcat stops replying completely. I had this happen before I had switched to CMS and the new Parallel GC. It will resolve after the 9 seconds.

Should I tweak NewSize and MaxNewSize? Or NewRatio?

Thanks,
Mike

[kjkoster]

Dear Mike,

Ok, so the entire server hangs briefly and then continues as normal, right?

The 9 seconds are reported on the ParNew collector, which works on the young generation. That gives me the feeling the younger generations may be too large, causing the GC for the young generation to take a lot of time. Perhaps you can reduce the heap size (application needs permitting) or limit the size of the young generation.

I cannot see the memory graphs, but I expect to see quite a large churn in your heap memory. You may also want to look at the code not to make as much garbage. :)

Kees Jan

[wakeboarder]

http://java-monitor.com/postedimages/ec99ebdf-2cd2-4ef7-91d6-fa9297ec6e15.png

Everything before 17:00 was normal. Then a crash and I restarted it. The jump in max was a full server reboot (and added mem).

[kjkoster]

Dear Wakeboarder,

Why did you give the JVM a larger heap? Memory usage looked normal before you gave it more memory. I would not have touched the memory configuration for that JVM after the crash.

A crash is not the same as a hang in my book. What exactly happened? Did you shut down Tomcat manually or did it shut down all by itself? Any error messages in the logs? What did the crash look like?

From here it looks like you are stabbing in the dark, tweaking and changing the configuration to chase ideas of what might be happening. Are you undoing the changes that have no effect, or are you just piling them on?

Kees Jan

[wakeboarder]

Hey Kees, thanks again for helping.

I gave it more mem because the server was getting upgraded anyways. Probably didn't need to.

I believe it was hung, but I had to do a manual restart since I couldn't diagnose the problem immediately and it was at a critical time.

No error messages in any of the log files.

I'm not changing any of the config or command line params; the ones in the original post are the ones I have been running for many months. I'd like to figure out how to not let this happen. I run a very active site and that is part of the problem. I'm logging a lot of stuff to the DB (probably causing the churn due to Hibernate stuff).

I thought the CMS GC wouldn't pause the during marking?

[kjkoster]

Dear wakeboarder,

Hmm. If I look at those GC times I cannot rhyme that with your statement that Tomcat was hung. That graph shows pause times of a few seconds and you could have waited for it to clear.

Are you 100% sure that your Tomcat is hanging on the GC. What makes you so sure?

What you might do is take some thread dumps while Tomcat is running fine and then wait for Tomcat to hang and (before restarting) take another few thread dumps. From those dumps it may be clear what Tomcat is doing at the time of the hang.

If you are logging things in Hibernate, perhaps you can switch that over to using a flat file log. Databases tend to be quite expensive if it is just for log gin. How many concurrent database connections do you have?

Kees Jan

[wakeboarder]

I'm trying to catch another hang, but it isn't easy. I can see from the graphs that it happens for 5 seconds here and there though.

DB connections never get that high, with on average under 10 connected.

Should I enable -XX:+PrintGCDetails to try and catch it?

Are you still running Kick starts?