Question about GC

[premesh]

http://java-monitor.com/postedimages/486c7b22-2c4b-4f7b-b2a9-ffd4ed336349.png

Hey , noob to the forum but finding its info VERY useful. I have a tomcat application which was periodically having OOM exceptions. We traced this back to our caching implementation (ehcache) and found that we were storing far too many things in cache than we needed for far to long. After adjusting our cache settings things got significantly better ( no outage for 24 hrs) but we still eventually had a crash... so seems like there must be a leak still?

I posted the graph of the GC as it seems that the outage usually coincides with a spike in the MarkSweep GC... i have to admit i know very little about GC but seems like this is an indication of something... any ideas?

[kjkoster]

Dear premesh,

Welcome to Java-monitor and I am glad you like our monitoring service. Good to hear that it helped you find some memory issues immediately.

The GC times that you posted are terrible. More than one minute is insane. How much memory do you allocate for your VM? Is that enough for your application? Did you do any calculation how much memory each request or user session may use and how many you can handle?

It would be useful if you could post the Heap memory graph. Are there any other graphs that seem to just go up and up? Maybe HTTP sessions? Threads?

Kees Jan

[premesh]

thanks, the webserver has 1GB of memory which is pretty much dedicated to tomcat (~900 mb). It definitely seems like enough most of the time aside from when this happens. I dont have any cacls on how much memory each request or user session may take. WE had user sessions timing out at 3 hrs last week which was one problem we noticed from the graphs as we would hold on to sessions for far too long.. now that is set to 10 mins to that helped ALOT as well .

Im more than happy to post the other graphs but how can i do so easily directly to this thread? should i just take a screenshot?

[premesh]

http://java-monitor.com/lemongrass/generator/linegraph.png?host=5df3956b-9af7-485c-a64c-e15ed039207f&delta=true&objectName=java.lang%3Atype%3DGarbageCollector%2Cn ame%3DPS%20MarkSweep&objectName=java.lang%3Atype%3DGarbageCollector%2Cn ame%3DPS%20Scavenge&attribute=CollectionCount
http://java-monitor.com/lemongrass/generator/linegraph.png?host=5df3956b-9af7-485c-a64c-e15ed039207f&objectName=java.lang%3Atype%3DMemory&attribute=HeapMemoryUsage.max&attribute=HeapMemoryUsage.committed&attribute=HeapMemoryUsage.used

[premesh]

http://java-monitor.com/lemongrass/generator/linegraph.png?host=5df3956b-9af7-485c-a64c-e15ed039207f&objectName=Catalina%3Atype%3DManager%2Cpath%3D/%2Chost%3Dlocalhost&attribute=maxActiveSessions&attribute=maxActive&attribute=activeSessions

http://java-monitor.com/lemongrass/generator/linegraph.png?host=5df3956b-9af7-485c-a64c-e15ed039207f&stacked=true&objectName=com.javamonitor%3Atype%3DThreading&attribute=ThreadsNew&attribute=ThreadsRunnable&attribute=ThreadsBlocked&attribute=ThreadsWaiting&attribute=ThreadsTimedWaiting&attribute=ThreadsTerminated

http://java-monitor.com/lemongrass/generator/linegraph.png?host=5df3956b-9af7-485c-a64c-e15ed039207f&objectName=Catalina%3Atype%3DThreadPool%2Cname%3Dh ttp-80&attribute=maxThreads&attribute=currentThreadCount&attribute=currentThreadsBusy

[kjkoster]

Dear premesh,

Unfortunately, posting the links to the graphs like that does not work. Java-monitor would have to disclose the monitoring data from your server live, which is probably not what you want. :)

Instead, I would suggest you start a new thread using the most relevant graph.

Another trick is to click the "post new thread" under a graph and then copy the image URL from the small graph shown in the editor. Collect those URLs in a text file and post them into this thread. It is not ideal, but there it is. :-/

Kees Jan

[premesh]

http://java-monitor.com/postedimages/3bd8eb17-21d1-466b-a399-a3f710de0cc8.png
http://java-monitor.com/postedimages/8518b61f-c0cd-40ab-8f3f-f82c2498471b.png
http://java-monitor.com/postedimages/957ec1bc-4a7f-4e9c-af25-2d258c572c4d.png
http://java-monitor.com/postedimages/6717b579-de47-4aef-ac0a-779bbaf53455.png

[premesh]

http://java-monitor.com/postedimages/76d0d573-88ba-4269-b615-b20cea6514a4.png

[kjkoster]

Dear premesh,

Excellent. Thanks, now I can actually see the graphs. :)

That threads graph is quite interesting. It just goes up and up. Perhaps you can take some thread dumps and see what all those waiting threads are doing?

Here's some reading on that subject:
http://java-monitor.com/forum/showthread.php?t=1170
http://java-monitor.com/forum/showthread.php?t=317

Hope that helps.

Kees Jan

[premesh]

OK took me while but i was finally able to take thread dumps using jstack while the server was hanging to take a look at what all the waiting threads were doing. I used thread dump analyzer (http://www.alphaworks.ibm.com/tech/jca) to look at the dump file.

I noticed from that dump that i had many threads waiting to get a connection from the db pool ( using spring ) . Took a look at what the pool was configured to and found it was set to 100 max connections. Our DB is on a pretty hefty server (dual quad core w/ 48 GB RAM) so it can certainly handle the load. I upped the pool to 300 and things have been holding still all day..

I can see however that im still getting pretty close to maxing out my heap through out the day but there is eventually some GC and things come down. Still seeing alot of threads waiting when the heap is that high though.. so i took a dump again and now i see a ton of threads like the following:
"e4c79ffef7507f59feeee47610a28896" daemon prio=10 tid=0x00007f8a67013c00 nid=0x172f in Object.wait() [0x00000000662fe000..0x00000000662feaa0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.awai t(AprEndpoint.java:1465)
- locked <0x00007f8a8506ffe0> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run( AprEndpoint.java:1490)
at java.lang.Thread.run(Thread.java:619)

Is this normal? Perhaps spike in traffic caused a surge that left these threads open? We have 3 webservers with a LB in front of them..

[kjkoster]

Dear premesh,

What connector are you using in Tomcat?

Kees Jan

[premesh]

should just be the default that comes with tomcat... something else i found was this http://ehcache.org/documentation/garbage_collection.html . We are using Ehcache in our app and considering the GC timings we are seeing i thought id give it a try. I updated the app to use the following settings :
-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC
-XX:NewSize=256m -XX:SurvivorRatio=16

things are stable right now but we will see how the next 24 hours hold up :)

[kjkoster]

Dear premesh,

Ok. Would be nice if you could post some after-tuning graphs, to show us the differences you achieved with your configuration changes.

My connection question was more: does Tomcat handle the web requests, or is there an Apache httpd in front of it?

Kees Jan

[premesh]

gotcha will do after 24 hours :). We do not have Apache sitting in front of tomcat . all of our static content comes of a CDN so there was little benefit to go that route.

[premesh]

New Graphs :)
http://java-monitor.com/postedimages/a2357717-f9f5-491f-8afe-1b1d9cd2683d.png
http://java-monitor.com/postedimages/697c4404-bd55-4f7f-ba95-deedeee08284.png
http://java-monitor.com/postedimages/57cfe19b-3af0-4408-a801-61b7a84f904b.png

[premesh]

http://java-monitor.com/postedimages/ae47c3e9-1dca-4874-b34b-fbf4f30c1d98.png
http://java-monitor.com/postedimages/e75ad5fd-4605-4e8a-9ee8-c0452dc176fe.png
http://java-monitor.com/postedimages/8fe28af7-f55e-4840-8e7e-b0a042d6245f.png