[runtastic]

Hello,
we currently have serious problems with our REST service. Every time we get some load on the server - the REST service starts to create new HTTP-Threads (that's OK) - BUT the REST service gets non-responsive every time we see the spikes like in the screenshot altough the CPU-load is rather low.

We've set
- Max-thread-Pool size for HTTP-Requests to about 500 - as you can see from the screenshot - they never get above 200 threads for the whole Glassfish. Max Queue size: 4096, Min Thread Pool Size: 2, Idle Thread Timeout: 900 Seconds
- For the http-listener we set max-connections (in keep-alive mode) to 250, timeout 30seconds
- EJB container: max pool size: 64,
- JDBC connection pool: max pool size 64, idle timeout 300seconds

Do you have a hint where the bottleneck could be in this case? First we thought that it would be the DB-connection because they are less than 500 (64) - but from our Glassfish-Monitoring the max-ever used DB-connections have been 64 - normally they use 3-8 - also when the REST service gets unresponsive.

Thanks for your ideas,
René

[kjkoster]

Dear René,

Interesting case.

A high number of threads combined with a low CPU usage percentage is indicative that your threads are not processing, but waiting. The question is: what are they waiting for? They could be waiting for each other (lock contention) or they could be waiting for I/O from an external entity (database, web service, file system). To find out, make a few thread dumps when the application is being unresponsive again. Then look at what the threads are doing.

To get started with thread dumps have a look at :
http://java-monitor.com/forum/showthread.php?t=317
http://java-monitor.com/forum/showthread.php?t=616

The database is a likely candidate, as you point out. 64 connections still sounds like a lot to me. May I suggest that you enable slow query logging for your database and see if there are any queries that take a long time?

Also, you may want to count the number of queries that each connection handles. A boatload of small, fast queries still takes a long time.

At any rate, make some thread dumps and see what lock or external component the threads are waiting for. Then let us know, so we can help you debug it further.

Hope this helps.

Kees Jan

[runtastic]

Hi Jan,

thanks for your input - I will make some thread-dumps when the effect occurs next time.
I've just enabled slow-query-logging for the database.

Hope I can find something! May I ask you again in the next few days - if I don't get it right. I will post further infos according to it then.

Thanks and all the best,
rené

[runtastic]

Hi there,
error happened again


I've checked the slow-queries - there aren't any above 1sec.
How can I check the number of queries each db-connection handles?

I've made a thread-dump out of glassfish-console, a memory-footprint and made some screenshots from java-monitor the time the responsiveness of our REST service was gone.


I've uploaded the thread-dump, a memory-footprint and the infos I grabbed from the java-monitor and added them as attachments.

thread_and_memory_dump.zip

java-monitor-graphs.zip

addInfos.zip

Do you see anything that could give a hint on the problem?

Thanks very much for your help!
rené

[kjkoster]

Dear René,

Your memory and garbage collector data looks fine. Memory is nicely filled, but no reason to change anything there. The only (minor) nit is that you could switch to using the CMS collector for better throughput. That is not going to resolve your current unresponsiveness issues, though. It is something to change once you have licked this problem.

Another optimization you might want to make is to start using the server VM and not the client VM that you are using today. Again, this will not resolve your unresponsiveness. It is just a standard optimization that I can highly recommend. http://java-monitor.com/forum/showthread.php?t=552

I see that you are using MySQL. What you can do is try to enable the general log in MySQL. http://dev.mysql.com/doc/refman/5.1/en/query-log.html Please note that that will log *every* SQL statement executed, so it will eat your disk space in for breakfast. For a test machine that is usually fine, though. Or for a few hours on a production system. From that log you can then check the sanity of your database query volumes. From the stack dump I don't think is this the issue, though. I see no threads parked on MySQL I/O, which usually points to database interaction problems.

Anyway, on to the thread dump. I see quite some things waiting on a blocking queue. These threads have generic names (Thread-nnn) so they are probably threads you started in your own code. Are you sure these are expected to block there? They seem to be waiting to read data from somewhere.

The rest looks fine to me. You seem to be using both Derby and MySQL, which strikes me as a bit odd, but I am sure there is a good reason for that?

It is possible that your code is waiting for a time-out of some kind? Is your REST service self-contained, or does it call its own external services?

Kees Jan