[vijay172]

Hi,

I am running a Tomcat 7 fronted by Apache 2.2 across 4 nodes served up from AWS on windows 2003 on jdk 1.6.0_27.The 4 nodes are load balanced by AWS ELB. Amazon has tuned the ELB's properly and are getting minimal throughput though they are seeing 6 secs lag sometimes from the backend instances.

I keep getting a NoReponseException from Apache client from Tomcat when i run a load test.

I have the following in server.xml for https connector:

maxThreads="300" acceptCount="400" connectionTimeout="120000" enableLookups="false" keepAliveTimeout="10" minSpareThreads="25" maxSpareThreads="25" maxKeepAliveRequests="1" scheme="https" secure="true" SSLEnabled="true"

Cannot change connectionTimeout because of a slow backend integration we have.Reduced keepalive to a minimum on Tomcat because of that.

I have an httpd.conf with:
ThreadsPerChild 300
MaxkeepAliveRequests 300
KeepAlive On
Timeout 300
KeepAliveTimeout 10

I need to be able to support 220 concurrent clients and right now, i get stuck at about 40 users even though i am using 4 large instances of AWS servers with 1 GB RAM allocated to each Tomcat instance.

TIA,
Vijay

[kjkoster]

Dear Vijay,

First off, keepAliveTimeout in Tomcat is in milliseconds, not seconds. Apache httpd is configured in seconds and Tomcat in ms. Did you want to set it to 10ms?

In order for us to help you we need more information. Are the CPU's of the machines busy or idle? If busy, what process is eating the CPU? Are the Tomcat HTTP thread pools filling up or not?

When you connect straight to Tomcat instead of going through Apache, are you able to get better results?

Are you running Java-monitor yet? If not, install Java-monitor on the servers so you can easily see things like CPU usage, memory usage and thread pool fill levels.

Kees Jan

[vijay172]

Hi Kees,

Thanks for replying.Apprecaite your help.

keepAliveTimeout was kept low so that connections close out right after the call is complete.

The CPU was getting close to 100% in today's run.

How do i check the tomcat HTTP thread pools ? I have installed sysinternals tcpview to look at the processes.I also have been looking at jvisualvm.exe from jdk1.6

memory usage was pretty low at about 550MB out of 1 GB.

I did see some errors with 502 proxy errors and Stream closed errors showing up on some calls made to an external service we are calling.

Still digging in there to get the actual load test completed.

TIA,
Vijay

[kjkoster]

Dear vijay172,

Since your CPUs are running at 100% you need to find out what they are doing. Is it the Tomcat process that is pegging the CPU or another process?

If it is Tomcat, try taking a few thread dumps of the servers during the load test and find out where the code hotspots are. Here is a thread that discusses analyzing load test results, and here is another one. They are a little UNIX centric (since I don't actually know anything of Windows), but I am sure you can translate this to your local tools.

Hope this helps. Let us know what you find.

Kees Jan

[kjkoster]

Dear vijay172,

Also, you can run Tomcat with JAVA_OPTS="-Xprof". That will cause Tomcat to dump a basic execution profile into catalina.out upon shutdown. So what you do is start Tomcat (with the -Xprof flag for Java), run the load test for 10 minutes and then stop Tomcat. Finally open catalina.out in your text editor and see what the profile tells you.

For the thread pools, have a look at http://java-monitor.com/install.html and follow instructions. That will give you performance graphs of your servers, such as the thread pool fill levels.

Please note that this slows Tomcat down a bit, but the result may be worth it.

Kees Jan