User sessions monitoring

[cfd]

Hi everyone,

I discover this community after a talk in Devoxx '09 from Kees Jan. Nice talk :-)

At my office, we have 20+ GFish 2.1 and I'm writing an app to monitor them (JSE to get some values from our gfishs, manage some alerts to our production alert console and store in a database to do some reports).

Now, I would like to monitor the "user load" in term of number of http sessions. I tried to find some keys into the JMX but without success.

Any idea of interesting key(s) ?

Thanks for your help,

Christophe.

edit> I'm reading the gfish's probe source and just found an interesting line: ServerGlassfish.java:62
( the key "com.sun.appserv:type=Selector,*", "http21048" (our admin port) )

I switch the http monitoring on on the gfish.
I see some data but if I connect few (admin) consoles on this port, I don't see any counter incrementation.
Maybe I missed something ?

[kjkoster]

Dear cfd,

Thanks for the compliment. Glad you liked the talk. :)

The selector is maybe a bit too low-level, as this deals with TCP connections. That number is pretty much irrelevant when you are looking to monitor more high-level stuff like user sessions.

I don't use GlassFish myself, so I cannot help you with user sessions. You might try my Spill the Beans JMX listing JSP (http://java-monitor.com/forum/showthread.php?p=1655) to see what MBeans there are in the JVM. The advantage of using that page over JConsole is that it gives you the overview in an easy to search format. No need to click around, just use your browser's find function.

In my experience, rolling your own monitoring app is a bad idea. I should know, because I wrote Java-monitor. :) The hardest part is dealing with false positives and with the large number of data points you get. Write one if you must, but there are literally dozens of systems out there that will do monitoring and save you tons and tons of work.

Kees Jan

[cfd]

Hi Kees Jan,

Thanks for your reply and first of all, I wish you a Happy New Year of Monitoring :)

I don't use GlassFish myself, so I cannot help you with user sessions. You might try my Spill the Beans JMX listing JSP (http://java-monitor.com/forum/showthread.php?p=1655) to see what MBeans there are in the JVM. The advantage of using that page over JConsole is that it gives you the overview in an easy to search format. No need to click around, just use your browser's find function.
I'll deploy the jsp on monday to check. On my side, I wrote for the fun a (quick and really dirty) plugin for VisualVM to be able to search a keyword into the MBean collection. I don't understand why they don't implement a search tool, an appserver could publish thousand of keys...

In my experience, rolling your own monitoring app is a bad idea. I should know, because I wrote Java-monitor. :) The hardest part is dealing with false positives and with the large number of data points you get. Write one if you must, but there are literally dozens of systems out there that will do monitoring and save you tons and tons of work.
You're right, it's not that easy, especially to show the good values and, of course, to raise the good alerts to our sys admins. But our monitoring is quite simple, so I can manage it (for now ;) )

I test your probe in one of our development gfish and didn't test the memory behavior yet. The very first version of my probe was based on event listener and on gfish management rules. It was really fun to write and works really well. But. As long as the appserver is working well (no OOME), the probe works well. When the appserver is working on the edge (heap, permgen,...), the monitoring becomes hazardous (timeout, strange values,...)

How did you manage that case in your probe ?

Christophe.

[kjkoster]

Dear cfd,

Looks like I can wish you the same thing: a happy new year of monitoring. I am looking forward to your talk about monitoring on the first anniversary of your own monitoring server. :)

Not only can app servers have gobs of keys, they can even be dynamic. They might come and go while the server is running. That is why some of the JMX queries that Java-monitor uses uses wildcards to find for example the thread pools and database connection pools.

I largely deal with error conditions by not dealing with them, if that makes any sense. If there is any form of error or problem while the probe does a collection run in your app server, it stops and goes back to sleep. This causes the central server not to get data at all and thus to start sending outage warnings.

Also, my probe relies solely on JDK libraries and *nothing* else. I don't allocate memory unless I have to and I try to use as few API's as possible.

So far, I have had no reports of bad values due to memory conditions in the JVM. Unfortunately, I learned that users tend not to report every problem they encounter (to put it mildly).

I'm curious to learn about the things you query from your GF server, and why?

Kees Jan

[cfd]

I'm curious to learn about the things you query from your GF server, and why?
Some basic things to be able to know if our appservers (and servers) are running well and if they can support a new deploy (in terms of memory).

Every 5 seconds, I grab and put all these values in database:

java.lang:type=Memory/HeapMemoryUsage/max
java.lang:type=Memory/HeapMemoryUsage/used
java.lang:type=MemoryPool,name=Perm Gen/Usage/max
java.lang:type=MemoryPool,name=Perm Gen/Usage/used
java.lang:type=OperatingSystem/OpenFileDescriptorCount
java.lang:type=OperatingSystem/MaxFileDescriptorCount
java.lang:type=Threading/ThreadCount
java.lang:type=Threading/DaemonThreadCount
java.lang:type=ClassLoading/LoadedClassCount
java.lang:type=ClassLoading/UnloadedClassCount
java.lang:type=Runtime/Uptime


I compute heap, permgen and file descriptors ratios to alert our production team if thresholds are passed.

The good thing about the solution I wrote is that I'm not limited to monitor our appservers, I can monitor every JVM running (one of the line in the screenshot is a "simple" JVM running a in-house light server e.g.)

Besides the alerts, I have a really simple GUI to be able to check visually the last data sent.
http://tbonefox.free.fr/monkey_console.jpg
(edit> some 'clouds' in the console are not linked to the values because it's in development process and few ratio thresholds are defined extremely low)

The head of this thread was to populate the column "Users" to have an idea of the user load.

The second step of this monitoring is to monitor our apps to (mainly) detect erroneous action like "disable" or "undeploy". It's in brainstorming mode for now :)

Christophe.

[kjkoster]

Dear cfd,

Looks nice, especially with the weather icons. Cool idea.

Hmm. I think your monitoring app would benefit from monitoring more hard limits. For instance, the number of threads in the HTTP worker pool and the database connection pools.

I found HTTP worker pools a nice measure of how well behaved the systems are. Sometimes web service consuming applications go crazy with one of my web services and that shows up nicely in the HTTP worker pool.

Good luck with your monitoring app, looks like it will be very useful to you.

Kees Jan

PS. Java-monitor can also monitor any JVM. It's just a little more work. :) The sample code is in the probe's sources.

[cfd]

The icons come from this project: http://tango.freedesktop.org/Tango_Desktop_Project

I'll check the http worker pools. Thanks for the pointer.

Christophe.

[Barry]

Sorry for the late reply, but allow me to chime in.

First of all, what kjkoster said about MBeans being dynamic, he is correct.
By default Glassfish will not publish monitoring data on the HTTP connectors (only the configuration settings)

You need to set the monitoring in Glassfish to at least "low" or preferably "high" for the "HTTP Service"

You will then receive the following MBeans: com.sun.appserv:name=http-listener-1,virtual-server=server,type=http-listener,category=monitor,server=<servername>

This contains all the stats you like, counts per HTTP status (200, 302, 304,40x)

Hope this helps!

[cfd]

Count per status... gee, I need to add some code to my agent :)

I'll check tomorrow, thanks for the info.

Christophe.

[cfd]

Hi there,

The load status is postponed to the next version with applications & logs monitoring...

So the current version will be in production in few days and gives this:
http://tbonefox.free.fr/monkey_web_console.jpg

modifications:
- data online directly from the monitor process (not fetched from the database)
- http jetty based
- alert data giving the level/timings of alerts sent to our system event console

next step:
- log monitoring (intelligent filtering SEVERE, stacks,...)
- application monitoring (enabled/disabled/...)
- user load monitoring

see you on next update ;)

Christophe.

edit> screenshot updated to show few errors.

[kjkoster]

Dear Christophe,

Wow, that really looks nice. You must be checking this every morning. :)

Kees Jan

[cfd]

Hi Kees Jan,

I check it everyday but it's not necessary, the monitor server sends the alerts to our sys admins console few seconds after the alert event.

This dashboard is mainly used by our Deploying Team to know the state of the appservers (permgen area).

Christophe

[kjkoster]

Dear Christophe,

Ah, so you just check it to see how cool it is then. :)

Kees Jan

[cfd]

Yep, to get the weather of the starting day :)