What is wrong

[cheraghi]

Hi
Our application is an internal app for university.
Below is some specifications :
Database size : 1.2 G mysql
Users : 100000 (each year grows up by 10000)
J2ee (struts 2 + hibernate)
JAVA_OPTS = -Xms1024m -Xmx8g -Xmn512m -XX:PermSize=512m -XX:MaxPermSize=1G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

This application works fine until 1 month ago.
After that, tomcat went to die periodically each 1 hour, then each 30 minutes and 10 minutes sometimes.

After changing tomcat parameters, this interval reach to 2 hours in normal and 30 minutes in peak usage hours of day.
Of course, at night there is no death.
Some graphs I attach to this post.
Please help me to find out the main problem.

Thanks

[Kees de Kooter]

Do you have an explanation for the high number of active hibernate sessions? Is there a massive amount of concurrent users? Or are hibernate sessions kept open for a very long time?

[cheraghi]

Thanks for reply
I delegate managing hibernate to Spring. And almost I sure sessions is not open for a long time.
High number of active session is because of number of concurrent users.

Thanks

[cheraghi]

Hi
2 new graphs I attached.
There are more info.
There are too many peaks and exactly when peaks occur, death occurs too.

Thanks

[Kees de Kooter]

Is it possible that the connection pool is exhausted? Can you tell us the configuration of your pool.

And can you check your logfiles (catalina.out, localhost.log). Check if there are exceptions just before the application dies.

[cheraghi]

Thanks for reply
Datasource : org.apache.commons.dbcp.BasicDataSource
session factory : org.springframework.orm.hibernate3.annotation.Anno tationSessionFactoryBean

I do not use pooling because I used pooling and there is no difference. So I decide to not use pooling to test in mode no-pooling.
And there are no exceptions in log files.
But after all, I guess strongly that problem is in layer between database and application (hibernate).
I see in graphs that when concurrent users increases , http sessions and hibernate sessions grows up. And technically, when a query is going to execute, hibernate open a session but does not open a database connection. So active sessions increases and accumulates.
I think there is deadlock in threads which manages database connections (pooling).

[Kees de Kooter]

Dear Cheragi,

org.apache.commons.dbcp.BasicDataSource is a database connection pool. Do you configure it within your application or is it configured in tomcat?

Please show us the configuration properties, search in files called server.xml or context.xml.

[cheraghi]

It's configured in application not tomcat.

applicationContext-dao.xml

<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.Anno tationSessionFactoryBean" >
<property name="dataSource" ref="dataSource"/>
<property name="configLocation" value="classpath:hibernate.cfg.xml"/>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
<prop key="hibernate.query.substitutions">true</prop>
<prop key="hibernate.cache.use_second_level_cache">false</prop>
<prop key="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</prop>
<prop key="hibernate.connection.release_mode">after_transaction</prop>
<prop key="hibernate.connection.autocommit">true</prop>
<prop key="hibernate.show_sql">false</prop>
<prop key="hibernate.format_sql">true</prop>
</props>
</property>
</bean>

datasource configuration is :
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close" >
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
</bean>

Yes BasicDataSource is a pool datasource but I do not configure session factory to use from pooled connection provider.

Thanks

[Kees de Kooter]

Based on your spring config your session factoring is using the bean named datasource, which really is a database connection pool.

Without further configuration dbcp datasources default to 8 maximum active connections.

I suggest you add the following parameter to the datasource in your spring config:

<property name="maxActive" value="250"/>

[cheraghi]

I tested before this.
I did set that to 300, but it did not help us.
But I change it again and test it for one day because configurations change a bit.
I will say result.

Thanks

[Kees de Kooter]

OK. Make sure that your database server allows you to open that number of connections.

[cheraghi]

Yes, we set that to 300 connections.

Thanks

[cheraghi]

Hi
I change maxActive in datasource config and everything is ok.
Our problems are :
1) Bad management of memory - changing garbage collection and memory configuration fix it
2) Connection pooling - After many changes to connection providers and datasources and their configurations, now everything is ok.

Thanks to people did answer me