kjkoster
31-08-2008, 20:44
Dear All,
Earlier I asked if there is such a thing as too much memory? (http://java-monitor.com/forum/showthread.php?t=6) Well, it turns out there may be such a problem, but it's not as clear-cut as simply too much. On the JBoss forums I ran into a post that talks about a stalling JVM (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=141155). I remember that this issue came up a few years ago in one of the other systems of my then-employer.
Imagine you have an application server running on a dedicated box with loads of RAM. Naturally, you decide to put that memory to good use and allocate it all to the JVM of your application server. The JVM slowly fills up that memory with objects, happily running it all the way to 70% full. Then it decides that now is a good time for a major garbage collect and it fires of the expensive garbage collector. Unfortunately, almost all of the objects are unused and can be collected, resulting in GC times that run into the tens of seconds. Ironically, this is precisely the type of behaviour that Sun's more modern garbage collectors have been designed to avoid. How can it be that this is still happening?
Research has shown that objects in a JVM tend to be either really short-lived, or they tend to be very long lived. In-between-lived (I made that up, don't worry) objects are surprisingly rare. Thus, the JVM has roughly two pools of memory: a young generation for the short-lived objects and an old generation for the longer lived ones.
As an aside, the precise name of that memory pool depends on the type of garbage collectors that are in use. We'll just use the name 'young' and 'old' and leave you to find the precise names using jconsole. We also ignore the code cache and the permanent generations for this discussion.
Each object first comes to being in the young generation's pool. The objects get created, hold some data and do a bit of work. Pretty quickly they are no longer needed and are left for the garbage collector to clean up. Objects that survive a few cycles from the garbage collector are expected to live longer, and are transferred into the old generation's memory pool. From what [URL="http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4172629#4172349"Peter said[/URL], I understand that objects that live longer than 32 garbage collector cycles are considered old and move on into the next pool.
Now imagine that the objects all live a little bit too long for the young generation. Maybe they live for 40 garbage collector cycles, instead of the expected 20-25. Instead of dying and being collected in the young generation, they die in the old generation. The old generation is usually a lot larger than the young generation. The garbage collector that is responsible for the old generation is a lot more expensive than the one working in the old generation. Now all these dead 40-somethings are piling up in the old generation and they stay there for a long time. The garbage collector for the old generation does not have to run until it is about 70% full, so it will leave things as they are.
And when the old generation is finally hit by a garbage collect, almost all of it is dead wood. The garbage collector has a lot of bookkeeping to do to clean it all out. So much, that for a sufficiently large old generation it takes a few seconds to do so. This results in the pauses you may have been seeing.
The fix is quite simple, once you know what is going on. You can try to increase the size of the young generation. This effectively raises the age of the objects that still live in that pool. That way, short lived objects die in the young generation and not in the old generation, making it cheaper to clean up their allocated memory. Of course, you don't want to increase the size of the young generation too much, because that would cause the garbage collector in that pool to start taking too much time and maybe even noticeably pausing the JVM.
The real fix is of course to reduce the number of short-lived objects you create in the code. :-)
Unfortunately for me, this is all theory. I cannot speak from experience with either stalling JVM's or tuning the young generations or even parallel garbage collectors. Do any of you have such experience? How do you tune your young generations and do you use the parallel garbage collector?
Does anyone have any good links to papers explaining garbage collector behaviour? In particular, does anyone have links to research that examines the longevity of objects in a JVM?
Kees Jan
Earlier I asked if there is such a thing as too much memory? (http://java-monitor.com/forum/showthread.php?t=6) Well, it turns out there may be such a problem, but it's not as clear-cut as simply too much. On the JBoss forums I ran into a post that talks about a stalling JVM (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=141155). I remember that this issue came up a few years ago in one of the other systems of my then-employer.
Imagine you have an application server running on a dedicated box with loads of RAM. Naturally, you decide to put that memory to good use and allocate it all to the JVM of your application server. The JVM slowly fills up that memory with objects, happily running it all the way to 70% full. Then it decides that now is a good time for a major garbage collect and it fires of the expensive garbage collector. Unfortunately, almost all of the objects are unused and can be collected, resulting in GC times that run into the tens of seconds. Ironically, this is precisely the type of behaviour that Sun's more modern garbage collectors have been designed to avoid. How can it be that this is still happening?
Research has shown that objects in a JVM tend to be either really short-lived, or they tend to be very long lived. In-between-lived (I made that up, don't worry) objects are surprisingly rare. Thus, the JVM has roughly two pools of memory: a young generation for the short-lived objects and an old generation for the longer lived ones.
As an aside, the precise name of that memory pool depends on the type of garbage collectors that are in use. We'll just use the name 'young' and 'old' and leave you to find the precise names using jconsole. We also ignore the code cache and the permanent generations for this discussion.
Each object first comes to being in the young generation's pool. The objects get created, hold some data and do a bit of work. Pretty quickly they are no longer needed and are left for the garbage collector to clean up. Objects that survive a few cycles from the garbage collector are expected to live longer, and are transferred into the old generation's memory pool. From what [URL="http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4172629#4172349"Peter said[/URL], I understand that objects that live longer than 32 garbage collector cycles are considered old and move on into the next pool.
Now imagine that the objects all live a little bit too long for the young generation. Maybe they live for 40 garbage collector cycles, instead of the expected 20-25. Instead of dying and being collected in the young generation, they die in the old generation. The old generation is usually a lot larger than the young generation. The garbage collector that is responsible for the old generation is a lot more expensive than the one working in the old generation. Now all these dead 40-somethings are piling up in the old generation and they stay there for a long time. The garbage collector for the old generation does not have to run until it is about 70% full, so it will leave things as they are.
And when the old generation is finally hit by a garbage collect, almost all of it is dead wood. The garbage collector has a lot of bookkeeping to do to clean it all out. So much, that for a sufficiently large old generation it takes a few seconds to do so. This results in the pauses you may have been seeing.
The fix is quite simple, once you know what is going on. You can try to increase the size of the young generation. This effectively raises the age of the objects that still live in that pool. That way, short lived objects die in the young generation and not in the old generation, making it cheaper to clean up their allocated memory. Of course, you don't want to increase the size of the young generation too much, because that would cause the garbage collector in that pool to start taking too much time and maybe even noticeably pausing the JVM.
The real fix is of course to reduce the number of short-lived objects you create in the code. :-)
Unfortunately for me, this is all theory. I cannot speak from experience with either stalling JVM's or tuning the young generations or even parallel garbage collectors. Do any of you have such experience? How do you tune your young generations and do you use the parallel garbage collector?
Does anyone have any good links to papers explaining garbage collector behaviour? In particular, does anyone have links to research that examines the longevity of objects in a JVM?
Kees Jan