- !! This page is under construction !!*
For some time Tomcat has had some means of protection against memory leaks when stopping or redeploying applications. This page tries to list them, and shows the situations where leaks can be detected and fixed.
...
Leak cause | Detected by tomcat | Fixed by tomcat | Possible enhancements | ||
>=6.0.24 | >= 67.0.246 | ||||
>=6.0.24 | >= 67.0.246 | ||||
Webapp class instance indirectly held through a ThreadLocal value | no | no | >= 7.0.6 | Renew threads in the worker pool | |
>=6.0.24 | >= 67.0.246 | ||||
>=6.0.24 | In 6.0.24-6.0.26 | Fix the application to stop the thread when the application is stoppedDetect child classloaders, don't stop TimerThread by default | |||
ContextClassLoader / Threads spawned by classes loaded by the common classloader | >=6.0.24 | In 6.0.24-6.0.26 | Detect child classloaders, don't stop TimerThread by default | but it may lead to problems. Optional from 6.0.27 with the | Fix the offending code (set the correct CCL when spawning the thread) |
no | >=6.0.24 pre-spawns some known offenders |
| |||
no | > 6.0.? . Disabled by default with tomcat 7 |
| |||
| > 6.0.? |
| |||
> 6.0.? | > 6.0.? |
| |||
| > 6.0.? |
|
...
In this particular case, the leak is detected , and a message is logged and . Tomcat 6.0.24 to 6.0.26 modify internal structures of the JDK (ThreadLocalMap
) are modified to remove the reference to the ThreadLocal
instance, but this is unsafe (see #48895) so that it became optional and disabled by default from 6.0.27. Starting with Tomcat 7.0.6, the threads of the pool are renewed so that the leak is safely fixed.
No Format |
---|
Mar 16, 2010 11:47:24 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [test.MyThreadLocal] (value [test.MyThreadLocal@4dbb9a58]) and a value of type [test.MyCounter] (value [test.MyCounter@57922f46]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. |
Note: this particular leak was actually already cured by previous versions of tomcat 6, because static references of classes loaded by the webappclassloader are nullified (see later).
...
If the servlet is invoked at least once, the webapp classloader would not be GCed when the app is stopped: since the classloader of ThreadScopedHolder
is the common classloader, it remains forever which is as expected. But its ThreadLocal
instance has a value bound to it (for the non-terminated thread(s) that served the sevlet), which is an instance of a class loaded by the webapp classloader...
Here again, tomcat >=6.0.24 will detect and fix the leak by removing this ThreadLocal
reference from each Thread.:
No Format |
---|
Mar 17, 2010 10:23:13 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@44676e3f]) and a value of type [test.leak.threadlocal.value.MyCounter] (value [test.leak.threadlocal.value.MyCounter@62770d2e]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. |
...
We have more or less the same kind of leak as the previous one, but this time tomcat 6.0.24 does not detect the leak when stopping the application (and does not fix it). The problem is that when it inspects the entries of ThreadLocalMap
, it checks whether either the key or the value is an instance of a class loaded by the webapp classloader. Here the key is an instance of ThreadLocal
, and the value is an instance of java.util.ArrayList
.
...
But it does not give any clue about what caused the leak, we would need to make a heapdump and analyse it with some tool like Eclipse MAT.
Tomcat 7.0.6 and later fix this leak by renewing threads in the pool.
Anchor | ||||
---|---|---|---|---|
|
...
If many threads were used to serve our leaking webapp, but after we stop it only a couple of threads are enough to serve other webapps, one could have some threads that are no longer used, waiting for some work. Since those threads are blocked, they have no interaction with their ThreadLocalMap
(i.e. there's no ThreadLocal
value bound to them or removed), so that there's no opportunity to expungeStaleEntries()
.
Tomcat 6.0.24-6.0.26 "speeds up" the removal of stale entries (and thus fixes the pseudo-leak), by calling expungeStaleEntries()
for each thread that has some stale entries. Since it's not thread-safe, it has been made optional and disabled by default from 6.0.27.
Tomcat 7.0.6 and later fix the problem by renewing threads in the pool.
Threads ContextClassLoader
...
So the leak is fixed, but unfortunately there's a side effect : it broke webapp B eviction timer. Note: as of That's why stopping TimerThread
has been made optional from 6.0.24, by default tomcat stops threads of class java.util.TimerThread
whose context classloader is the WebAppClassLoader
of the app being stopped. It does not stop other threads, it only warns about them. It can try to stop them if the clearReferencesStopThreads
option of the standard context is set to true. Anchor
Threads spawned by JRE classes
Just like third-party libraries may spawn threads that provoke leaks, some JRE classes also spawn threads that inherit from the current class loader and thus provoke leaks.
Instead of trying to stop such threads, tomcat prefers to force the creation of such threads when the container is started, before webapps are started. The JreMemoryLeakPreventionListener
does it for a few known offenders in the JRE.
Child classloaders
When an app is stopped, Tomcat 6.0.24 detects leaks caused by ThreadLocal}}s and Threads context classloader only by checking for the current {{WebAppClassLoader
. If a child classloader is involved, the leak is not detected. That should be improved in a future release.
...
static class variables
When an app is stopped, Tomcat (even before 6.0.24) nullifies the value of all static class variables of classes loaded by the WebAppClassLoader
. In some cases, it may fix a classloader leak (for example because of a custom ThreadLocal
class, see above), but even if we still have a leak, it may decrease the amount of memory lost:
Imagine a class with the following variable :
No Format |
---|
private final static byte[] BUFFER = new byte[1024*1024]; //1MB buffer
|
Normally, the 1MB buffer should be freed when the app is stopped, but only if the classloader itself can be garbage-collected. Since there are still possibilities to have a leak of the classloader, clearing the BUFFER variable allows to recover 1MB of memory.
...
LogFactory
to be completed
JavaBean Introspector cache
Tomcat calls java.beans.Introspector.flushCaches();
when an app is stoppped.
...
JDBC driver registration
If a webapp contains a JDBC driver (e.g. in WEB-INF/lib), the driver will be registered with the DriverManager
when it is first used. When the application is stopped, the driver should be deregistered with DriverManager
to avoid a classloader leak. Since applications usually forget this, tomcat helps by deregistering the driver.
...
RMI target
to be completed
References
Related issues
...
27.
Anchor | ||||
---|---|---|---|---|
|
Threads spawned by JRE classes
Just like third-party libraries may spawn threads that provoke leaks, some JRE classes also spawn threads that inherit from the current class loader and thus provoke leaks.
Instead of trying to stop such threads, tomcat prefers to force the creation of such threads when the container is started, before webapps are started. The JreMemoryLeakPreventionListener
does it for a few known offenders in the JRE.
Anchor | ||||
---|---|---|---|---|
|
static class variables
When an app is stopped, Tomcat (even before 6.0.24) nullifies the value of all static class variables of classes loaded by the WebAppClassLoader
. In some cases, it may fix a classloader leak (for example because of a custom ThreadLocal
class, see above), but even if we still have a leak, it may decrease the amount of memory lost:
Imagine a class with the following variable :
No Format |
---|
private final static byte[] BUFFER = new byte[1024*1024]; //1MB buffer
|
Normally, the 1MB buffer should be freed when the app is stopped, but only if the classloader itself can be garbage-collected. Since there are still possibilities to have a leak of the classloader, clearing the BUFFER variable allows to recover 1MB of memory.
Anchor | ||||
---|---|---|---|---|
|
LogFactory
to be completed
JavaBean Introspector cache
Tomcat calls java.beans.Introspector.flushCaches();
when an app is stoppped.
Anchor | ||||
---|---|---|---|---|
|
JDBC driver registration
If a webapp contains a JDBC driver (e.g. in WEB-INF/lib), the driver will be registered with the DriverManager
when it is first used. When the application is stopped, the driver should be deregistered with DriverManager
to avoid a classloader leak. Since applications usually forget this, tomcat helps by deregistering the driver.
Anchor | ||||
---|---|---|---|---|
|
RMI target
to be completed
...
References
Related issues
49159 - Improve ThreadLocal memory leak clean-upAnchor 49159 49159 - Sun bug 4957990 - In some cases the Server JVM fails to collect classloaders. According to this page it should have been fixed with java 6u16 but actually it was not. It seems to be fixed with 6u21 (documented here and verified by the author of this wiki page).
- Sun bug 6916498 - An exception can keep a classloader in memory if the stack trace that was recorded when it was created contains a reference to one of its classes. Fixes were done in Tomcat for its own classes that had this issue (see BZ 50460), but some library or JRE code may still create a leak that is undetected by tools because of this JVM bug. See also BZ 53936 for a workaround that you can implement if you are unable to fix a buggy library.
...
...