THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
- !! This page is under construction !!*
For some time Tomcat has had some means of protection against memory leaks when stopping or redeploying applications. This page tries to list them, and shows the situations where leaks can be detected and fixed.
...
Leak cause | Detected by tomcat | Fixed by tomcat | Possible enhancements | ||
>=6.0.24 | >= 67.0.246 | ||||
>=6.0.24 | >= 67.0.246 | ||||
Webapp class instance indirectly held through a ThreadLocal value | no | no | >= 7.0.6 | Renew threads in the worker pool | |
>=6.0.24 | >= 67.0.246 | ||||
>=6.0.24 | In 6.0.24-6.0.26 | Fix the application to stop the thread when the application is stoppedDetect child classloaders, don't stop | |||
ContextClassLoader / Threads spawned by classes loaded by the common classloader | >=6.0.24 | In 6.0.24-6.0.26 | Detect child classloaders, don't stop | but it may lead to problems. Optional from 6.0.27 with the | Fix the offending code (set the correct CCL when spawning the thread) |
no | >=6.0.24 pre-spawns some known offenders |
| |||
no | > 6.0.? . Disabled by default with tomcat 7 |
| |||
| > 6.0.? |
| |||
> 6.0.? | > 6.0.? |
| |||
| > 6.0.? |
|
...
In this particular case, the leak is detected , and a message is logged and . Tomcat 6.0.24 to 6.0.26 modify internal structures of the JDK (ThreadLocalMap) are modified to remove the reference to the ThreadLocal instance, but this is unsafe (see #48895) so that it became optional and disabled by default from 6.0.27. Starting with Tomcat 7.0.6, the threads of the pool are renewed so that the leak is safely fixed.
| No Format |
|---|
Mar 16, 2010 11:47:24 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [test.MyThreadLocal] (value [test.MyThreadLocal@4dbb9a58]) and a value of type [test.MyCounter] (value [test.MyCounter@57922f46]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. |
Note: this particular leak was actually already cured by previous versions of tomcat 6, because static references of classes loaded by the webappclassloader are nullified (see later).
...
If the servlet is invoked at least once, the webapp classloader would not be GCed when the app is stopped: since the classloader of ThreadScopedHolder is the common classloader, it remains forever which is as expected. But its ThreadLocal instance has a value bound to it (for the non-terminated thread(s) that served the sevlet), which is an instance of a class loaded by the webapp classloader...
Here again, tomcat >=6.0.24 will detect and fix the leak by removing this ThreadLocal reference from each Thread.:
| No Format |
|---|
Mar 17, 2010 10:23:13 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@44676e3f]) and a value of type [test.leak.threadlocal.value.MyCounter] (value [test.leak.threadlocal.value.MyCounter@62770d2e]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. |
...
We have more or less the same kind of leak as the previous one, but this time tomcat 6.0.24 does not detect the leak when stopping the application (and does not fix it). The problem is that when it inspects the entries of ThreadLocalMap, it checks whether either the key or the value is an instance of a class loaded by the webapp classloader. Here the key is an instance of ThreadLocal, and the value is an instance of java.util.ArrayList.
...
But it does not give any clue about what caused the leak, we would need to make a heapdump and analyse it with some tool like Eclipse MAT.
Tomcat 7.0.6 and later fix this leak by renewing threads in the pool.
| Anchor | ||||
|---|---|---|---|---|
|
...
If many threads were used to serve our leaking webapp, but after we stop it only a couple of threads are enough to serve other webapps, one could have some threads that are no longer used, waiting for some work. Since those threads are blocked, they have no interaction with their ThreadLocalMap (i.e. there's no ThreadLocal value bound to them or removed), so that there's no opportunity to expungeStaleEntries().
Tomcat 6.0.24-6.0.26 "speeds up" the removal of stale entries (and thus fixes the pseudo-leak), by calling expungeStaleEntries() for each thread that has some stale entries. Since it's not thread-safe, it has been made optional and disabled by default from 6.0.27.
Tomcat 7.0.6 and later fix the problem by renewing threads in the pool.
Threads ContextClassLoader
...
So the leak is fixed, but unfortunately there's a side effect : it broke webapp B eviction timer.
(TODO: file a bug report to make the clearReferencesStopTimerThread optional and disabled by default)
Note: as of That's why stopping TimerThread has been made optional from 6.0.24, by default tomcat stops threads of class java.util.TimerThread whose context classloader is the WebAppClassLoader of the app being stopped. It does not stop other threads, it only warns about them. It can try to stop them if the clearReferencesStopThreads option of the standard context is set to true.Anchor
| Anchor | ||||
|---|---|---|---|---|
|
Threads spawned Threads spawned by JRE classes
Just like third-party libraries may spawn threads that provoke leaks, some JRE classes also spawn threads that inherit from the current class loader and thus provoke leaks.
Instead of trying to stop such threads, tomcat prefers to force the creation of such threads when the container is started, before webapps are started. The JreMemoryLeakPreventionListener does it for a few known offenders in the JRE.
...
| Anchor | ||||
|---|---|---|---|---|
|
static class variables
When an app is stopped, Tomcat (even before 6.0.24 detects leaks caused by ThreadLocal}}s and Threads context classloader only by checking for the current {{WebAppClassLoader. If a child classloader is involved, the leak is not detected. That should be improved in a future release.Anchor
static class variables
When an app is stopped, Tomcat (even before 6.0.24) ) nullifies the value of all static class variables of classes loaded by the WebAppClassLoader. In some cases, it may fix a classloader leak (for example because of a custom ThreadLocal class, see above), but even if we still have a leak, it may decrease the amount of memory lost:
...
Related issues
48837 - Memory leaks protection does not cure leaks triggered by JSP pages codeAnchor 4883749159 48837 Anchor 48895 48895 48895 - WebAppClassLoader.clearThreadLocalMap() concurrency issues
49159 - Improve ThreadLocal memory leak clean-up49159 - Sun bug 4957990 - In some cases the Server JVM fails to collect classloaders. According to this page it should have been fixed with java 6u16 but actually it was not. It seems to be fixed with 6u21 (documented here and verified by the author of this wiki page).
- Sun bug 6916498 - An exception can keep a classloader in memory if the stack trace that was recorded when it was created contains a reference to one of its classes. Fixes were done in Tomcat for its own classes that had this issue (see BZ 50460), but some library or JRE code may still create a leak that is undetected by tools because of this JVM bug. See also BZ 53936 for a workaround that you can implement if you are unable to fix a buggy library.
...