1 问题现象
执行shutdown.sh之后，Web已经无法访问：
Tomcat关联java进程仍然存活：
#/home/apache-tomcat-7.0.34/bin/shutdown.sh2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesJdbccatalina.out报内存泄露警告：
Using CATALINA_BASE: /home/apache-tomcat-7.0.34
Using CATALINA_HOME: /home/apache-tomcat-7.0.34
Using CATALINA_TMPDIR: /home/apache-tomcat-7.0.34/temp
Using JRE_HOME: /home/jdk1.6.0_34
Using CLASSPATH: /home/apache-tomcat-7.0.34/bin/bootstrap.jar:/home/apache-tomcat-7.0.34/bin/tomcat-juli.jar
# ps aux | grep 'java.*tomcat' | grep -v grep
root 22465 15.3 8.6 1294300 346264 pts/0 Sl 14:55 1:08 /home/jdk1.6.0_34/bin/java -Djava.util.logging.config.file=/home/apache-tomcat-7.0.34/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/apache-tomcat-7.0.34/endorsed -classpath /home/apache-tomcat-7.0.34/bin/bootstrap.jar:/home/apache-tomcat-7.0.34/bin/tomcat-juli.jar -Dcatalina.base=/home/apache-tomcat-7.0.34 -Dcatalina.home=/home/apache-tomcat-7.0.34 -Djava.io.tmpdir=/home/apache-tomcat-7.0.34/temp org.apache.catalina.startup.Bootstrap start
严重: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Thread-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [thread-snatch-picture] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-0] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-1] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-3] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-4] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-5] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-6] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-7] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Heal-Session-Thread] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-3] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-4] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@1d97383]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@9a827]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@12e71f1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@855793]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@5164df]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2 问题分析
这个在windows下没有碰到过，因为此前跑Tomcat都是以服务而不是命令脚本的形式跑的，而且已经换了一个项目，所以暂时不考察windows下是否存在该问题。
难道是Tomcat版本问题？或者用带内存泄漏保护的Tomcat 7可以解决该问题？尝试将web应用跑在apache-tomcat-6.0.18、apache-tomcat-6.0.35、apache-tomcat-7.0.34，发现均存在无法shutdown.sh进程问题。
难道在CentOS（Linux）下，Tomcat无法用shutdown.sh停掉进程？显然不可能的。于是又把web应用从webapps中移走，换回tomcat自带的ROOT，果然启动后再shutdown.sh，查找不到原来的tomcat进程了，也就证明了是自己web应用的问题了。
3 解决方案
现在已经确定是web应用的问题了，所以可以提解决方案了。
3.1 Kill进程
忽略日志中的严重警告，因为这是关闭tomcat时候引起的，正常情况下不会发生这种内存泄露情况，而且Tomcat6.18以上版本的Tomcat已经做了内存泄露保护，交给Tomcat完成吧，我们只需要在shutdown.sh之后，补上一个kill -9 pid即可。要是嫌这样太麻烦了，可以如下这样改：
==============================bin/shutdown.sh
exec "$PRGDIR"/"$EXECUTABLE" stop -force "$@" #加上 -force
==============================bin/catalina.sh
在PRGDIR=`dirname "$PRG"`后面加上
if [ -z "$CATALINA_PID" ]; then
CATALINA_PID=$PRGDIR/CATALINA_PID
cat $CATALINA_PID
fi
3.2 web应用中关闭后台线程
shutdown.sh后存在tomcat僵尸进程的原因是shutdown.sh后web应用没有去关闭其启用的后台线程，所以处理的办法是监听到tomcat关闭事件时，关闭掉其启用的后台线程。
Pstree -p pid打印进程的线程树，查看shutdown.sh后未关闭的后台线程：
通过Java VisualVM查看shutdown.sh后未关闭的活动线程
在web应用中，如果有后台线程，这时用shutdown.sh是关闭不掉Tomcat的。我们需要重写相关的ServletContextListener的contextDestroyed(ServletContextEventarg0)方法，在它里面关闭后台线程。
但是，现在很多web应用都是由spring来管理bean的，实际情况下我们可能需要注入很多第三方jar中的bean。比如在applicationContext.xml中配置如下bean：
<bean id ="dataSource" class ="org.apache.commons.dbcp.BasicDataSource"destroy-method="close">
则在org.apache.commons.dbcp.BasicDataSource需要做好销毁数据库线程池的逻辑。
当然我们也可以自己设置销毁逻辑。在web.xml文件中增加监听器。
<listener>MySpecialListener实现如下：
<listener-class>com.mysite.MySpecialListener</listener-class>
</listener>
public class MySpecialListener extends ApplicationContextListener {
@Override
public void contextInitialized(ServletContextEvent sce) {
// On Application Startup, please…
// Usually I'll make a singleton in here, set up my pool, etc.
}
@Override
public void contextDestroyed(ServletContextEvent sce) {
// On Application Shutdown, please…
// 1. Go fetch that DataSource
Context initContext = new InitialContext();
Context envContext = (Context)initContext.lookup("java:/comp/env");
DataSource datasource = (DataSource)envContext.lookup("jdbc/database");
// 2. Call the close() method on it
datasource.close();
// 3. Deregister Driver
try {
java.sql.Driver mySqlDriver = DriverManager.getDriver("jdbc:mysql://localhost:3306/");
DriverManager.deregisterDriver(mySqlDriver);
} catch (SQLException ex) {
logger.info("Could not deregister driver:".concat(ex.getMessage()));
}
dataSource = null;
}
}