一.目的
利用apache的mod_jk模块,实现tomcat集群服务器的负载均衡以及会话复制,这里用到了<Cluster>.
二.环境
1.基础:3台主机,系统Centos6.5,4G内存,50G硬盘. yum源已更换为阿里源(如何更换可参考博客的另外一篇文章CDN)
2.软件:development tools,jdk-7u9-linux-x64.rpm;源码编译软件apr-util-1.5.4.tar.gz,apr-1.5.2.tar.gz,tomcat-connectors-1.2.40-src.tar.gz,httpd-2.4.16.tar.gz
3.源码包下载地址:可以去官网下载,或者在此网盘内下载:点击链接
4.拓扑图
5.现在就可以安装部署了。
三.部署安装
Apache环节:
1.编译安装apr
1 [root@apache down]# tar xf apr-1.5.2.tar.gz 2 [root@apache down]# cd apr-1.5.2 3 [root@apache apr-1.5.2]# ./configure --prefix=/usr/local/apr --disable-ipv6 4 [root@apache apr-1.5.2]# make && make install
2.编译安装apr-util
1 [root@apache down]# tar xf apr-util-1.5.4.tar.gz 2 [root@apache down]# cd apr-util-1.5.4 3 [root@apache apr-util-1.5.4]# ./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr 4 [root@apache apr-util-1.5.4]# make && make install
3.编译安装httpd
1 [root@apache ~]# yum -y install pcre-devel openssl-devel (编译安装前,先安装这两个软件,解决编译httpd时的依赖问题)
解压httpd,并编译安装。(如果在编译出错情况,请仔细排查错误,记得重新编译时,需要make clean 清除一下)
1 [root@apache down]# tar xf httpd-2.4.16.tar.gz 2 [root@apache down]# cd httpd-2.4.16 3 [root@apache httpd-2.4.16]# ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util --enable-mpms-shared=all --with-mpm=event --enable-proxy --enable-proxy-http --enable-proxy-ajp --enable-proxy-balancer --enable-lbmethod-heartbeat --enable-heartbeat --enable-slotmem-shm --enable-slotmem-plain --enable-watchdog 4 [root@apache httpd-2.4.16]# make && make install
为apache 提供httpd脚本,并加入系统服务,设为开机启动,方便以后的服务管理,文件路径 /etc/rc.d/init.d/httpd
[root@apache httpd-2.4.16]# vim /etc/rc.d/init.d/httpd #!/bin/bash # # httpd Startup script for the Apache HTTP Server # # chkconfig: - 85 15 # description: Apache is a World Wide Web server. It is used to serve \ # HTML files and CGI. # processname: httpd # config: /etc/httpd/conf/httpd.conf # config: /etc/sysconfig/httpd # pidfile: /var/run/httpd.pid # Source function library. . /etc/rc.d/init.d/functions if [ -f /etc/sysconfig/httpd ]; then . /etc/sysconfig/httpd fi # Start httpd in the C locale by default. HTTPD_LANG=${HTTPD_LANG-"C"} # This will prevent initlog from swallowing up a pass-phrase prompt if # mod_ssl needs a pass-phrase from the user. INITLOG_ARGS="" # Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server # with the thread-based "worker" MPM; BE WARNED that some modules may not # work correctly with a thread-based MPM; notably PHP will refuse to start. # Path to the apachectl script, server binary, and short-form for messages. apachectl=/usr/local/apache/bin/apachectl httpd=${HTTPD-/usr/local/apache/bin/httpd} prog=httpd pidfile=${PIDFILE-/var/run/httpd.pid} lockfile=${LOCKFILE-/var/lock/subsys/httpd} RETVAL=0 start() { echo -n $"Starting $prog: " LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS RETVAL=$? echo [ $RETVAL = 0 ] && touch ${lockfile} return $RETVAL } stop() { echo -n $"Stopping $prog: " killproc -p ${pidfile} -d 10 $httpd RETVAL=$? echo [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile} } reload() { echo -n $"Reloading $prog: " if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then RETVAL=$? echo $"not reloading due to configuration syntax error" failure $"not reloading $httpd due to configuration syntax error" else killproc -p ${pidfile} $httpd -HUP RETVAL=$? fi echo } # See how we were called. case "$1" in start) start ;; stop) stop ;; status) status -p ${pidfile} $httpd RETVAL=$? ;; restart) ;; condrestart) if [ -f ${pidfile} ] ; then stop start fi ;; reload) reload ;; graceful|help|configtest|fullstatus) $apachectl $@ RETVAL=$? ;; *) echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}" exit 1 esac exit $RETVAL
脚本编辑完成后,添加为系统服务
1 [root@apache httpd-2.4.16]# chkconfig --add httpd 2 [root@apache httpd-2.4.16]# chkconfig httpd on
现在,去修改一下httpd.conf文件,开启两个模块的支持,修改之前,我们先来备份一下httpd.conf
[root@apache ~]# cd /etc/httpd/ (因为在编译httpd时,将配置文件的目录即conf下的文件放在了/etc/httpd/下面,所以配置需要去这里。) [root@apache httpd]# cp httpd.conf httpd.conf-10-1.bak [root@apache httpd]# ls extra httpd.conf httpd.conf-10-1.bak magic mime.types original
1 [root@apache httpd]# vim httpd.conf 2 #找到以下两行,取消注释,如果找不到,手动添加进去亦可 3 LoadModule slotmem_shm_module modules/mod_slotmem_shm.so 4 LoadModule slotmem_plain_module modules/mod_slotmem_plain.so
配置完成后,现在检查下配置文件,并启动httpd
[root@apache httpd]# service httpd configtest
Syntax OK
[root@apache httpd]# service httpd start
访问网站,http://192.168.40.184,出现It Works!字样,即表示成功。下面是安装两台tomcat的记录。
Tomcat环节:
1.tomcatA软件的安装。先来安装下jdk。
1 [root@tomcat1 download]# rpm -ivh jdk-7u79-linux-x64.rpm
添加环境变量的文件,新建java.sh文件,内容如下
[root@tomcat1 download]# vim /etc/profile.d/java.sh export JAVA_HOME=/usr/java/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin
测试一下
1 [root@tomcat1 download]# java -version 2 java version "1.7.0_79" 3 Java(TM) SE Runtime Environment (build 1.7.0_79-b15) 4 Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
2.安装tomcat,解压到/usr/local下
1 [root@tomcat1 download]# tar xf apache-tomcat-7.0.42.tar.gz -C /usr/local/ && cd /usr/local 2 [root@tomcat1 local]# ln -sv apache-tomcat-8.0.26 tomcat 3 [root@tomcat1 local]# ll 4 lrwxrwxrwx 1 root root 21 Oct 1 17:30 tomcat -> apache-tomcat-8.0.26/
现在配置tomcat的环境变量/etc/profile.d/tomcat.sh
1 [root@tomcat1 local]# vim /etc/profile.d/tomcat.sh 2 3 export CATALINA_HOME=/usr/local/tomcat 4 export PATH=$PATH:$CATALINA_HOME/bin
Ctrl + d 键注销一下,再次登陆,试一下命令
[root@tomcat1 local]# catalina.sh version Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/java/jdk1.7.0_79 Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar Server version: Apache Tomcat/8.0.26 Server built: Aug 18 2015 11:38:37 UTC Server number: 8.0.26.0 OS Name: Linux OS Version: 2.6.32-431.el6.x86_64 Architecture: amd64 JVM Version: 1.7.0_79-b15 JVM Vendor: Oracle Corporation
方便tomcat服务的管理,我们添加一个脚本,并加入系统服务
1 [root@tomcat1 local]# vim /etc/init.d/tomcat 2 3 #!/bin/sh 4 # Tomcat init script for Linux. 5 # 6 # chkconfig: 2345 96 14 7 # description: The Apache Tomcat servlet/JSP container. 8 CATALINA_HOME=/usr/local/tomcat #注意你的脚本路径 9 export CATALINA_HOME 10 # export CATALINA_OPTS="-Xms128m -Xmx256m" 11 exec $CATALINA_HOME/bin/catalina.sh $* 12 13 [root@tomcat1 local]# chmod +x /etc/init.d/tomcat 14 [root@tomcat1 local]# chkconfig --add tomcat 15 [root@tomcat1 local]# chkconfig tomcat on 16 #之后启动一下,测试脚本是否正常 17 18 [root@tomcat1 local]# service tomcat start 19 Using CATALINA_BASE: /usr/local/tomcat 20 Using CATALINA_HOME: /usr/local/tomcat 21 Using CATALINA_TMPDIR: /usr/local/tomcat/temp 22 Using JRE_HOME: /usr 23 Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar 24 Tomcat started.
检查一下启动的端口号,正常应该会有以下几个监听的端口:8080,8005,然后可以打开浏览器,访问一下http://192.168.40.178.可以看到小猫咪。安装tomcat完成,现在就去修改配置文件,添加一个虚拟主机。
3.虚拟主机配置,文件路径/usr/local/tomcat/conf/server.xml,修改之前先备份此文件。将默认的8080端口修改成80端口,在Engine中,将默认主机localhost修改成www.test.com,并在后面追加一句 jvmRoute = “TomcatA”,最后在文件倒数第三行前(即</Engine>上方)添加一个虚拟主机,如下标红字体。
[root@tomcat1 local]# vim /usr/local/tomcat/conf/server.xml <Connector port="80" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> <Engine name="Catalina" defaultHost="www.test.com" jvmRoute="TomcatA"> <Host name="www.test.com" appBase="/web" unpackWARs="true" autoDeploy="true"> <Context path="/" docBase="webapp" reloadable="true"/> </Host> </Engine>
现在检查一下tomcat的配置文件,检查之前需要先停止tomcat,不然会报错。
[root@tomcat1 local]# service tomcat stop [root@tomcat1 local]# service tomcat configtest .... Oct 02, 2015 4:51:35 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2130 ms
创建虚拟主机文件文档目录并创建测试网页
[root@tomcat1 local]# mkdir -p /web/webapp/ [root@tomcat1 local]# cd /web/webapp/ [root@tomcat1 webapp]# vim index.jsp <%@ page language="java" %> <html> <head><title>TomcatA</title></head> <body> <h1><font color="red">TomcatA </h1> <table align="centre" border="1"> <tr> <td>Session ID</td> <% session.setAttribute("abc","abc"); %> <td><%= session.getId() %></td> </tr> <tr> <td>Created on</td> <td><%= session.getCreationTime() %></td> </tr> </table> </body> </html> [root@tomcat1 webapp]# service tomcat start
启动服务后,可以打开网站,http://192.168.40.178,出现以下说明配置成功了。
另外一台服务器TomcatB也要按照上述的操作部署。一模一样的去部署。仔细,每个环节都不能少了。(注意,唯一不同的是tomcatB服务器的server.xml中的jvmRoute填写的是TomcatB)
四.利用mod_jk实现负载均衡
mod_jk 是一个工作在apache端基于AJP协议与Tomcat通信的连接器,是apache的一个模块,是AJP协议的客户端。(服务端是Tomcat的AJP连接器)
1.下面安装tomcat-connectors
1 [root@apache down]# tar xf tomcat-connectors-1.2.40-src.tar.gz 2 [root@apache down]# cd tomcat-connectors-1.2.40-src/native/ 3 [root@apache native]# ./configure --with-apxs=/usr/local/apache/bin/apxs 4 [root@apache native]# make && make install
2.模块安装完成后,我们可以查看一下。ls /usr/local/apache/modules/ | grep mod_jk .为了便于管理与mod_jk模块相关的配置,在此使用一个专门的配置文件/etc/httpd/extra/httpd-jk.conf来保存相关配置,内容如下(需要新建):
[root@apache native]# vim /etc/httpd/extra/httpd-jk.conf #Load the mod_jk LoadModule jk_module modules/mod_jk.so #加载mod_jk模块 JkWorkersFile /etc/httpd/extra/workers.properties #配置文件位置 JkLogFile logs/mod_jk.log #日志 JkLogLevel debug #日志级别 JkMount /* lbcluster # 负载均衡器名称 JkMount /jkstatus/ stat1 #(最后单词是个数字1) #状态信息
然后去编辑/etc/httpd/extra/workers.properties文件,内容如下所示:
[root@apache native]# vim /etc/httpd/extra/workers.properties worker.list=lbcluster,stat1 #列表信息 worker.TomcatA.port=8009 #TomcatA实例端口 worker.TomcatA.host=192.168.40.178 #TomcatA主机 worker.TomcatA.type=ajp13 #支持协议 worker.TomcatA.lbfactor=1 #负载均衡权重为1 worker.TomcatB.host=192.168.40.180 worker.TomcatB.port=8009 worker.TomcatB.type=ajp13 worker.TomcatB.lbfactor=1 worker.lbcluster.type = lb #负载均衡work,lb内置类 worker.lbcluster.sticky_session = 0 #会话是否绑定 worker.lbcluster.balance_workers = TomcatA,TomcatB #集群中实例 worker.stat1.type = status #状态信息
现在去修改apache的配置文件,引用上一步我们写入的mod_jk的配置文件,并注释一条信息,如下所示。
[root@apache native]# vim /etc/httpd/httpd.conf #DocumentRoot "/usr/local/apache/htdocs" #在文件的末尾添加下面这行内容 Include /etc/httpd/extra/httpd-jk.conf
下面重启一下httpd服务器,并先停止后端的TomcatA和TomcatB服务器,在启动服务(没有重启功能)。然后打开浏览器,进入测试,访问http://192.168.40.184,这里是apache的网址。刷新网页,则会来回跳转到TomcatA 和TomcatB后端服务器上。如下所示:
通过mod_jk提供的接口,可以查看群集的状态信息。浏览器访问http://192.168.40.184/jkstatus,如下图所示
五.DeltaManager实现session保持
Tomcat 配置
1.修改TomcatA的server.xml配置文件,放在倒数第三行</Engine>上,此处放置的位置定义在Engine容器中,则是对所有主机均启用集群功能,如果定义在某Host主机中,则表示近对此主机启用集群功能。
注意事项:有的文章里配置文件里有这句话:<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> ,此选项在tomcat8中被移除了。所以如果你的版本是tomcat8的话,需要注意一下
[root@tomcat1 conf]# vim server.xml <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.40.178" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="/"/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster>
2.修改TomcatB的server.xml文件,同样在倒数第三行的</Engine>上方添加以下内容。
[root@tomcat2 conf]# vim server.xml <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.40.180" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="/"/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster>
分别在tomcatA和tomcatB上 执行以下命令测试,检测配置文件。
[root@tomcat2 conf]# service tomcat configtest ..... Oct 02, 2015 6:21:59 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2105 ms
此外,所有启用集群功能的web应用程序,其web.xml中都必须添加<distributable/>才能实现集群功能。如果某web应用程序没有自己的web.xml,也可以通过复制默认的web.xml到其WEB-INF目录中实现。
3.修改web.xml
TomcatA:
1 [root@tomcat1 ~]# vim /usr/local/tomcat/conf/web.xml 2 <web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" 3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 4 xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee 5 http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" 6 version="3.1"> 7 8 <distributable/> (只需要添加这一句话即可) 9 10 [root@tomcat1 ~]# mkdir -p /web/webapp/WEB-INF && cp /usr/local/tomcat/conf/web.xml /web/webapp/WEB-INF/
TomcatB:
[root@tomcat2 ~]# vim /usr/local/tomcat/conf/web.xml <web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" version="3.1"> <distributable/> [root@tomcat2 ~]# mkdir -p /web/webapp/WEB-INF && cp /usr/local/tomcat/conf/web.xml /web/webapp/WEB-INF/
4.启动tomcat服务器
TomcatB:
1 [root@tomcat1 ~]# service tomcat start 2 [root@tomcat1 ~]# netstat -aultnp 3 Active Internet connections (servers and established) 4 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 5 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 20397/sshd 6 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1240/master 7 tcp 0 52 192.168.40.178:22 192.168.30.192:49761 ESTABLISHED 30625/sshd 8 tcp 0 0 :::80 :::* LISTEN 32724/java 9 tcp 0 0 :::22 :::* LISTEN 20397/sshd 10 tcp 0 0 ::1:25 :::* LISTEN 1240/master 11 tcp 0 0 ::ffff:192.168.40.178:4000 :::* LISTEN 32724/java 12 tcp 0 0 ::ffff:127.0.0.1:8005 :::* LISTEN 32724/java 13 tcp 0 0 :::8009 :::* LISTEN 32724/java 14 tcp 0 0 ::ffff:192.168.40.178:43142 ::ffff:192.168.40.180:4000 TIME_WAIT - 15 tcp 0 0 ::ffff:192.168.40.178:43143 ::ffff:192.168.40.180:4000 ESTABLISHED 32724/java 16 tcp 0 0 ::ffff:192.168.40.178:4000 ::ffff:192.168.40.180:48954 ESTABLISHED 32724/java 17 udp 0 0 :::45564 :::* 32724/java
TomcatB:
[root@tomcat2 ~]# service tomcat start [root@tomcat2 ~]# netstat -altpnu Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1159/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1235/master tcp 0 52 192.168.40.180:22 192.168.30.192:49762 ESTABLISHED 24958/sshd tcp 0 0 :::80 :::* LISTEN 26540/java tcp 0 0 :::22 :::* LISTEN 1159/sshd tcp 0 0 ::1:25 :::* LISTEN 1235/master tcp 0 0 ::ffff:192.168.40.180:4000 :::* LISTEN 26540/java tcp 0 0 ::ffff:127.0.0.1:8005 :::* LISTEN 26540/java tcp 0 0 :::8009 :::* LISTEN 26540/java tcp 0 0 ::ffff:192.168.40.180:4000 ::ffff:192.168.40.178:43143 ESTABLISHED 26540/java tcp 0 0 ::ffff:192.168.40.180:48954 ::ffff:192.168.40.178:4000 ESTABLISHED 26540/java udp 0 0 :::45564 :::* 26540/java
5.现在打开浏览器,访问http://192.168.40.184,进行验证。
无论如何刷新,图中的SessionID都是不变的。
通过以上的观察,我们tomcat集群负载功能已经实现,并且配置了DeltaManager得到session的复制。
六.总结
通过以上的安装配置,以及过程中遇到的故障的解决,对apache的以及tomcat得到一些了解和认识。编译安装虽说很麻烦,确实很容易出错。。。。
本文大多数参考freeloda的博客,在原文基础上加入了一点点自己的修改,以及最新版软件使用时的不兼容。原文: “Share your knowledge …” 博客,请务必保留此出处http://freeloda.blog.51cto.com/2033581/1301382
参考资料:关于tomcat8的权威指南:http://wiki.jikexueyuan.com/project/tomcat/