这个问题我也是无意间碰到的,之前一直是使用单机的ActiveMQ,所以也没这个问题,但是做集群时碰到这个问题,问题是这样子出现的:
首先,我准备了三台虚拟机,然后使用 Replicated LevelDB 的方式配置集群,配置如下:
<persistenceAdapter>
<!--<kahaDB directory="${activemq.data}/kahadb"/>-->
<replicatedLevelDB
directory="${activemq.data}/leveldb"
replicas="3"
bind="tcp://0.0.0.0:61619"
zkAddress="192.168.209.133:2181,192.168.209.134:2181,192.168.209.135:2181"
zkPath="/activemq"
hostname="test3"
/>
</persistenceAdapter>
之后使用 ./activemq console 命令将三个虚拟机中的ActiveMQ成功启动。
接着我断开master节点,发现剩下两台slave节点会自动选举一个成为master节点,再把原来的master节点启动,就相当于一个新的slave节点加入了集群。
这样子,我以为集群就是好的了,然后写代码,发布消费,都是很正常的。然后问题就出现了。
我在把master节点关闭,本来期望着剩下的节点能自动选一个成为master节点,可结果发现本该成为master节点的服务抛出了异常,重启都无效,而且此时代码也访问不了,ActiveMQ的管理后台也访问不了,就相当于整个集群挂了!!!
报错大概是这样子的:
java.io.IOException: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)
at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:667)
at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)
at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
at org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:230)
at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472)
at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:90)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:78)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:73)
at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:1011)
at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:952)
at org.iq80.leveldb.impl.DbImpl.recoverLogFile(DbImpl.java:564)
at org.iq80.leveldb.impl.DbImpl.<init>(DbImpl.java:209)
at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:82)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply$mcV$sp(LevelDBClient.scala:687)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply(LevelDBClient.scala:667)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_init$2.apply(LevelDBClient.scala:667)
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)
... 11 more
Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 63 more
然后花了半个小时的时间百度(Google进不去,可惜了),得到的解决方法主要有四种:
1、移除lib目录中的pax-url-aether-1.5.2.jar包(对我情况无效,lib目录下没有这个包,园友们可以试试)
2、注释或者删除conf/activemq.xml中id="logQuery"的bean,它的class="io.fabric8.insight.log.log4j.Log4jLogQuery"(对我情况无效,园友们可以试试)
3、清除所有的数据,也就是删除每个节点的上面配置的 replicatedLevelDB 节点中 directory 属性指向的目录 ,比如我这里就是删除每个节点下的 data/leveldb 目录(证实有效)
4、换低版本的activemq试试(这个没试)
唯一能解决这个问题的办法是清除所有数据,这让我难以接受,如果哪天线上环境不小心把master节点停了,然道要清除所有数据才能重启?这种情况谁都无法接受。
只能自己想办法了,看异常信息,开头是:
java.io.IOException: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)
at org.apache.activemq.leveldb.LevelDBClient.replay_init(LevelDBClient.scala:667)
at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:558)
at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
at org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:110)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:230)
at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
....
这个大致信息可以理解为,本节点在作为master节点启动时出错了,在 IOExceptionSupport.create()方法想创建一个对象时抛出异常,异常信息是下面的:
Caused by: java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
...
Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 63 more
这个明显是说找不到类:com.google.common.util.concurrent.internal.InternalFutureFailureAccess,那好吧,我给你去找这个类。
于是接着百度,果然度娘最喜欢辜负了苦心人,没有!
这时脑子突然机灵了一下,为什么不去Maven上面找找呢?maven直通车:https://mvnrepository.com/
然后我搜索 com.google.common.util.concurrent.internal.InternalFutureFailureAccess 这个类,果然在上面找到一个包存在这个类:
进去看详情,好家伙,这个包就两个版本,还两三年没更新了,然后我选择1.0.1版本进去,下载jar包:
为避免不熟悉maven的朋友找不到,我将两个版本的jar包都下载再来了,因为担心每个版本依赖不一样,大家可以从百度网盘获取:https://pan.baidu.com/s/1T7sxBYuqqrPnyGWU4VJz9g (提取码: mtaj )
jar下载下来后,我将包放到每个节点ActiveMQ的lib目录下,然后重启每个节点,问题完美解决!!!!
这个问题写这么多,是因为确实度娘上的资料太少了,相信以后还有很多园友会碰到这个问题了,特此记一下!
- - - - - - - - - - - - - - - - - - - 分割线- - - - - - - - - - - - - - - - - - -
另外,还有一个小问题,在使用过程中,发现ActiveMQ启动后,抛出下面的异常:An IOException was thrown (should never happen in this method).
java.lang.RuntimeException: An IOException was thrown (should never happen in this method).
at org.apache.activemq.leveldb.record.CollectionKey$Buffer.bean(CollectionKey.java:264)
at org.apache.activemq.leveldb.record.CollectionKey$Buffer.getKey(CollectionKey.java:284)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1$$anonfun$apply$mcV$sp$4.apply(LevelDBClient.scala:757)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1$$anonfun$apply$mcV$sp$4.apply(LevelDBClient.scala:740)
at scala.Option.map(Option.scala:146)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:740)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:707)
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:707)
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)
at org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:706)
at org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:185)
at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
at org.fusesource.hawtbuf.proto.CodedInputStream.readRawByte(CodedInputStream.java:346)
at org.fusesource.hawtbuf.proto.CodedInputStream.readRawVarint32(CodedInputStream.java:240)
at org.fusesource.hawtbuf.proto.CodedInputStream.skipField(CodedInputStream.java:117)
at org.apache.activemq.leveldb.record.CollectionKey$Bean.mergeUnframed(CollectionKey.java:172)
at org.apache.activemq.leveldb.record.CollectionKey$Buffer.bean(CollectionKey.java:259)
... 14 more
虽然有这个异常,但是ActiveMQ还是能正常启动正常使用,不过我还是有些担心,所以就查了一下,结果发现后台跑了两个ActiveMQ进程,杀掉再启动还是会再启动两个进程,没办法,只好试试重启一下机器,结果问题解决