小编的最终目的是搭建solr集群,只要是用到solr的童鞋对其概念肯定不会陌生哈,直接上安装步骤~
Solr cloud 安装过程
部署规划
软件下载地址【相关软件都安装到/usr/local/soft目录】
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
2.zookeeper 3.4.7 下载地址
http://www.apache.org/dyn/closer.cgi/zookeeper/
3.solr 5.2.1 下载地址:
http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
4. ik中文分词器下载地址
https://codeload.github.com/lostsquirrel/ik-analyzer/zip/master
5. maven 下载地址
http://mirrors.cnnic.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
6. mysql-connector-java下载地址
http://dev.mysql.com/downloads/file/?id=459312
软件安装
禁用防火墙和selinux
三台服务器都需要操作-创建虚拟机的时候已经禁用了防火墙,这里再不用执行;# iptables -F;service iptables stop;chkconfig iptables off
#sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config;setenforce 0
JDK1.7 安装
三台服务器都需要操作
(1)rpm包安装
[root@localtest soft]# pwd
/usr/local/soft
[root@localtest soft]# rpm -ivh jdk-7u79-linux-x64.rpm
(1-1)解压jdk-7u79-linux-x64.gz
[root@localtest soft]# tar -zxvf jdk-7u79-linux-x64.gz
(2)更改环境变量
[root@localtest soft]# vi /etc/profile
JAVA_HOME=/usr/local/soft/jdk1.7.0_79
CLASSPATH=$JAVA_HOME/jre/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
[root@localtest soft]# source /etc/profile
(3)验证 jdk 版本
[root@localtest soft]# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
ssh client 安装
互信配置
ssh-keygen -t rsa
ll
cat ./id_rsa.pub > ./authorized_keys
scp -r ~/.ssh 192.168.247.129:~/
scp -r ~/.ssh 192.168.247.130:~/
scp -r ~/.ssh 192.168.247.131:/
vi /etc/selinux/config
scp /etc/selinux/config 192.168.247.131:/etc/selinux/
scp /etc/selinux/config 192.168.247.130:/etc/selinux/
ssh -p 22 192.168.247.129 date
ssh -p 22 192.168.247.131 date
第一次可能需要输入密码
zookeeper安装
安装【三台服务器都需要操作】
(1)将zookeeper 解压并安装到/usr/local/soft下cd /usr/local/soft
tar -zxvf zookeeper-3.4.7
(2)创建zookeeper 数据目录及日志目录
mkdir -p /data/zookeeper/data /logs
(3)配置zookeeper的配置文件zoo.cfg
cp /usr/local/soft/zookeeper-3.4.7/conf/zoo_sample.cfg /usr/local/soft/zookeeper-3.4.7/conf/zoo.cfg
修改配置文件
vi /usr/local/soft/zookeeper-3.4.7/conf/zoo.cfg
<span style="font-size:14px;">initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 dataLogDir=/data/zookeeper/logs server.1=192.168.247.129:2888:3888 server.2=192.168.247.130:2888:3888 server.3=192.168.247.131:2888:3888 </span>(4)更改每台服务器的myid
192.168.247.129机器:
echo 1 > /data/zookeeper/data/myid
192.168.247.130机器:
echo 2 > /data/zookeeper/data/myid
192.168.247.131机器:
echo 3 > /data/zookeeper/data/myid
zookeeper启动及关闭
启动/usr/local/soft/zookeeper-3.4.7/bin/zkServer.sh start
关闭
/usr/local/soft/zookeeper-3.4.7/bin/zkServer.sh stop
zookeeper 日志查看
如果zookeeper在启动或关闭过程中遇到一些异常情况,我们可以通过下面的日志查看日志:cat /usr/local/soft/zookeeper-3.4.7/bin/zookeeper.out 第一次配置完没发现有该文件,不知道啥时候生成的
zkCli.sh基本使用说明
我们可以通过zkCli.sh 命令登录到zookeeper集群中,查看,删除等操作。cd /usr/local/soft/zookeeper-3.4.7/bin
如果直接输入./zkCli.sh 默认以本机器登录zookeeper集群中。也可以加上 -server指定登录的服务器,这里以本台机器登录
./zkCli.sh
….(省略部分内容)
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
看到这里,说明登录成功
可以使用help等命令~
Solr cloud 安装
solr 安装【三台服务器都需要操作】
(1)将Solr 解压并安装到/usr/local/soft下cd /usr/local/soft
tar -zxvf /soft/solr-5.2.1.tgz
(2)创建solr 数据目录
mkdir /data/solrhome
(3)配置环境变量
[root@localtest bin]# cd /usr/local/soft/solr-5.2.1/bin
[root@localtest bin]# pwd
/usr/local/soft/solr-5.2.1/bin
[root@localtest bin]# vi solr.in.sh
配置文件内容如下:
<span style="font-size:14px;">SOLR_HEAP="512m" SOLR_JAVA_MEM="-Xms512m -Xmx512m" GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \ -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime" GC_TUNE="-XX:NewRatio=3 \ -XX:SurvivorRatio=4 \ -XX:TargetSurvivorRatio=90 \ -XX:MaxTenuringThreshold=8 \ -XX:+UseConcMarkSweepGC \ -XX:+UseParNewGC \ -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \ -XX:+CMSScavengeBeforeRemark \ -XX:PretenureSizeThreshold=64m \ -XX:+UseCMSInitiatingOccupancyOnly \ -XX:CMSInitiatingOccupancyFraction=50 \ -XX:CMSMaxAbortablePrecleanTime=6000 \ -XX:+CMSParallelRemarkEnabled \ -XX:+ParallelRefProcEnabled" ZK_HOST="192.168.247.129:2181,192.168.247.130:2181,192.168.247.131:2181" ZK_CLIENT_TIMEOUT="15000" #所在主机名 SOLR_HOST="192.168.247.129" 三台各自的ip #时区 SOLR_TIMEZONE="UTC+8" ENABLE_REMOTE_JMX_OPTS="false" #RMI_PORT=18983 #SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=3000" #SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=60000" #SOLR_OPTS="$SOLR_OPTS -Dsolr.clustering.enabled=true" SOLR_PID_DIR=/data/solrhome/ SOLR_HOME=/data/solrhome/ #LOG4J_PROPS=/var/solr/log4j.properties #SOLR_LOGS_DIR= SOLR_PORT=8983 </span>
(4)更改启动参数
vi /etc/init.d/solr
只要更改三项:
SOLR_INSTALL_DIR=/usr/local/soft/solr-5.2.1/
SOLR_ENV=/usr/local/soft/solr-5.2.1/bin/solr.in.sh
RUNAS=root
cp /usr/local/soft/solr-5.2.1/server/solr/solr.xml /data/solrhome/
(6)启动测试
[root@localtest bin]# pwd
/usr/local/soft/solr-5.2.1/bin
[root@localtest bin]# ./solr start -s /data/solrhome
Started Solr server on port 8983 (pid=16359). Happy searching!
当我们看到 “Happy searching!” 说明就已经启动成功了!
以192.168.247.130为例。
scp -r -P 22 /usr/local/soft/solr-5.2.1 192.168.247.130:/usr/local/soft/
scp -r -P 22 /etc/init.d/solr 192.168.247.130:/etc/init.d/
scp -r -P 22 /data/solrhome/ 192.168.247.130:/data/
注意:
更改各自的solr.in.sh(环境变量SOLR_HOST="192.168.247.XX"项)
启动solr cloud
/etc/init.d/solr start
或者;./solr -e techproducts
Waiting to see Solr listening on port 8983 [\]
Started Solr server on port 8983 (pid=16714). Happy searching!
这里将三台机器的solr都启动。
(2)日志查看
cd /usr/local/soft/solr-5.2.1/server/logs
tail -f solr.log
(3)登录界面验证
在浏览器输入:http://192.168.247.129:8983/
Solr cloud配置
创建collection
建立目录
mkdir -p /data/solrhome/jyalltest/conf拷贝配置文件
cp -r /usr/local/soft/solr-5.2.1/example/files/conf/ /data/solrhome/jyalltest/3.1.3 确认zookeeper启动成功
###192.168.247.129
cd /usr/local/soft/zookeeper-3.4.7/bin/
./zkServer.sh status
JMX enabled by default
Using config: /usr/local/soft/zookeeper-3.4.7/bin/../conf/zoo.cfg
Mode: leader
###192.168.247.130
cd /usr/local/soft/zookeeper-3.4.7/bin/
./zkServer.sh status
JMX enabled by default
Using config: /usr/local/soft/zookeeper-3.4.7/bin/../conf/zoo.cfg
Mode: follower
###192.168.247.131
cd /usr/local/soft/zookeeper-3.4.7/bin/
./zkServer.sh status
JMX enabled by default
Using config: /usr/local/soft/zookeeper-3.4.7/bin/../conf/zoo.cfg
Mode: follower
检查所有机器是否都在live_node下
cd /usr/local/soft/zookeeper-3.4.7/bin./zkCli.sh
[zk: localhost:2181(CONNECTED) 1] ls /live_nodes
[192.168.247.129:8983_solr, 192.168.247.130:8983_solr, 192.168.247.131:8983_solr]
上传配置文件到zookeeper集群中
(1)上传文件sh /usr/local/soft/solr-5.2.1/server/scripts/cloud-scripts/zkcli.sh -zkhost 192.168.247.129:2181,192.168.247.130:2181,192.168.247.131:2181 -cmd upconfig -confdir /data/solrhome/jyalltest/conf -confname myconf
(2)验证
cd /usr/local/soft/zookeeper-3.4.7/bin
./zkCli.sh
[zk: localhost:2181(CONNECTED) 2] ls /configs/myconf
[currency.xml, protwords.txt, data-config.xml, solrconfig.xml, lang, stopwords.txt, synonyms.txt, update-script.js, velocity, params.json, elevate.xml, email_url_types.txt, managed-schema]
创建collection
(1)创建我们需要创建一个名为jyalltest 的collection,两个shard分片,每个节点最多两个shard,每个shard的复制因子为2(一个replica,一个leader)
http://192.168.247.129:8983/solr/admin/collections?action=CREATE&name=jyalltest&maxShardsPerNode=2&numShards=3&replicationFactor=2
(2)验证
http:// 192.168.247.129:8983/solr/#/~cloud
重新加载配置文件
http://192.168.247.129:8983/solr/admin/collections?action=RELOAD&name=jyalltest删除collection(可以删除后重新建)
http://192.168.247.129:8983/solr/admin/collections?action=DELETE&name=jyalltest3 安装ant maven
因为是源代码,此处需要使用maven打包。(1)yum安装ant
yum install ant -y
这里yum只能安装ant ,不能安装maven
注意:安装完成后,jdk版本变成1.6了,我们需要重新/etc/profile 更改为1.7 的jdk版本
(2)解压缩maven软件
tar -zxvf apache-maven-3.3.9-bin.tar.gz
mv apache-maven-3.3.9 /usr/local/
(3)更改环境变量
vi /etc/profile
JAVA_HOME=/usr/local/soft/jdk1.7.0_79
CLASSPATH=$JAVA_HOME/jre/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
MAVEN_HOME= /usr/local/soft/apache-maven-3.3.9
export MAVEN_HOME
export PATH=${PATH}:${MAVEN_HOME}/bin
重新加载配置文件:
source /etc/profile
(4)检查版本
检测mvn版本
mvn -v
[root@localtest bin]# mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /usr/local/soft/apache-maven-3.3.9
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/local/soft/jdk1.7.0_79/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-504.el6.x86_64", arch: "amd64", family: "unix"
[root@localtest bin]#
配置中文分词
IKAnalyzer是一个开源的,基于java语言开发的轻量级的中文分词工具包。从2006年12月推出1.0版开始,IKAnalyzer已经推出了3个大版本。最初,它是以开源项目Luence为应用主体的,结合词典分词和文法分析算法的中文分词组件。新版本的 IKAnalyzer3.0则发展为面向Java的公用分词组件,独立于Lucene项目,同时提供了对Lucene的默认优化实现。解压缩软件
unzip ik-analyzer-master.zip编译词库
[root@localtest ik-analyzer-master]# pwd
/usr/local/soft/ik-analyzer-master
[root@localtest ik-analyzer-master]# mvn install -Dmaven.test.skip=true
[root@localtest solr-5.2.1]# pwd
/usr/local/soft/solr-5.2.1
[root@localtest solr-5.2.1]# cp -r /usr/local/soft/ik-analyzer-master/target/classes/ ./server/solr-webapp/webapp/WEB-INF/
拷贝软件包到其他机器
[root@localtest solr-5.2.1]# cd /usr/local/soft/solr-5.2.1/拷贝WEB-INFO下所有文件到其他机器
[root@localtest solr-5.2.1]# scp -P 22 -r ./server/solr-webapp/webapp/WEB-INF 192.168.247.130:/usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/
[root@localtest solr-5.2.1]# scp -P 22 -r ./server/solr-webapp/webapp/WEB-INF 192.168.247.131:/usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/
编译text_zh类(此操作只需在一个节点进行)
cd /usr/local/soft/solr-5.2.1/(1)获取managed-schema 文件
[root@localtest solr-5.2.1]# ./server/scripts/cloud-scripts/zkcli.sh -zkhost 192.168.247.129:2181 -cmd get /configs/myconf/managed-schema > ./managed-schema
(2)修改managed-schema文件(添加该选项)
vi managed-schema
<fieldType name="text_zh" class="solr.TextField">
<analyzer type="index" isMaxWordLength="false" class="org.wltea.analyzer.lucene.IKAnalyzer"/>
<analyzer type="query" isMaxWordLength="true" class="org.wltea.analyzer.lucene.IKAnalyzer"/>
</fieldType>
(3)替换managed-schema文件
[root@localtest solr-5.2.1]# mv ./managed-schema /data/solrhome/jyalltest/conf/
(4)上传更改后的文件
[root@localtest solr-5.2.1]# /usr/local/soft/solr-5.2.1/server/scripts/cloud-scripts/zkcli.sh -c jyalltest -z 192.168.247.129:2181 -cmd upconfig -confdir /data/solrhome/jyalltest/conf -confname myconf
重新加载配置文件(在重载前需要重启三个solr)
重启solr以后,129,131已经有中文分词功能;130没有,检查日志,再导入文件时报错,重新拷贝文件,然后重启130,分词功能正常[root@localtest solr-5.2.1]# scp -P 22 -r ./server/solr-webapp/webapp/WEB-INF 192.168.247.130:/usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/
http://192.168.247.129:8983/solr/admin/collections?action=RELOAD&name=jyalltest
验证分词
配置数据导入
下载配置文件
[root@localtest solr-5.2.1]# pwd
/usr/local/soft/solr-5.2.1
[root@localtest solr-5.2.1]# ./server/scripts/cloud-scripts/zkcli.sh -c jyalltest -z 192.168.247.129:2181 -cmd downconfig -confdir /data/solrhome/jyalltest/conf/ -confname myconf
更改solrconfig.xml(以图片为主)
[root@localtest conf]# vi ./solrconfig.xml
<span style="font-size:14px;"> <!-- A request handler that returns indented JSON by default -->后面添加以下部分: <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str name="config">data-config.xml</str> <!-- <str name="update.chain">dedupe</str> --> </lst> </requestHandler></span>
<bool name="overwriteDupes">true</bool> ==========》true 更改成flase
(3)注释内容
<span style="font-size:14px;"><!-- <processor class="solr.StatelessScriptUpdateProcessorFactory"> <str name="script">update-script.js</str> <lst name="params"> <str name="config_param">example config parameter</str> </lst> </processor> --> </span>
创建数据泵文件
cd /data/solrhome/jyalltest/conf/(1)测试mysql信息:
ip地址:192.168.247.128 库名:test ,表名:m_building,用户名:w11, 密码:w11
(2)创建data-config.xml 文件
vi ./data-config.xml
<span style="font-size:14px;"><dataConfig> <dataSource name="jdbc" driver="com.mysql.jdbc.Driver" url="jdbc:mysql:// 192.168.247.128:3306/test?zeroDateTimeBehavior=convertToNull&characterEncoding=utf8&useUnicode=true" user="w11" password="w11"/> <document name="solr_build"> <entity name="build" pk="id" query="SELECT id,name,introductionweb,introduction FROM m_building" deltaImportQuery="SELECT id,name,introductionweb,introduction FROM m_building where id='${dih.delta.id}'" deltaQuery="select id from m_building where update_time > '${dih.last_index_time}'" > <field column="id" name="id" /> <field column="name" name="buildname" /> <field column="introductionweb" name="introductionweb" /> <field column="introduction" name="introduction" /> </entity> </document> </dataConfig> </span>
schema中加入新列
[root@localtest conf]# pwd/data/solrhome/jyalltest/conf
[root@localtest conf]# vi ./managed-schema
增加以下内容(id列,我们使用solr自带的就可以)
<span style="font-size:14px;"><field name="buildname" type="text_zh" indexed="true" stored="true"/> <field name="introductionweb" type="text_zh" indexed="true" stored="true"/> <field name="introduction" type="text_zh" indexed="true" stored="true"/> <field name="fullcol" type="text_zh" multiValued="true" indexed="true" stored="false"/> <copyField source="*" dest="fullcol"/></span>
上传和reload配置文件
(1)上传配置文件/usr/local/soft/solr-5.2.1/server/scripts/cloud-scripts/zkcli.sh -c jyalltest -z 192.168.247.129:2181 -cmd upconfig -confdir /data/solrhome/jyalltest/conf/ -confname myconf
(2)重新加载配置文件(加载前要重启三个solr)
http://192.168.247.129:8983/solr/admin/collections?action=RELOAD&name=jyalltes
• jyalltest_shard3_replica2: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler'
• jyalltest_shard2_replica1: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler'
拷贝软件包并同步到其他机器
(1)拷贝solr-dataimporthandler-*cp /usr/local/soft/solr-5.2.1/dist/solr-dataimporthandler-* /usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib/
重启后上述异常消失
(2)拷贝mysql-connector-java
[root@localtest soft]# pwd
/usr/local/soft
unzip mysql-connector-java.zip
cp mysql-connector-java-5.1.26/mysql-connector-java-5.1.26-bin.jar /usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib/
其余两台文件拷贝失败,重新上传,然后重启solr
(3)同步其他节点(130,131)
scp -P 22 -r /usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib 192.168.247.130:/usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib/
scp -P 22 -r /usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib 192.168.247.131:/usr/local/soft/solr-5.2.1/server/solr-webapp/webapp/WEB-INF/lib/
重启solr服务
在各个机器上重启solr (192.128.247.129/130/131)service solr restart
全量导入数据
http://192.168.247.129:8983/solr/jyalltest/dataimport?command=full-import增量导入数据
http://192.168.247.129:8983/solr/jyalltest/dataimportsolr cloud 使用
select查询
SELECT * FROM m_building WHERE NAME='空中楼阁'http://192.168.247.129:8983/solr/jyalltest_shard2_replica1/select?q=*:*&fq=city:北京
返回指定列(fieldlist)
http://192.168.247.129:8983/solr/jyalltest_shard2_replica1/select?q=*%3A*&fl=buildname<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">286</int>
</lst>
<lst name="failure">
<str name="192.168.247.131:8983_solr">
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Errorfrom server at http://192.168.247.131:8983/solr: Error handling 'reload' action
</str>
<str name="192.168.247.131:8983_solr">
...· jyalltest_shard3_replica2: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler'
· jyalltest_shard2_replica1: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler'
3.创建conllection时:
<response>
<lst name="responseHeader">
<int name="status">400</int>
<int name="QTime">63</int>
</lst>
<str name="Operation createcaused exception:">
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:No config set found to associate with the collection.
</str>
<lst name="exception">
<str name="msg">
No config set found to associate with thecollection.
</str>
<int name="rspCode">400</int>
</lst>
<lst name="error">
<str name="msg">
No config set found to associate with thecollection.
</str>
<int name="code">400</int>
</lst>
</response>
4.添加mysql jar包后
HTTP ERRPR : 503 problem accessing /solr/. reason: Service Unavailable