ElasticSearch安装ik分词插件

时间：2022-09-12 23:56:52

一、IK简介

       IK Analyzer是一个开源的，基于java语言开发的轻量级的中文分词工具包。从2006年12月推出1.0版开始， IKAnalyzer已经推出了4个大版本。最初，它是以开源项目Luence为应用主体的，结合词典分词和文法分析算法的中文分词组件。从3.0版本开 始，IK发展为面向Java的公用分词组件，独立于Lucene项目，同时提供了对Lucene的默认优化实现。在2012版本中，IK实现了简单的分词 歧义排除算法，标志着IK分词器从单纯的词典分词向模拟语义分词衍化。

       IK Analyzer 2012特性:

        1. 采用了特有的“正向迭代最细粒度切分算法“，支持细粒度和智能分词两种切分模式；

        2. 在系统环境：Core2 i7 3.4G双核，4G内存，window 7 64位， Sun JDK 1.6_29 64位 普通pc环境测试，IK2012具有160万字/秒（3000KB/S）的高速处理能力。

        3. 2012版本的智能分词模式支持简单的分词排歧义处理和数量词合并输出。

        4. 采用了多子处理器分析模式，支持：英文字母、数字、中文词汇等分词处理，兼容韩文、日文字符

        5. 优化的词典存储，更小的内存占用。支持用户词典扩展定义。特别的，在2012版本，词典支持中文，英文，数字混合词语。

二、安装IK分词插件

假设读者已经安装好ES，如果没有的话，请参考ElasticSearch入门 —— 集群搭建。安装IK分词需要的资源可以从这里下载，整个安装过程需要三个步骤：

1、获取分词的依赖包

通过git clone https://github.com/medcl/elasticsearch-analysis-ik，下载分词器源码，然后进入下载目录，执行命令：mvn clean package，打包生成elasticsearch-analysis-ik-1.2.5.jar。将这个jar拷贝到ES_HOME/plugins/analysis-ik目录下面，如果没有该目录，则先创建该目录。

2、ik目录拷贝

将下载目录中的ik目录拷贝到ES_HOME/config目录下面。

3、分词器配置

打开ES_HOME/config/elasticsearch.yml文件，在文件最后加入如下内容：

index:
  analysis:                   
    analyzer:      
      ik:
          alias: [ik_analyzer]
type: org.elasticsearch.index.analysis.IkAnalyzerProvider
      ik_max_word:
type: ik
          use_smart: false
      ik_smart:
type: ik
          use_smart: true
index.analysis.analyzer.default.type: ik

转载者注：这里ik, ik_max_word, ik_smart 是三个不同配置的分词器，由于ik_max_word中配置use_smart: false, 所以ik_max_word完全继承于ik， 是完全一模一样的。

ok！插件安装已经完成，请重新启动ES，接下来测试ik分词效果啦！

三、ik分词测试

1、创建一个索引，名为index。

curl -XPUT http://localhost:9200/index

2、为索引index创建mapping。

curl -XPOST http://localhost:9200/index/fulltext/_mapping -d'
{
"fulltext": {
"_all": {
"analyzer": "ik"
        },
"properties": {
"content": {
"type" : "string",
"boost" : 8.0,
"term_vector" : "with_positions_offsets",
"analyzer" : "ik",
"include_in_all" : true
            }
        }
    }
}'

3、测试

curl 'http://localhost:9200/index/_analyze?analyzer=ik&pretty=true' -d '
{
"text":"世界如此之大"
}'

显示结果如下：

{
  "tokens" : [ {
 "token" : "text",
 "start_offset" : 4,
 "end_offset" : 8,
 "type" : "ENGLISH",
 "position" : 1
}, {
 "token" : "世界",
 "start_offset" : 11,
 "end_offset" : 13,
 "type" : "CN_WORD",
 "position" : 2
}, {
 "token" : "如此",
 "start_offset" : 13,
 "end_offset" : 15,
 "type" : "CN_WORD",
 "position" : 3
}, {
 "token" : "之大",
 "start_offset" : 15,
 "end_offset" : 17,
 "type" : "CN_WORD",
 "position" : 4
} ]
}

我使用的是 elasticsearch-1.6.0 + analysis-ik-1.4.0

安装ik遇到错误请看此处：

错误内容：

[2015-09-25 09:57:54,475][WARN ][cluster.action.shard     ] [local] [order][3] received shard failed for [order][3], node[oYzaqPm8S1GqCUgZM0nFSg], [P], s[INITIALIZING], indexUUID [EIrG6ASTRKOuwU0C6x80AA], reason [shard failure [failed to create index][IndexCreationException[[order] failed to create index]; nested: NoClassDefFoundError[org/apache/http/client/ClientProtocolException]; nested: ClassNotFoundException[org.apache.http.client.ClientProtocolException]; ]]
[2015-09-25 09:57:54,533][WARN ][indices.cluster          ] [local] [[order][1]] marking and sending shard failed due to [failed to create index]
org.elasticsearch.indices.IndexCreationException: [order] failed to create index
        at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:338)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:313)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:182)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:480)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/http/client/ClientProtocolException
        at org.elasticsearch.index.analysis.IkTokenizerFactory.<init>(IkTokenizerFactory.java:25)
        at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.InjectorImpl$5$1.call(InjectorImpl.java:781)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.InjectorImpl$5.get(InjectorImpl.java:777)
        at org.elasticsearch.common.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:221)
        at com.sun.proxy.$Proxy15.create(Unknown Source)
        at org.elasticsearch.index.analysis.AnalysisService.<init>(AnalysisService.java:82)
        at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
        at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
        at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:336)
        ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.ClientProtocolException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 61 more

解决方法：
复制 http-client.jar 和http-core.jar 两个包到 elasticsearch/lib 目录下即可。

在java中使用ik分词器

请点击 java api

参考原文：
https://github.com/medcl/elasticsearch-analysis-ik
http://my.oschina.net/xiaohui249/blog/232784
标签：

相关文章