docker搭建elasticsearch、kibana，并集成至spring boot

步骤如下：

一、基于docker搭建elasticsearch环境

1、拉取镜像

docker pull elasticsearch5.6.8

2、制作elasticsearch的配置文件

master配置

http.host: 0.0.0.0
#集群名称 所有节点要相同
cluster.name: "estest"
#本节点名称
node.name: master
#作为master节点
node.master: true
#是否存储数据
node.data: true
bootstrap.system_call_filter: false
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

slave配置

http.host: 0.0.0.0
#集群名称 所有节点要相同
cluster.name: "estest"
#子节点名称
node.name: slave1
#不作为master节点
node.master: false
node.data: true
bootstrap.system_call_filter: false
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["esmaster:9300"]

注意salve节点的discovery.zen.ping.unicast.hosts设置为esmaster:9300,这里的esmaster是master节点的docker容器名字

3、启动容器

master容器

docker run -d --name esmaster --ulimit nofile=65536:131072 -p 9200:9200 -p 9300:9300 -v /home/es/config/master.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /home/es/master:/usr/share/elasticsearch/data -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" elasticsearch:5.6.8

参数说明：

--name esmaster
  指定容器名称为esmater,salve节点配置文件中的discovery.zen.ping.unicast.hosts要和这里对应，因为salve容器和master容器之间的是通过 --linkM指令来通信的。

--ulimit nofile=65536:131072
  避免容器启动时会报bootstrap checks failed异常

-- -p 9200:9200 -p 9300:9300

暴露出容器的9200,9300端口到宿主机的9200,9300端口

-- -v C:/work/docker_volume/elasticsearch/elasticsearchmaster.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v C:/work/docker_volume/elasticsearch/data/master:/usr/share/elasticsearch/data
挂载卷到容器中，其实就是设置容器配置文件关联上面写好的配置文件，容器存储数据关联到宿主机的外部文件中

-- -e "ES_JAVA_OPTS=-Xms512m -Xmx512m"

设置容器内Java虚拟机的内存大小

salve容器

docker run -d --name esslave1 --ulimit nofile=65536:131072 --link esmaster:esmaster -v /home/es/config/slave1.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /home/es/slave1:/usr/share/elasticsearch/data -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" elasticsearch:5.6.8

主要是使用--link esmaster:esmaster指令，来连接master容器，别的和master容器启动参数基本一样。

浏览器输入curl localhost:9200验证一下

{
  "name" : "master",
  "cluster_name" : "estest",
  "cluster_uuid" : "x1-qThTBRA6GQr_sdjYMGw",
  "version" : {
    "number" : "5.6.8",
    "build_hash" : "688ecce",
    "build_date" : "2018-02-16T16:46:30.010Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

说明elasticsearch已经启动成功

二、kibana

1、拉取镜像

docker pull kibana:5.6.14

2、启动镜像

docker run -it -d -e ELASTICSEARCH_URL=http://172.17.0.2:9200 -p 5601:5601 --name kibana  kibana:5.6.14

注意ELASTICSEARCH_URL中配的是es容器的docker内部ip，可以通过如下方式查看

docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container-ID>

3、http://localhost:5601访问即可

三、es与springboot集成

1、pom

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
    <groupId>io.searchbox</groupId>
    <artifactId>jest</artifactId>
</dependency>
<dependency>
    <groupId>net.java.dev.jna</groupId>
    <artifactId>jna</artifactId>
</dependency>

2、工具类

@Slf4j
@Component
@ConditionalOnProperty(name = "elasticsearch.enabled")
public class ElasticUtil {

    private static JestClient jestClient;

    /**
     * 向elasticsearch中插入一条记录
     *
     * @param t     数据
     * @param index 索引名
     * @param type  类型名
     * @param <T>
     * @return 插入结果
     */
    public static <T> Result save(T t, String index, String type) {
        if (t == null || StringUtils.isBlank(index) || StringUtils.isBlank(type)) {
            return Result.getResult(Constant.FAIL, "缺少参数");
        }
        Index indexBuilder = new Index.Builder(t).index(index).type(type).build();
        try {
            DocumentResult execute = jestClient.execute(indexBuilder);
            return Result.getResult(Constant.SUCCESS, "插入elasticsearch成功", execute);
        } catch (IOException e) {
            log.error("插入elasticsearch异常", e);
            return Result.getResult(Constant.ERROR, "插入elasticsearch异常");
        }
    }

    /**
     * 向elasticsearch中批量插入多条记录
     *
     * @param list  数据
     * @param index 索引名
     * @param type  类型名
     * @param <T>
     * @return 插入结果
     */
    public static <T> Result saveAll(List<T> list, String index, String type) {
        if (list == null || list.size() == 0 || StringUtils.isBlank(index) || StringUtils.isBlank(type)) {
            return Result.getResult(Constant.FAIL, "缺少参数");
        }
        Bulk.Builder bulkBuilder = new Bulk.Builder();
        /*list.stream().forEach(data -> {
            Index indexBuilder = new Index.Builder(data).index(index).type(type).build();
            bulkBuilder.addAction(indexBuilder);
        });*/
        List<Index> indexList = list.stream().map(data -> new Index.Builder(data).index(index).type(type).build()).collect(toList());
        bulkBuilder.addAction(indexList);
        try {
            jestClient.execute(bulkBuilder.build());
            return Result.getResult(Constant.SUCCESS, "批量插入elasticsearch成功");
        } catch (IOException e) {
            log.error("批量插入elasticsearch异常", e);
            return Result.getResult(Constant.ERROR, "批量插入elasticsearch异常");
        }
    }

    /**
     * 根据id修改文档
     *
     * @param t     修改数据
     * @param index 索引
     * @param type  类型
     * @param id    主键
     * @param <T>
     * @return 修改结果
     */
    public static <T> Result updateById(T t, String index, String type, String id) {
        if (t == null || StringUtils.isBlank(index) || StringUtils.isBlank(type) || StringUtils.isBlank(id)) {
            return Result.getResult(Constant.FAIL, "缺少参数");
        }
        try {
            String doc = XContentFactory.jsonBuilder().startObject().field("doc", JSONObject.toJSON(t)).endObject().string();
            Update updateBuilder = new Update.Builder(doc).index(index).type(type).id(id).build();
            DocumentResult execute = jestClient.execute(updateBuilder);
            return Result.getResult(Constant.SUCCESS, "修改elasticsearch文档成功", execute);
        } catch (IOException e) {
            log.error("XContentFactory构造JSON异常", e);
            return Result.getResult(Constant.ERROR, "XContentFactory构造JSON异常");
        } catch (Exception e) {
            log.error("修改elasticsearch文档异常", e);
            return Result.getResult(Constant.ERROR, "修改elasticsearch文档异常");
        }
    }

    /**
     * 根据主键删除文档
     *
     * @param index 索引
     * @param type  类型
     * @param id    主键
     * @return
     */
    public static Result deleteById(String index, String type, String id) {
        if (StringUtils.isBlank(index) || StringUtils.isBlank(type) || StringUtils.isBlank(id)) {
            return Result.getResult(Constant.FAIL, "缺少参数");
        }
        try {
            Delete deleteBuilder = new Delete.Builder(id).index(index).type(type).build();
            DocumentResult execute = jestClient.execute(deleteBuilder);
            return Result.getResult(Constant.SUCCESS, "删除elasticsearch文档成功", execute);
        } catch (Exception e) {
            log.error("删除elasticsearch文档异常", e);
            return Result.getResult(Constant.ERROR, "删除elasticsearch文档异常");
        }
    }

    /**
     * 分页查询
     *
     * @param clazz 数据类型
     * @param elasticSearchBean 查询对象
     * @param <T>
     * @return 查询结果
     */
    public static <T> Result get(Class<T> clazz, ElasticSearchBean elasticSearchBean) {
        try {
            SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
            AbstractQueryBuilder queryStringQueryBuilder = null;
            if(elasticSearchBean.isFullTextMatching()){
                queryStringQueryBuilder = QueryBuilders.queryStringQuery(elasticSearchBean.getKeyword());
            }else {
                queryStringQueryBuilder = QueryBuilders.multiMatchQuery(elasticSearchBean.getKeyword(), (String[]) elasticSearchBean.getMatchingFields().toArray());
            }
            searchSourceBuilder.query(queryStringQueryBuilder);
            if(null != elasticSearchBean.getHighLightsFields()){
                HighlightBuilder highlightBuilder = new HighlightBuilder();
                elasticSearchBean.getHighLightsFields().stream().forEach(field -> highlightBuilder.field(field));
//                HighlightBuilder highlightBuilder = new HighlightBuilder().field("*").requireFieldMatch(false);
                highlightBuilder.requireFieldMatch(false);
                //高亮标签
                highlightBuilder.preTags("<em>").postTags("</em>");
                //高亮内容长度
                highlightBuilder.fragmentSize(200);
                searchSourceBuilder.highlighter(highlightBuilder);
            }
            Search.Builder searchBuilder = new Search.Builder(
                    searchSourceBuilder.toString())
                    .addIndex(elasticSearchBean.getIndex())
                    .addType(elasticSearchBean.getType());
            if(null != elasticSearchBean.getFrom()){
                searchBuilder.setParameter("from", elasticSearchBean.getFrom());
            }
            if(null != elasticSearchBean.getSize()){
                searchBuilder.setParameter("size", elasticSearchBean.getSize());
            }

            if(null != elasticSearchBean.getSortFields() && elasticSearchBean.getSortFields().size() > 0){
                searchBuilder.addSort(elasticSearchBean.getSortFields());
            }
//            .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)  //设置查询类型：1.SearchType.DFS_QUERY_THEN_FETCH 精确查询； 2.SearchType.SCAN 扫描查询,无序
            Search search = searchBuilder.build();
            SearchResult searchResult = jestClient.execute(search);
            if(!searchResult.isSucceeded()){
                return Result.getResult(Constant.FAIL,searchResult.getErrorMessage());
            }
            List<JSONObject> data = searchResult.getHits(clazz).stream().map(hit -> {
                JSONObject json = (JSONObject) JSONObject.toJSON(hit.source);
                json.put("esId", hit.id);
                Map<String, List<String>> highlight = hit.highlight;
                for (String key : highlight.keySet()) {
                    json.put(key, highlight.get(key).get(0));
                }
                return json;
            }).collect(toList());

            return Result.getResult(Constant.SUCCESS, "elasticsearch查询成功", data, searchResult.getTotal().intValue());
        } catch (IOException e) {
            log.error("elasticsearch查询异常", e);
            return Result.getResult(Constant.ERROR, "elasticsearch查询异常");
        }
    }

    @Resource
    public void setJestClient(JestClient jestClient) {
        ElasticUtil.jestClient = jestClient;
    }
}

3、抽象出一个类，封装查询时要传递的各种参数

@Data
public class ElasticSearchBean implements Serializable {
    private static final long serialVersionUID = -5188975204258390540L;
    /**
     * 索引
     */
    private String index;
    /**
     * 类型
     */
    private String type;
    /**
     * 要查询关键字
     */
    private String keyword;
    /**
     * 从第几行开始
     */
    private Integer from;
    /**
     * 查询几条数据
     */
    private Integer size;
    /**
     * 高亮字段
     */
    private List<String> highLightsFields;
    /**
     * 排序字段
     */
    private List<Sort> sortFields;
    /**
     * 是否是全文匹配,true:是;false;不是
     */
    private boolean isFullTextMatching;
    /**
     * 要匹配的字段
     */
    private List<String> MatchingFields;

}

4、测试

@RunWith(SpringRunner.class)
@WebAppConfiguration
@SpringBootTest(classes = SswApplication.class, webEnvironment = SpringBootTest.WebEnvironment.MOCK)
public class ElasticSearchTest {
    @Test
    public void saveTest() {
        User user = new User();
        user.setRealName("王五");
        user.setLoginName("wangwu");
        user.setAge(19);
        user.setGender(true);
        user.setStatus(0);
        Result result = ElasticUtil.save(user, "user", "user");
        System.out.println(result.getFlag());
        System.out.println(result.getMsg());
    }
    @Test
    public void updateTest() {
        User user = new User();
        user.setRealName("王五2号");
        user.setLoginName("wangwu");
        user.setAge(19);
        user.setGender(true);
        user.setStatus(0);
        Result result = ElasticUtil.updateById(user, "user", "user","AWLxcPgxbdesqcfPx6TL");
        System.out.println(result.getFlag());
        System.out.println(result.getMsg());
    }
    @Test
    public void deleteTest() {
        Result result = ElasticUtil.deleteById("user", "user","AWLxcPgxbdesqcfPx6TL");
        System.out.println(result.getFlag());
        System.out.println(result.getMsg());
    }
    @Test
    public void searchTest() {
        Class clazz = User.class;
        String index = "user";
        String type = "user";
        String keyword = "王";
        int from = 0;
        int size = 10;
        List<String> highLightsFields = Arrays.asList(new String[]{"realName","loginName"});
        List<Sort> sortFields = Arrays.asList(new Sort[]{new Sort("realName",Sort.Sorting.DESC),new Sort("loginName",Sort.Sorting.ASC)});
        List<String> matchingFields = Arrays.asList(new String[]{"password","email"});
        ElasticSearchBean elasticSearchBean = new ElasticSearchBean();
        elasticSearchBean.setIndex(index);
        elasticSearchBean.setType(type);
        elasticSearchBean.setKeyword(keyword);
        elasticSearchBean.setFrom(from);
        elasticSearchBean.setSize(size);
        elasticSearchBean.setHighLightsFields(highLightsFields);
        elasticSearchBean.setFullTextMatching(true);
        elasticSearchBean.setSortFields(sortFields);
        elasticSearchBean.setMatchingFields(matchingFields);

        Result result = ElasticUtil.get(clazz, elasticSearchBean);
        System.out.println(result.getFlag());
        System.out.println(result.getMsg());
        System.out.println(result.getTotal());
        System.out.println(result.getData());
    }
}

注意，在排序的时候，有时候会报异常

Fielddata is disabled on text fields by default. Set fielddata=true on [realName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.

这是因为elasticsearch对于text类型的字段，默认是不支持排序的，需要设置，方法如下：

发一个put请求：

http://localhost:9200/index/_mapping/type

请求体为

{
    "properties": {
        "realName": {
            "type": "text",
            "fielddata": true
        }
    }
}

如果返回

{"acknowledged":true}

说明设置成功，再排序就没有问题了