1. 概述

ELK

官网：https://www.elastic.co/cn

ELK是ElasticSearch、Logstash、Kibana三大开源框架字母大写简称。市面上也被称为Elastic Stack。ElasticSearch是一种分布式全文搜索引擎，Logstash是ELK的中央数据流引擎，用于从不同目标（文件/数据存储/MQ）收集的不同格式数据，经过过滤后支持输出到不同目的地（文件/MQ/Redis/ElasticSearch/Kafka）等。Kibana可以将ElasticSearch的数据通过友好的页面展示出来，提供实时分析的功能。

ElasticSearch

ElasticSearch，简称es，es是一个开源的高扩展的分布式全文检索引擎，它可以近乎实时的存储、检索数据；本身扩展性很好，可以扩展到上百台服务器，处理PB级别的数据。es也使用Java开发并使用Lucence作为其核心来实现所有的索引和搜索的功能，但是它的目的是通过简单的RESTful API来隐藏Lucence的复杂性，从而让全文搜索变得简单。

2. 安装

使用华为云下载

配置ElasticSearch

  • elasticsearch.yml：主要配置文件
  • jvm.options：JVM配置文件
  • log4j2.properties：日志配置文件

配置跨域，在elasticsearch.yml中添加：

  1. http.cors.enabled: true
  2. http.cors.allow-origin: '*'

机器内存比较小的，减小JVM堆大小：

  1. -Xms512m
  2. -Xmx512m

启动ElasticSearch

双击bin目录下的elasticsearch.bat文件即可启动，访问localhost:9200即可看到相关信息。

可视化

  1. $ get clone https://github.com/mobz/elasticsearch-head
  2. $ cnpm install
  3. $ npm run start

配置Kibana

在config/kibana.yml中添加：

  1. i18n.locale: "zh-CN"

启动Kibana

双击bin目录下的kibana.bat文件即可启动，访问localhost:5601即可看到界面。

image-20210319210407333.png

3. 核心

ElasticSearch与关系型数据库对比

Relational DB ElasticSearch
数据库(database) 索引(indices)
表(tables) 类型(type)
行(rows) 文档(documents)
字段(columns) 字段(fields)

ElasticSearch中可以包含多个索引，每个索引中中可以包含多个类型，每个类型下可以包含多个文档，每个文档可以包含多个域。

物理设计

ElasticSearch在后台把每个索引划分成多个分片，每分分片可以在集群中的不同服务间迁移。

逻辑设计

一个索引类型中，包含多个文档，比如说文档1，文档2。当我们索引一篇文档时，可以通过这样的顺序找到它：索引 > 类型 > 文档ID，通过这个组合我们就能索引到某个具体的文档。注意：ID不必是整数，实际上它是个字符串。

4. IK分词器

安装

1、下载-github: https://github.com/medcl/elasticsearch-analysis-ik/releases/tag/v7.6.1

2、下载完毕之后，在ElasticSearch的plugins文件夹新建ik文件夹，并把解压内容复制进去

3、在ElasticSearch的bin目录下打开命令行，输入elasticsearch-plugin list，可以看到所有插件

  1. PS C:\Java\es\elasticsearch-7.6.1\bin> .\elasticsearch-plugin list
  2. future versions of Elasticsearch will require Java 11; your Java version from [C:\Program Files\Java\jdk1.8.0_221\jre] does not meet this requirement
  3. ik

说明

IK提供了两个分词算法：ik_smart和ik_max_word，其中ik_smart为最少切分（不存在重复切分），ik_max_word为最细粒度划分（任何可能组成词的都会分出来）。

测试

ik_smart测试：

  1. GET _analyze
  2. {
  3.   "analyzer": "ik_smart",
  4.   "text": "枫叶凋零"
  5. }
  1. {
  2.   "tokens" : [
  3.     {
  4.       "token" : "枫叶",
  5.       "start_offset" : 0,
  6.       "end_offset" : 2,
  7.       "type" : "CN_WORD",
  8.       "position" : 0
  9.     },
  10.     {
  11.       "token" : "凋零",
  12.       "start_offset" : 2,
  13.       "end_offset" : 4,
  14.       "type" : "CN_WORD",
  15.       "position" : 1
  16.     }
  17.   ]
  18. }

ik_max_word分词：

  1. GET _analyze
  2. {
  3.   "analyzer": "ik_max_word",
  4.   "text": "文化工作"
  5. }
  1. {
  2.   "tokens" : [
  3.     {
  4.       "token" : "文化工作",
  5.       "start_offset" : 0,
  6.       "end_offset" : 4,
  7.       "type" : "CN_WORD",
  8.       "position" : 0
  9.     },
  10.     {
  11.       "token" : "文化",
  12.       "start_offset" : 0,
  13.       "end_offset" : 2,
  14.       "type" : "CN_WORD",
  15.       "position" : 1
  16.     },
  17.     {
  18.       "token" : "化工",
  19.       "start_offset" : 1,
  20.       "end_offset" : 3,
  21.       "type" : "CN_WORD",
  22.       "position" : 2
  23.     },
  24.     {
  25.       "token" : "工作",
  26.       "start_offset" : 2,
  27.       "end_offset" : 4,
  28.       "type" : "CN_WORD",
  29.       "position" : 3
  30.     }
  31.   ]
  32. }

自定义分词

自己需要的词，比如名字，需要自己加入到分词器的字典中。

需要在ElasticSearch的plugins/ik/config下新建一个字典khighness.dic(扩展名为.dic)，在里面添加词，比如自己的名字，然后在IKAnalyzer.cfg.xml中添加如下配置：

  1. <entry key="ext_dict">khighness.dic</entry>

5. CRUD

测试

（1）创建索引

  • 字符串类型：text、keyword
  • 数值类型：long、integer、short、byte、double、float、half、scaled float
  • 日期类型：date
  • 布尔类型：boolean
  • 二进制类型：binary
  1. PUT /k
  2. {
  3.   "mappings": {
  4.     "properties": {
  5.       "name": {
  6.         "type": "text"
  7.       },
  8.       "age": {
  9.         "type": "long"
  10.       },
  11.       "birth": {
  12.         "type": "date"
  13.       }
  14.     }
  15.   }
  16. }

（2）添加字段

  1. PUT /k/_mapping
  2. {
  3.   "properties": {
  4.     "phone": {
  5.       "type": "text"
  6.     }
  7.   }
  8. }

（3）添加数据

  1. POST k/_doc/1
  2. {
  3.   "name":"Khighness",
  4.   "age":19,
  5.   "birth":"2001-09-11",
  6.   "phone":"13611339955"
  7. }
  8. POST k/_doc/2
  9. {
  10.   "name":"FlowerK",
  11.   "age":17,
  12.   "birth":"2001-07-24",
  13.   "phone":"18371398911"
  14. }

（4）查询所有

  1. GET k/_search

（5）查询指定ID

  1. GET k/_doc/1

（6）条件查询

  1. GET k/_search?q=age:19

（7）指定ID全量修改

  1. POST k/_doc/1
  2. {
  3.   "name":"Khighness",
  4.   "age":19,
  5.   "birth":"2001-09-11",
  6.   "phone":"1823676372"
  7. }

（8）指定id部分字段修改

  1. POST k/_update/2
  2. {
  3.   "doc": {
  4.     "age":18
  5.   }
  6. }

（9）删除索引

  1. DELETE /k

（10）查看索引情况

  1. GET _cat/indices?v

（11）获取健康值

  1. GET _cat/indices?v

6. 集成

Maven

  1. <parent>
  2.     <groupId>org.springframework.boot</groupId>
  3.     <artifactId>spring-boot-starter-parent</artifactId>
  4.     <version>2.3.2.RELEASE</version>
  5. </parent>
  7. <properties>
  8.     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  9.     <maven.compiler.source>1.8</maven.compiler.source>
  10.     <maven.compiler.target>1.8</maven.compiler.target>
  11.     <java.version>1.8</java.version>
  12.     <elasticsearch.version>7.6.2</elasticsearch.version>
  13.     <fastjson.version>1.2.75</fastjson.version>
  14. </properties>
  16. <dependencies>
  17.     <dependency>
  18.         <groupId>org.springframework.boot</groupId>
  19.         <artifactId>spring-boot-starter-web</artifactId>
  20.     </dependency>
  21.     <dependency>
  22.         <groupId>org.springframework.boot</groupId>
  23.         <artifactId>spring-boot-starter-test</artifactId>
  24.     </dependency>
  25.     <dependency>
  26.         <groupId>org.springframework.boot</groupId>
  27.         <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
  28.     </dependency>
  29.     <dependency>
  30.         <groupId>org.elasticsearch.client</groupId>
  31.         <artifactId>elasticsearch-rest-high-level-client</artifactId>
  32.         <version>${elasticsearch.version}</version>
  33.     </dependency>
  34.     <dependency>
  35.         <groupId>org.projectlombok</groupId>
  36.         <artifactId>lombok</artifactId>
  37.     </dependency>
  38.     <dependency>
  39.         <groupId>com.alibaba</groupId>
  40.         <artifactId>fastjson</artifactId>
  41.         <version>${fastjson.version}</version>
  42.     </dependency>
  43. </dependencies>

添加客户端bean

  1. import org.apache.http.HttpHost;
  2. import org.elasticsearch.client.RestClient;
  3. import org.elasticsearch.client.RestHighLevelClient;
  4. import org.springframework.context.annotation.Bean;
  5. import org.springframework.context.annotation.Configuration;
  7. /**
  8.  * @author KHighness
  9.  * @since 2021-03-20
  10.  */
  11. @Configuration
  12. public class ElasticSearchConfig {
  13.     @Bean
  14.     public RestHighLevelClient restHighLevelClient() {
  15.         RestHighLevelClient client = new RestHighLevelClient(
  16.                 RestClient.builder(
  17.                         new HttpHost("127.0.0.1", 9200, "http")));
  18.         return client;
  19.     }
  20. }

API测试

  1. import com.alibaba.fastjson.JSON;
  2. import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
  3. import org.elasticsearch.action.bulk.BulkRequest;
  4. import org.elasticsearch.action.bulk.BulkResponse;
  5. import org.elasticsearch.action.delete.DeleteRequest;
  6. import org.elasticsearch.action.delete.DeleteResponse;
  7. import org.elasticsearch.action.get.GetRequest;
  8. import org.elasticsearch.action.get.GetResponse;
  9. import org.elasticsearch.action.index.IndexRequest;
  10. import org.elasticsearch.action.index.IndexResponse;
  11. import org.elasticsearch.action.search.SearchRequest;
  12. import org.elasticsearch.action.search.SearchResponse;
  13. import org.elasticsearch.action.support.master.AcknowledgedResponse;
  14. import org.elasticsearch.action.update.UpdateRequest;
  15. import org.elasticsearch.action.update.UpdateResponse;
  16. import org.elasticsearch.client.RequestOptions;
  17. import org.elasticsearch.client.RestHighLevelClient;
  18. import org.elasticsearch.client.indices.CreateIndexRequest;
  19. import org.elasticsearch.client.indices.CreateIndexResponse;
  20. import org.elasticsearch.client.indices.GetIndexRequest;
  21. import org.elasticsearch.common.unit.TimeValue;
  22. import org.elasticsearch.common.xcontent.XContentType;
  23. import org.elasticsearch.index.query.MatchAllQueryBuilder;
  24. import org.elasticsearch.index.query.QueryBuilders;
  25. import org.elasticsearch.index.query.TermQueryBuilder;
  26. import org.elasticsearch.search.SearchHit;
  27. import org.elasticsearch.search.builder.SearchSourceBuilder;
  28. import org.elasticsearch.search.fetch.subphase.FetchSourceContext;
  29. import org.junit.jupiter.api.Test;
  30. import org.springframework.beans.factory.annotation.Autowired;
  31. import org.springframework.beans.factory.annotation.Qualifier;
  32. import org.springframework.boot.test.context.SpringBootTest;
  33. import top.parak.entity.User;
  35. import java.io.IOException;
  36. import java.util.ArrayList;
  37. import java.util.List;
  38. import java.util.Map;
  39. import java.util.concurrent.TimeUnit;
  41. /**
  42.  * @author KHighness
  43.  * @since 2021-03-20
  44.  */
  45. @SpringBootTest
  46. public class ElasticSearchTest {
  48.     @Autowired
  49.     @Qualifier("restHighLevelClient")
  50.     private RestHighLevelClient client;
  52.     // 创建索引
  53.     @Test
  54.     void test1() throws IOException {
  55.         CreateIndexRequest request = new CreateIndexRequest("khighness");
  56.         CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT);
  57.         System.out.println(response);
  58.     }
  60.     // 是否存在
  61.     @Test
  62.     void test2() throws IOException {
  63.         GetIndexRequest request = new GetIndexRequest("khighness");
  64.         boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);
  65.         System.out.println(exists);
  66.     }
  68.     // 删除索引
  69.     @Test
  70.     void test3() throws IOException {
  71.         DeleteIndexRequest request = new DeleteIndexRequest("khighness");
  72.         AcknowledgedResponse response = client.indices().delete(request, RequestOptions.DEFAULT);
  73.         System.out.println(response.isAcknowledged());
  74.     }
  76.     // 创建文档
  77.     @Test
  78.     void test4() throws IOException {
  79.         User kHighness = User.builder().age(19).name("KHighness").build();
  80.         User flowerK = User.builder().age(19).name("FlowerK").build();
  81.         User rubbishK = User.builder().age(19).name("RubbishK").build();
  82.         User unknownK = User.builder().age(19).name("UnknownK").build();
  83.         IndexRequest request = new IndexRequest()
  84.                 .index("khighness").id("1").timeout(TimeValue.timeValueSeconds(1));
  85.         request.source(JSON.toJSONString(kHighness), XContentType.JSON);
  86.         IndexResponse response = client.index(request, RequestOptions.DEFAULT);
  87.         System.out.println(response.toString());
  88.     }
  90.     // 是否存在
  91.     @Test
  92.     void test5() throws IOException {
  93.         GetRequest request = new GetRequest("khighness", "1");
  94.         request.fetchSourceContext(new FetchSourceContext(false));
  95.         boolean exists = client.exists(request, RequestOptions.DEFAULT);
  96.         System.out.println(exists);
  97.     }
  99.     // 文档信息
  100.     @Test
  101.     void test6() throws IOException {
  102.         GetRequest request = new GetRequest("khighness", "1");
  103.         GetResponse response = client.get(request, RequestOptions.DEFAULT);
  104.         Map<String, Object> map = response.getSourceAsMap();
  105.         for (Map.Entry<String, Object> entry : map.entrySet()) {
  106.             System.out.println(entry.getKey() + ": " + entry.getValue());
  107.         }
  108.     }
  110.     // 更新文档
  111.     @Test
  112.     void test7() throws IOException {
  113.         UpdateRequest request = new UpdateRequest("khighness", "1");
  114.         request.timeout("1s");
  115.         User kHighness = User.builder().age(17).name("KHighness").build();
  116.         request.doc(JSON.toJSONString(kHighness), XContentType.JSON);
  117.         UpdateResponse response = client.update(request, RequestOptions.DEFAULT);
  118.         System.out.println(response.toString());
  119.     }
  121.     // 删除文档
  122.     @Test
  123.     void test8() throws IOException {
  124.         DeleteRequest request = new DeleteRequest("khighness", "1");
  125.         request.timeout("1s");
  126.         DeleteResponse response = client.delete(request, RequestOptions.DEFAULT);
  127.         System.out.println(response.status());
  128.     }
  130.     // 批量插入
  131.     @Test
  132.     void test9() throws IOException {
  133.         BulkRequest bulkRequest = new BulkRequest();
  134.         bulkRequest.timeout("10s");
  136.         List<User> userList = new ArrayList<>();
  137.         userList.add(User.builder().age(19).name("KHighness").build());
  138.         userList.add(User.builder().age(19).name("FlowerK").build());
  139.         userList.add(User.builder().age(19).name("RubbishK").build());
  140.         userList.add(User.builder().age(19).name("UnknownK").build());
  142.         for (int i = 0; i < userList.size(); i++) {
  143.             bulkRequest.add(new IndexRequest("khighness")
  144.                     .id("" + (i + 1)).source(JSON.toJSONString(userList.get(i)), XContentType.JSON));
  145.         }
  146.         BulkResponse response = client.bulk(bulkRequest, RequestOptions.DEFAULT);
  147.         System.out.println(response.hasFailures()); // 返回false代表成功
  148.     }
  150.     // 查询
  151.     @Test
  152.     void test10() throws IOException {
  153.         // 搜索请求
  154.         SearchRequest request = new SearchRequest("khighness");
  155.         // 查询条件
  156.         SearchSourceBuilder builder = new SearchSourceBuilder();
  157.         // 构建高亮
  158.         builder.highlighter();
  159.         // term：精确匹配
  160.         TermQueryBuilder tempQuery = QueryBuilders.termQuery("age", "19");
  161.         // match：模糊查询
  162.         MatchAllQueryBuilder matchQuery = QueryBuilders.matchAllQuery();
  163.         // 设置查询和限时
  164.         builder.query(tempQuery).timeout(new TimeValue(60, TimeUnit.SECONDS));
  165.         request.source(builder);
  166.         SearchResponse response = client.search(request, RequestOptions.DEFAULT);
  167.         System.out.println(JSON.toJSONString(response.getHits()));
  168.         for (SearchHit documentFields : response.getHits().getHits()) {
  169.             System.out.println(documentFields.getSourceAsMap());
  170.         }
  171.     }
  173. }