4.1、概述

通过上述3章的简单入门，可以大概了解 es 的入门使用，至此做一个小实战项目，模仿 “京东商品搜索”。
实战要求：

  • 能准备、分词筛选出商品信息（名称，价格，图片等）
  • 搜索词高亮

大致的效果如下
image.png

4.2、项目介绍

因为要模仿京东商品搜索，所以就需要通过爬虫从京东上拉部分数据下来，并且保存到es中，便于后续项目的搜索。并且因为还需要用到前端交互，所以引入基本的 vue 操作。以下是项目的大致步骤

  • 爬取京东商品数据，并解析（jsoup 能做基础爬虫操作）
  • 将数据录入 ES 中（数据批量存入ES）
  • 编写暴露ES 查询接口（全量或者分页）
  • 编写前端代码

  • 前后端联调（引入 Axios，便于前端 Ajax 网络请求）

    4.3、项目编写

    4.3.1、项目环境

    软件说明：

  • Java:1.8

  • Elastic Search：7.16.3
  • kibana：7.16.3
  • elasticsearch-head-master
  • node
  • vue
  • axios

image.png

4.3.2、后端

4.3.2.1、Maven

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  3.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  4.     <modelVersion>4.0.0</modelVersion>
  5.     <groupId>com.es</groupId>
  6.     <artifactId>springboot-es-jd</artifactId>
  7.     <version>0.0.1-SNAPSHOT</version>
  8.     <name>springboot-es-jd</name>
  9.     <description>springboot-es-jd</description>
  10.     <parent>
  11.         <groupId>org.springframework.boot</groupId>
  12.         <artifactId>spring-boot-starter-parent</artifactId>
  13.         <version>2.6.3</version>
  14.         <relativePath/>
  15.     </parent>
  17.     <properties>
  18.         <java.version>1.8</java.version>
  19.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  20.         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  21.         <spring-boot.version>2.4.1</spring-boot.version>
  22.         <fast-json.version>1.2.72</fast-json.version>
  23.         <jsoup.version>1.14.3</jsoup.version>
  24.         <fast-json.version>1.2.79</fast-json.version>
  25.     </properties>
  27.     <dependencies>
  28.         <dependency>
  29.             <groupId>org.springframework.boot</groupId>
  30.             <artifactId>spring-boot-starter-thymeleaf</artifactId>
  31.         </dependency>
  32.         <!--        web 相关依赖-->
  33.         <dependency>
  34.             <groupId>org.springframework.boot</groupId>
  35.             <artifactId>spring-boot-starter-web</artifactId>
  36.         </dependency>
  37.         <!--        java 爬虫-->
  38.         <dependency>
  39.             <groupId>org.jsoup</groupId>
  40.             <artifactId>jsoup</artifactId>
  41.             <version>${jsoup.version}</version>
  42.         </dependency>
  43.         <!--        es 相关依赖-->
  44.         <dependency>
  45.             <groupId>org.springframework.boot</groupId>
  46.             <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
  47.         </dependency>
  49.         <!--        json -->
  50.         <dependency>
  51.             <groupId>com.alibaba</groupId>
  52.             <artifactId>fastjson</artifactId>
  53.             <version>${fast-json.version}</version>
  54.         </dependency>
  56.         <!--        lombok-->
  57.         <dependency>
  58.             <groupId>org.projectlombok</groupId>
  59.             <artifactId>lombok</artifactId>
  60.             <optional>true</optional>
  61.         </dependency>
  63.         <!--        常规基础依赖-->
  64.         <dependency>
  65.             <groupId>org.springframework.boot</groupId>
  66.             <artifactId>spring-boot-devtools</artifactId>
  67.             <optional>true</optional>
  68.         </dependency>
  70.         <dependency>
  71.             <groupId>org.springframework.boot</groupId>
  72.             <artifactId>spring-boot-starter-test</artifactId>
  73.         </dependency>
  74.     </dependencies>
  76.     <build>
  77.         <plugins>
  78.             <plugin>
  79.                 <groupId>org.springframework.boot</groupId>
  80.                 <artifactId>spring-boot-maven-plugin</artifactId>
  81.                 <configuration>
  82.                     <excludes>
  83.                         <exclude>
  84.                             <groupId>org.projectlombok</groupId>
  85.                             <artifactId>lombok</artifactId>
  86.                         </exclude>
  87.                     </excludes>
  88.                 </configuration>
  89.             </plugin>
  90.         </plugins>
  91.     </build>
  93. </project>

4.3.2.2、配置文件

修改配置文件，去除 thymeleaf 的页面缓存。还有一个ElasticSearchConfig ES 配置类，但是内容和 3_SpringBoot集成ES 里面内容一致，就不再重复

  1. spring:
  2.   thymeleaf:
  3.     cache: false

4.3.2.3、爬取京东商品数据

通过 jsoup 工具进行简易的网页数据爬取，如果需要进一步加强，可先爬取解析页面的分页信息，再通过请求地址 配置分页数据循环爬取数据，es 也支持 from 和 size 进行后续数据分页查询

  1. package com.es.jsoup;
  3. import com.es.data.ESDataOperator;
  4. import com.es.model.Goods;
  5. import org.jsoup.Jsoup;
  6. import org.jsoup.nodes.Document;
  7. import org.jsoup.nodes.Element;
  8. import org.jsoup.select.Elements;
  9. import org.springframework.beans.factory.annotation.Autowired;
  10. import org.springframework.stereotype.Component;
  12. import java.io.IOException;
  13. import java.net.URL;
  14. import java.util.ArrayList;
  15. import java.util.List;
  17. /**
  18.  * 爬取，解析京东数据
  19.  */
  20. @Component
  21. public class JDData {
  23.     @Autowired
  24.     private ESDataOperator esDataOperator;
  26.     static String url = "https://search.jd.com/Search?enc=utf-8&keyword=";
  29.     /**
  30.      * 爬取数据
  31.      *
  32.      * @param keyword
  33.      * @return
  34.      */
  35.     private Document getJDData(String keyword) {
  36.         try {
  37.             return Jsoup.parse(new URL(url + keyword), 1000 * 30);
  38.         } catch (IOException e) {
  39.             e.printStackTrace();
  40.         }
  41.         return null;
  42.     }
  44.     /**
  45.      * 解析数据
  46.      *
  47.      * @param keyword
  48.      * @return
  49.      */
  50.     private List<Goods> analysis(String keyword) throws IOException {
  51.         Document document = getJDData(keyword);
  52.         //获取商品列表
  53.         Element goodsElement = document.getElementById("J_goodsList");
  54.         //商品一个个项
  55.         List<Element> goodsLi = goodsElement.getElementsByTag("li");
  57.         //解析
  58.         List<Goods> goodsList = new ArrayList<>();
  59.         for (Element element : goodsLi) {
  60.             goodsList.add(parse2Goods(element));
  61.         }
  62.         return goodsList;
  63.     }
  65.     /**
  66.      * li 节点解析数据
  67.      *
  68.      * @param element
  69.      * @return
  70.      */
  71.     private Goods parse2Goods(Element element) {
  72.         Goods goods = new Goods();
  73.         //图片
  74.         Elements imgElements = element.getElementsByTag("img");
  75.         String img = imgElements.first().attr("data-lazy-img");
  76.         goods.setImg(img);
  77.         //名称
  78.         Elements nameElement = element.select(".p-name");
  79.         String name = nameElement.tagName("a").text();
  80.         goods.setName(name);
  81.         //价格
  82.         String price = element.select(".p-price").tagName("i").text();
  83.         goods.setPrice(price);
  84.         return goods;
  85.     }
  87.     public List<Goods> search(String keyword) throws IOException {
  88.         return esDataOperator.search(keyword);
  89.     }
  92.     /**
  93.      * 组装数据
  94.      *
  95.      * @param keyword
  96.      * @return
  97.      */
  98.     public void add2ES(String keyword) throws IOException {
  99.         //判断库是否存在
  100.         Boolean exists = esDataOperator.indexExists();
  101.         if (!exists) {
  102.             esDataOperator.createIndex();
  103.         }
  104.         //不存在则新增
  105.         List<Goods> goodsList = analysis(keyword);
  106.         //批量入库
  107.         esDataOperator.bulkAdd(goodsList);
  108.     }
  109. }

4.3.3.4、商品实体类

  1. @Data
  2. @ToString
  3. public class Goods {
  5.     private String name;
  7.     private String price;
  9.     private String img;
  11. }

4.3.3.5、ES数据操作

  1. package com.es.data;
  3. import com.alibaba.fastjson.JSON;
  4. import com.es.model.Goods;
  5. import org.elasticsearch.action.bulk.BulkRequest;
  6. import org.elasticsearch.action.bulk.BulkResponse;
  7. import org.elasticsearch.action.index.IndexRequest;
  8. import org.elasticsearch.action.search.SearchRequest;
  9. import org.elasticsearch.action.search.SearchResponse;
  10. import org.elasticsearch.client.RequestOptions;
  11. import org.elasticsearch.client.RestHighLevelClient;
  12. import org.elasticsearch.client.indices.CreateIndexRequest;
  13. import org.elasticsearch.client.indices.CreateIndexResponse;
  14. import org.elasticsearch.client.indices.GetIndexRequest;
  15. import org.elasticsearch.common.text.Text;
  16. import org.elasticsearch.common.xcontent.XContentType;
  17. import org.elasticsearch.index.query.MatchQueryBuilder;
  18. import org.elasticsearch.index.query.QueryBuilders;
  19. import org.elasticsearch.search.SearchHit;
  20. import org.elasticsearch.search.builder.SearchSourceBuilder;
  21. import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
  22. import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
  23. import org.springframework.beans.factory.annotation.Autowired;
  24. import org.springframework.stereotype.Component;
  26. import java.io.IOException;
  27. import java.util.ArrayList;
  28. import java.util.List;
  29. import java.util.Map;
  31. /**
  32.  * 数据操作
  33.  */
  34. @Component
  35. public class ESDataOperator {
  37.     @Autowired
  38.     private RestHighLevelClient restHighLevelClient;
  40.     private final static String GOODS_INDEX = "goods";
  42.     /**
  43.      * 判断是否存在
  44.      *
  45.      * @return
  46.      * @throws IOException
  47.      */
  48.     public Boolean indexExists() throws IOException {
  49.         GetIndexRequest indexRequest = new GetIndexRequest(GOODS_INDEX);
  50.         return restHighLevelClient.indices().exists(indexRequest, RequestOptions.DEFAULT);
  51.     }
  53.     /**
  54.      * 创建索引
  55.      *
  56.      * @return
  57.      * @throws IOException
  58.      */
  59.     public Boolean createIndex() throws IOException {
  60.         CreateIndexRequest indexRequest = new CreateIndexRequest(GOODS_INDEX);
  62.         CreateIndexResponse response = restHighLevelClient.indices().create(indexRequest, RequestOptions.DEFAULT);
  63.         return response.isAcknowledged();
  64.     }
  66.     /**
  67.      * 批量追加 es 数据
  68.      *
  69.      * @param goodsList
  70.      * @return
  71.      */
  72.     public boolean bulkAdd(List<Goods> goodsList) throws IOException {
  73.         BulkRequest bulkRequest = new BulkRequest();
  75.         for (Goods goods : goodsList) {
  76.             IndexRequest indexRequest = new IndexRequest(GOODS_INDEX);
  77.             indexRequest.source(JSON.toJSONString(goods), XContentType.JSON);
  78.             bulkRequest.add(indexRequest);
  79.         }
  81.         BulkResponse response = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
  82.         return !response.hasFailures();
  83.     }
  85.     public List<Goods> search(String keyword) throws IOException {
  86.         SearchRequest searchRequest = new SearchRequest(GOODS_INDEX);
  88.         SearchSourceBuilder requestBuilder = new SearchSourceBuilder();
  90.         MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("name", keyword);
  92.         //高亮
  93.         HighlightBuilder highlightBuilder = new HighlightBuilder();
  94.         highlightBuilder.preTags("<span style='color:red'>");
  95.         highlightBuilder.postTags("</span>");
  96.         highlightBuilder.field("name");
  97.         //设置查询
  98.         requestBuilder.highlighter(highlightBuilder);
  99.         requestBuilder.query(queryBuilder);
  101.         searchRequest.source(requestBuilder);
  103.         SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
  104.         SearchHit[] searchHits = searchResponse.getHits().getHits();
  106.         List<Goods> goodsList = new ArrayList<>();
  107.         //常规
  108. //        for (int i = 0; i < searchHits.length; i++) {
  109. //            Goods goods = new Goods();
  110. //            Map<String, Object> sourceMap = searchHits[i].getSourceAsMap();
  111. //            goods.setName((String) sourceMap.get("name"));
  112. //            goods.setImg((String) sourceMap.get("img"));
  113. //            goods.setPrice((String) sourceMap.get("price"));
  114. //            goodsList.add(goods);
  115. //        }
  117.         //设置替换高亮
  118.         for (int i = 0; i < searchHits.length; i++) {
  119.             Goods goods = new Goods();
  120.             Map<String, Object> sourceMap = searchHits[i].getSourceAsMap();
  121.             goods.setName(highlighter(searchHits[i], "name"));
  122.             goods.setImg((String) sourceMap.get("img"));
  123.             goods.setPrice((String) sourceMap.get("price"));
  124.             goodsList.add(goods);
  125.         }
  126.         return goodsList;
  127.     }
  129.     /**
  130.      * 高亮转换
  131.      *
  132.      * @param searchHit
  133.      * @param fileName
  134.      * @return
  135.      */
  136.     private String highlighter(SearchHit searchHit, String fileName) {
  137.         Map<String, HighlightField> highlighterMap = searchHit.getHighlightFields();
  138.         HighlightField highlightField = highlighterMap.get("name");
  139.         StringBuffer sb = new StringBuffer();
  140.         if (highlightField != null) {
  141.             Text[] texts = highlightField.fragments();
  142.             for (Text text : texts) {
  143.                 sb.append(text.toString());
  144.             }
  145.         }
  146.         return sb.toString();
  147.     }
  148. }

4.3.3.6、编写后端业务（新增、查询）

  1. package com.es.service;
  3. import com.es.jsoup.JDData;
  4. import com.es.model.Goods;
  5. import org.springframework.beans.factory.annotation.Autowired;
  6. import org.springframework.stereotype.Service;
  8. import java.io.IOException;
  9. import java.util.List;
  11. @Service
  12. public class SearchService {
  14.     @Autowired
  15.     private JDData jdData;
  17.     public List<Goods> search(String keyword) throws IOException {
  18.         return jdData.search(keyword);
  19.     }
  21.     public void add2ES(String keyword) throws IOException {
  22.         jdData.add2ES(keyword);
  23.     }
  24. }

4.3.3.7、后端接口

提供两个接口

  • add2ES：根据搜索词，从京东爬取第一页商品部分数据
  • search：根据搜索词，搜索本地 ES 服务数据 ```java package com.es.controller;

import com.es.model.Goods; import com.es.service.SearchService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.*;

import java.io.IOException; import java.util.List;

@RestController public class SearchController {

  1. @Autowired
  2. private SearchService service;
  4. @GetMapping("/search/{keyword}")
  5. public List<Goods> search(@PathVariable("keyword") String keyword) throws IOException {
  6.     return service.search(keyword);
  7. }
  9. @GetMapping("/{keyword}")
  10. public void add2ES(@PathVariable("keyword") String keyword) throws IOException {
  11.     service.add2ES(keyword);
  12. }

}

  1. <a name="ceJUb"></a>
  2. ### 4.3.3、前端
  3. 前端通过 vue 进行快速页面开发，并借助 axios 进行快速 ajax 请求发送，文件结构如下：<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/25709938/1643340486657-ef846e74-895a-4934-b812-04b70d88ea1f.png#clientId=u85e1f7b6-f9fa-4&from=paste&height=200&id=u99f759b5&margin=%5Bobject%20Object%5D&name=image.png&originHeight=400&originWidth=646&originalType=binary&ratio=1&size=27889&status=done&style=none&taskId=u74562e91-7d11-40db-be77-2900800560d&width=323)
  4. <a name="psaAa"></a>
  5. #### 4.3.3.1、css
  6. ```css
  7. /*app*/
  8. #app {
  9.     width: 100%;
  10.     height: 100%;
  11. }
  13. /*header*/
  14. .header {
  15.     background: #e3e4e5;
  16.     color: #999;
  17.     height: 23px;
  18.     padding-top: 5px;
  19.     font-size: 6px;
  20. }
  22. .header .left {
  23.     padding-left: 50px;
  24.     float: left;
  25. }
  27. .header .left span {
  28.     margin-right: 20px;
  29.     font-size: 14px;
  30. }
  32. .header .right {
  33.     float: right;
  34.     padding-right: 50px;
  35. }
  37. .header .right span {
  38.     font-size: 14px;
  39.     margin-right: 15px;
  40. }
  43. /*search*/
  44. .search {
  45.     display: flex;
  46.     padding-top: 10px;
  47.     padding-bottom: 10px;
  48. }
  50. .search > .logo {
  51.     width: 20%;
  52.     margin: 20px 30px;
  53. }
  55. .logo img {
  56.     width: 120px;
  57. }
  59. .search-content {
  60.     display: grid;
  61.     padding-top: 20px;
  62. }
  64. .search-component {
  65.     display: flex;
  66. }
  68. .search-content .search-input input {
  69.     height: 28px;
  70.     width: 626px;
  71.     border: solid red;
  72. }
  74. .search-content .search-btn button {
  75.     height: 35px;
  76.     width: 80px;
  77.     border: solid red;
  78.     background-color: red;
  79.     color: white;
  80.     font-size: 16px;
  81. }
  83. .host-keyword span {
  84.     color: #999;
  85.     padding-right: 10px;
  86.     margin-right: 10px;
  87.     border-right: solid #e3e4e5;
  88.     font-size: 12px;
  89. }
  91. .line {
  92.     height: 2px;
  93.     background-color: red;
  94. }
  96. /*body*/
  97. .body {
  98.     display: flex;
  99.     padding-top: 20px;
  100.     flex-wrap: wrap;
  101.     padding-left: 50px;
  102.     padding-right: 50px;
  103. }
  105. .body .goods-item {
  106.     width: 17%;
  107.     padding: 15px;
  108.     border: 2px solid rgba(100, 100, 100, 0);
  109. }
  111. .goods-item .book-price {
  112.     color: red;
  113. }
  115. .goods-item .book-name {
  116.     font-size: 12px;
  117.     color: #666;
  118.     padding: 2px 5px;
  119. }
  121. .body .goods-item:hover {
  122.     border: solid 2px #e3e4e5;
  123.     color: red;
  124. }

4.3.3.2 html

页面需要放在 template 中，防止不被识别

  1. <!DOCTYPE html>
  2. <html lang="en" xmlns="http://www.w3.org/1999/html">
  3. <head>
  4.     <meta charset="UTF-8">
  5.     <title>Title</title>
  6.         <script th:src="@{/js/axios.min.js}"></script>
  7.         <script th:src="@{/js/vue.js}"></script>
  8.         <link type="text/css" rel="styleSheet" th:href="@{/css/index.css}"/>
  9. <!--    <script src="../static/js/vue.js"></script>-->
  10. <!--    <script src="../static/js/axios.min.js"></script>-->
  11.     <!--    <link rel="stylesheet" href="../static/css/index.css">-->
  12. </head>
  13. <body>
  14. <div id="app">
  15.     <div class="header">
  16.         <div class="left">
  17.             <span>京东首页</span>
  18.             <span>福建</span>
  19.         </div>
  20.         <div class="right">
  21.             <span>你好 请登录</span>
  22.             <span>免费注册</span>
  23.             <span>我的订单</span>
  24.             <span>我的京东</span>
  25.             <span>京东会员</span>
  26.             <span>企业采购</span>
  27.             <span>客户服务</span>
  28.             <span>网站导航</span>
  29.         </div>
  30.     </div>
  31.     <div class="search">
  32.         <div class="logo">
  33.                         <img th:src="@{/img/logo.png}">
  34. <!--            <img src="../static/img/logo.png">-->
  35.         </div>
  36.         <div class="search-content">
  37.             <div class="search-component">
  38.                 <div class="search-input">
  39.                     <input v-model="keyword"/>
  40.                 </div>
  41.                 <div class="search-btn">
  42.                     <button @click="search">搜索</button>
  43.                 </div>
  44.             </div>
  45.             <div class="host-keyword">
  46.               <span v-for="key in hostKeyword">
  47.                   {{key}}
  48.               </span>
  49.             </div>
  50.         </div>
  51.     </div>
  52.     <div class="line"></div>
  53.     <div class="body">
  54.         <div v-for="item in result" class="goods-item">
  55.             <div class="book-img">
  56.                 <img :src="item.img">
  57.             </div>
  58.             <div class="book-price">
  59.                 <div class="price-text">{{item.price}}</div>
  60.             </div>
  61.             <div class="book-name">
  62.                 <div class="price-text" v-html="item.name"></div>
  63.             </div>
  64.         </div>
  65.     </div>
  66. </div>
  67. </body>
  68. <script>
  69.     new Vue({
  70.         el: "#app",
  71.         data: {
  72.             keyword: "",
  73.             hostKeyword: ["python", "spring", "mysql", "spring boot", "java核心技术", "java web", "c++", "vue", "linux", "java从入门到精通"],
  74.             result: []
  75.         },
  76.         methods: {
  77.             search: function () {
  78.                 let url = "http://localhost:8080/search/" + this.keyword;
  79.                 axios.get(url).then(res => {
  80.                     console.log(res.data)
  81.                     this.result = res.data
  82.                 })
  83.             }
  84.         }
  85.     })
  86. </script>
  87. </html>

4.3.4、测试

  • 启动 es
  • 启动 springboot-es-jd 项目
  • 先通过浏览器请求 http://localhost:8080/java 进行从京东抓取部分数据入es，其中 java 可以是任何关键字

浏览器访问：http://localhost:8080/，在搜索框中输入搜索关键字，效果如下
image.png