概览

  1. 主要分两种
  2. - URI Search
  3. 示例: curl -XGET "http://localhost:9200/test_index/_search?q=cusomer"
  5. - Request Body Search
  6. 示例: curl -XGET "http://localhost:9200/test_index/_search" -H 'Content-Type: application/json' -d
  7. '{
  8.     "query":{
  9.         "match_all":{}
  10.     }
  11. }'

URI Search

  1. 示例
  2. GET /movies/_search?q=2012&df=title&sort=year:desc&from=0&size=10&timeout=1s
  3. {
  4.     "profile":true
  5. }
  7. - q 指定查询语句, 使用Query String Syntax
  8. - df 默认字段, 不指定会对所有字段进行查询, q=2012&df=title 等价于 q=title:2012
  9. - sort 排序, from size 分页
  10. - profile 可以查看查询是如何被执行的
  12. 具体不同查询类型操作, 略

Request Body Search

  1. 示例
  2. POST /movies/_search
  3. {
  4.   "script_fileds":{ // 脚本字段
  5.     "new_filed":{
  6.         "script":{
  7.           "lang":"painless",
  8.         "source":"doc['order_date'].value+'hello'"
  9.       }
  10.     }
  11.   },
  12.   "_source":["order_date","category.keyword"], // 指定返回的字段
  13.   "sort":[{"order_date":"desc"}],
  14.     "from":10,
  15.   "size":20,
  16.   "query":{
  17.       "match_all":{}
  18.   }
  19. }
  21. - 使用查询表达式
  22. POST movies/_search
  23. {
  24.   "query": {
  25.     "match": {
  26.       "title": {
  27.         "query": "last christmas",
  28.         "operator": "and"
  29.       }
  30.     }
  31.   }
  32. }
  34. POST movies/_search
  35. {
  36.   "query": {
  37.     "match_phrase": {
  38.       "title":{
  39.         "query": "one love",
  40.         "slop": 1 
  41.       }
  42.     }
  43.   }
  44. }

Query String & Simple Query String

简单来说后者禁止了一些高级查询并忽略了一些语法错误

Mapping

Mapping类似数据库的schema定义, 作用:

  • 定义索引中字段名称
  • 定义字段数据类型
  • 定义字段的倒排索引的相关配置, 如analyzed, analyzer

字段数据类型:

  • 简单类型
    • Text/Keyword
    • Date
    • Integer/Floating
    • Boolean
    • IPv4/IPv6
  • 复杂类型-对象和嵌套对象
  • 特殊类型
    • geo_point&geo_shape/percolator

Dynamic Mapping

写入文档时, 如果索引不存在, 则会自动创建索引, 此时es会根据文档信息, 推断出字段类型
image.png

  1. 示例:
  2. # dynamic mapping，推断字段的类型
  3. PUT mapping_test/_doc/1
  4. {
  5.     "uid" : "123",
  6.     "isVip" : false,
  7.     "isAdmin": "true",
  8.     "age":19,
  9.     "heigh":180
  10. }
  12. # 查看 Dynamic
  13. GET mapping_test/_mapping
  14. 从下面结果可以看到uid,isAdmin只会被识别为字符串, 被设置为Text, 并添加了keyword子字段
  16. {
  17.   "mapping_test" : {
  18.     "mappings" : {
  19.       "properties" : {
  20.         "age" : {
  21.           "type" : "long"
  22.         },
  23.         "heigh" : {
  24.           "type" : "long"
  25.         },
  26.         "isAdmin" : {
  27.           "type" : "text",
  28.           "fields" : {
  29.             "keyword" : {
  30.               "type" : "keyword",
  31.               "ignore_above" : 256
  32.             }
  33.           }
  34.         },
  35.         "isVip" : {
  36.           "type" : "boolean"
  37.         },
  38.         "uid" : {
  39.           "type" : "text",
  40.           "fields" : {
  41.             "keyword" : {
  42.               "type" : "keyword",
  43.               "ignore_above" : 256
  44.             }
  45.           }
  46.         }
  47.       }
  48.     }
  49.   }
  50. }

Dynamic Mapping配置
image.png

  1. #修改为dynamic false
  2. PUT dynamic_mapping_test/_mapping
  3. {
  4.   "dynamic": "false"
  5. }

显式定义Mapping

  1. 示例:
  2. PUT movies
  3. {
  4.     "mappings":{...}
  5. }
  7. 初建时可以基于dynamic mapping进行修改

常见参数:

  • index, 控制当前字段是否被索引, 默认为true
  • index options, 控制倒排索引记录的内容, text默认position, 其他默认docs
    • docs, 记录doc id
    • freqs, 记录doc id/term frequencies
    • positions. 记录doc id/term frequencies/term position
    • offsets, 记录doc id/term frequencies/term position/character offects
  • null_value, 需要对Null值实现搜索, 只有keyword类型支持设定null_value
  • copy_to, 字段拷贝 ```json

    设置 index 为 false

    PUT users { “mappings” : {
    1. "properties" : {
    2.   "firstName" : {
    3.     "type" : "text"
    4.   },
    5.   "lastName" : {
    6.     "type" : "text"
    7.   },
    8.   "mobile" : {
    9.     "type" : "text",
    10.     "index": false
    11.   }
    12. }
    } }

设定Null_value

PUT users { “mappings” : { “properties” : { “firstName” : { “type” : “text” }, “lastName” : { “type” : “text” }, “mobile” : { “type” : “keyword”, “null_value”: “NULL” }

  1.   }
  2. }

}

设置 Copy to

PUT users { “mappings”: { “properties”: { “firstName”:{ “type”: “text”, “copy_to”: “fullName” }, “lastName”:{ “type”: “text”, “copy_to”: “fullName” } } } }

PUT users/_doc/1 { “firstName”:”Ruan”, “lastName”: “Yiming” }

GET users/_search?q=fullName:(Ruan Yiming)

POST users/_search { “query”: { “match”: { “fullName”:{ “query”: “Ruan Yiming”, “operator”: “and” } } } }

  2. <a name="NjWSn"></a>
  3. ### 多字段特性
  5. - mapping中的keyword指的是不需要分词的精确值, text是会被分词的.
  6. - text类型字段自动生成mapping时, 会自动添加一个keyword子字段供精确检索
  7. - 此外, 还可以添加自定义子字段, 采用自定义分词器, 以满足不同条件下的搜索
  9. 自定义分词补充
  11. 1. Character Filters, 提前进行文本处理, 如增加删除以及替换字符, 可配置多个
  13. 自带的Character Filters:
  15. - HTML strip, 去除html标签
  16. - Mapping, 字符串替换
  17. - Pattern replace, 正则匹配替换
  19. 2. Tokenizer, 将原始文本按照一定的规则, 切分为词
  21. es内置的Tokenizers:<br />whitespace/standard/uax_url_email/pattern/keyword/path hierarchy
  23. 3. Token Filters, 将Tokenizer输出的单词进行增加修改删除
  25. 自带的Token Filters:<br />Lowercase/stop/synonym(添加近义词)
  27. ```json
  28. PUT logs/_doc/1
  29. {"level":"DEBUG"}
  31. GET /logs/_mapping
  33. POST _analyze
  34. {
  35.   "tokenizer":"keyword",
  36.   "char_filter":["html_strip"],
  37.   "text": "<b>hello world</b>"
  38. }
  41. POST _analyze
  42. {
  43.   "tokenizer":"path_hierarchy",
  44.   "text":"/user/ymruan/a/b/c/d/e"
  45. }
  49. #使用char filter进行替换
  50. POST _analyze
  51. {
  52.   "tokenizer": "standard",
  53.   "char_filter": [
  54.       {
  55.         "type" : "mapping",
  56.         "mappings" : [ "- => _"]
  57.       }
  58.     ],
  59.   "text": "123-456, I-test! test-990 650-555-1234"
  60. }
  62. //char filter 替换表情符号
  63. POST _analyze
  64. {
  65.   "tokenizer": "standard",
  66.   "char_filter": [
  67.       {
  68.         "type" : "mapping",
  69.         "mappings" : [ ":) => happy", ":( => sad"]
  70.       }
  71.     ],
  72.     "text": ["I am felling :)", "Feeling :( today"]
  73. }
  75. // white space and snowball
  76. GET _analyze
  77. {
  78.   "tokenizer": "whitespace",
  79.   "filter": ["stop","snowball"],
  80.   "text": ["The gilrs in China are playing this game!"]
  81. }
  84. // whitespace与stop
  85. GET _analyze
  86. {
  87.   "tokenizer": "whitespace",
  88.   "filter": ["stop","snowball"],
  89.   "text": ["The rain in Spain falls mainly on the plain."]
  90. }
  93. //remove 加入lowercase后，The被当成 stopword删除
  94. GET _analyze
  95. {
  96.   "tokenizer": "whitespace",
  97.   "filter": ["lowercase","stop","snowball"],
  98.   "text": ["The gilrs in China are playing this game!"]
  99. }
  101. //正则表达式
  102. GET _analyze
  103. {
  104.   "tokenizer": "standard",
  105.   "char_filter": [
  106.       {
  107.         "type" : "pattern_replace",
  108.         "pattern" : "http://(.*)",
  109.         "replacement" : "$1"
  110.       }
  111.     ],
  112.     "text" : "http://www.elastic.co"
  113. }

Index Template & Dynamic Template

Index Template

索引模板, 可以设定一个模板固化mapping和setting, 并按照一定规则, 自动匹配到新创建的索引上

  • 模板仅在索引新建时有用, 修改模板不会影响已有的索引
  • 可以设定多个索引模板, 指定order, 多个设置会按规则merge
  • 优先级为: 用户指定 > order高的Index Template > order底的index Template ```json

    Create a default template

    PUT _template/template_default { “index_patterns”: [“*”], “order” : 0, “version”: 1, “settings”: { “number_of_shards”: 1, “number_of_replicas”:1 } }

PUT /_template/template_test { “index_patterns” : [“test*”], “order” : 1, “settings” : { “number_of_shards”: 1, “number_of_replicas” : 2 }, “mappings” : { “date_detection”: false, “numeric_detection”: true } }

查看template信息

GET /_template/template_default GET /_template/temp*

DELETE /_template/template_default DELETE /_template/template_test

  2. <a name="xfObH"></a>
  3. ### Dynamic Template
  4. 应用在某一个具体的索引上, 可以自定义一些字段类型推断的规则, 如:
  6. - is开头的字段都设置成boolean
  8. ```json
  9. #Dynaminc Mapping 根据类型和字段名
  10. DELETE my_index
  12. PUT my_index/_doc/1
  13. {
  14.   "firstName":"Ruan",
  15.   "isVIP":"true"
  16. }
  18. GET my_index/_mapping
  19. DELETE my_index
  21. #示例, 字符串设置成boolean, 字符串设置成keyword
  22. PUT my_index
  23. {
  24.   "mappings": {
  25.     "dynamic_templates": [
  26.             {
  27.         "strings_as_boolean": {
  28.           "match_mapping_type":   "string",
  29.           "match":"is*",
  30.           "mapping": {
  31.             "type": "boolean"
  32.           }
  33.         }
  34.       },
  35.       {
  36.         "strings_as_keywords": {
  37.           "match_mapping_type":   "string",
  38.           "mapping": {
  39.             "type": "keyword"
  40.           }
  41.         }
  42.       }
  43.     ]
  44.   }
  45. }
  48. DELETE my_index
  49. #示例, 结合路径, 组合姓名
  50. PUT my_index
  51. {
  52.   "mappings": {
  53.     "dynamic_templates": [
  54.       {
  55.         "full_name": {
  56.           "path_match":   "name.*",
  57.           "path_unmatch": "*.middle",
  58.           "mapping": {
  59.             "type":       "text",
  60.             "copy_to":    "full_name"
  61.           }
  62.         }
  63.       }
  64.     ]
  65.   }
  66. }
  69. PUT my_index/_doc/1
  70. {
  71.   "name": {
  72.     "first":  "John",
  73.     "middle": "Winston",
  74.     "last":   "Lennon"
  75.   }
  76. }
  78. GET my_index/_search?q=full_name:John

聚合分析

es的聚合(aggregation)是对数据进行统计分析的功能
image.png

image.png

  1. Elasticsearch聚合分析简介
  2. 课程Demo
  4. 需要通过Kibana导入Sample Data的飞机航班数据。具体参考“2.2节-Kibana的安装与界面快速浏览”
  5. #按照目的地进行分桶统计
  6. GET kibana_sample_data_flights/_search
  7. {
  8.     "size": 0,
  9.     "aggs":{
  10.         "flight_dest":{
  11.             "terms":{
  12.                 "field":"DestCountry"
  13.             }
  14.         }
  15.     }
  16. }
  20. #查看航班目的地的统计信息，增加平均，最高最低价格
  21. GET kibana_sample_data_flights/_search
  22. {
  23.     "size": 0,
  24.     "aggs":{
  25.         "flight_dest":{
  26.             "terms":{
  27.                 "field":"DestCountry"
  28.             },
  29.             "aggs":{
  30.                 "avg_price":{
  31.                     "avg":{
  32.                         "field":"AvgTicketPrice"
  33.                     }
  34.                 },
  35.                 "max_price":{
  36.                     "max":{
  37.                         "field":"AvgTicketPrice"
  38.                     }
  39.                 },
  40.                 "min_price":{
  41.                     "min":{
  42.                         "field":"AvgTicketPrice"
  43.                     }
  44.                 }
  45.             }
  46.         }
  47.     }
  48. }
  52. #价格统计信息+天气信息
  53. GET kibana_sample_data_flights/_search
  54. {
  55.     "size": 0,
  56.     "aggs":{
  57.         "flight_dest":{
  58.             "terms":{
  59.                 "field":"DestCountry"
  60.             },
  61.             "aggs":{
  62.                 "stats_price":{
  63.                     "stats":{
  64.                         "field":"AvgTicketPrice"
  65.                     }
  66.                 },
  67.                 "wather":{
  68.                   "terms": {
  69.                     "field": "DestWeather",
  70.                     "size": 5
  71.                   }
  72.                 }
  74.             }
  75.         }
  76.     }
  77. }
  79. 相关阅读
  81. https://www.elastic.co/guide/en/elasticsearch/reference/7.1/search-aggregations.html