Fielddata is disabled on text fields by default. Set fielddata=true on [event.dataset]

错误来源，在清理Elasticsearch索引时删除了Filebeat相关的索引之后在打开Kibana的Stack Monitoring页面出现如下错误。
image.png
随便找一台Elasticsearch服务器查看日志发现如下异常日志，建议查看所有的Elasticsearch服务器日志。
异常日志：
Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[:filebeat-, filebeat-*],

Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [event.dataset] in order to load fielddata in memory by uninverting the inverted index.
Note that this can however use significant memory. Alternatively use a keyword field instead.

详细异常日志如下：**

  1. [2020-11-11T13:36:52,040][DEBUG][o.e.a.s.TransportSearchAction] [node-2] [filebeat-7.3.0][0], node[KTj-KyjBSfGyJHoRyEkpPg], [R], s[STARTED], a[id=tDPNScWRSD2BChrFRDmReg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[*:filebeat-*, filebeat-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"query":{"bool":{"filter":[{"term":{"service.type":{"value":"elasticsearch","boost":1.0}}},{"range":{"@timestamp":{"from":1605069454955,"to":1605073054955,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"term":{"elasticsearch.cluster.uuid":{"value":"74K_caRxTO6yLtAd8vPVnA","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"desc"}}],"aggregations":{"types":{"terms":{"field":"event.dataset","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"levels":{"terms":{"field":"log.level","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}}}]
  2. org.elasticsearch.transport.RemoteTransportException: [node-2][10.144.104.158:9300][indices:data/read/search[phase/query]]
  3. Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [event.dataset] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
  4.     at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:759) ~[elasticsearch-7.3.0.jar:7.3.0]
  5.     at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116) ~[elasticsearch-7.3.0.jar:7.3.0]
  6.     at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:191) ~[elasticsearch-7.3.0.jar:7.3.0]
  7.     at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-7.3.0.jar:7.3.0]
  8.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:321) ~[elasticsearch-7.3.0.jar:7.3.0]
  9.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
  10.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.3.0.jar:7.3.0]
  11.     at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.3.0.jar:7.3.0]
  12.     at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:332) ~[elasticsearch-7.3.0.jar:7.3.0]
  13.     at org.elasticsearch.search.SearchService.parseSource(SearchService.java:789) ~[elasticsearch-7.3.0.jar:7.3.0]
  14.     at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-7.3.0.jar:7.3.0]
  15.     at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.3.0.jar:7.3.0]
  16.     at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.3.0.jar:7.3.0]
  17.     at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.3.0.jar:7.3.0]
  18.     at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.3.0.jar:7.3.0]
  19.     at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.0.jar:7.3.0]
  20.     at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.3.0.jar:7.3.0]
  21.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
  22.     at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.0.jar:7.3.0]
  23.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
  24.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
  25.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  26.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  27.     at java.lang.Thread.run(Thread.java:835) [?:?]
  28. [2020-11-11T13:36:52,044][DEBUG][o.e.a.s.TransportSearchAction] [node-2] All shards failed for phase: [query]
  29. org.elasticsearch.ElasticsearchException$1: Fielddata is disabled on text fields by default. Set fielddata=true on [event.dataset] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
  30.     at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639) ~[elasticsearch-7.3.0.jar:7.3.0]
  31.     at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:137) [elasticsearch-7.3.0.jar:7.3.0]
  32.     at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) [elasticsearch-7.3.0.jar:7.3.0]
  33.     at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.3.0.jar:7.3.0]
  34.     at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.3.0.jar:7.3.0]
  35.     at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.3.0.jar:7.3.0]
  36.     at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.3.0.jar:7.3.0]
  37.     at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.3.0.jar:7.3.0]
  38.     at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.3.0.jar:7.3.0]
  39.     at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) [elasticsearch-7.3.0.jar:7.3.0]
  40.     at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1223) [elasticsearch-7.3.0.jar:7.3.0]
  41.     at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197) [elasticsearch-7.3.0.jar:7.3.0]
  42.     at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.3.0.jar:7.3.0]
  43.     at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.3.0.jar:7.3.0]
  44.     at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.3.0.jar:7.3.0]
  45.     at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.3.0.jar:7.3.0]
  46.     at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.3.0.jar:7.3.0]
  47.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
  48.     at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.0.jar:7.3.0]
  49.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.0.jar:7.3.0]
  50.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.0.jar:7.3.0]
  51.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  52.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  53.     at java.lang.Thread.run(Thread.java:835) [?:?]
  54. Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [event.dataset] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
  55.     at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:759) ~[elasticsearch-7.3.0.jar:7.3.0]
  56.     at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116) ~[elasticsearch-7.3.0.jar:7.3.0]
  57.     at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:191) ~[elasticsearch-7.3.0.jar:7.3.0]
  58.     at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-7.3.0.jar:7.3.0]
  59.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:321) ~[elasticsearch-7.3.0.jar:7.3.0]
  60.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
  61.     at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.3.0.jar:7.3.0]
  62.     at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.3.0.jar:7.3.0]
  63.     at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:332) ~[elasticsearch-7.3.0.jar:7.3.0]
  64.     at org.elasticsearch.search.SearchService.parseSource(SearchService.java:789) ~[elasticsearch-7.3.0.jar:7.3.0]
  65.     at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-7.3.0.jar:7.3.0]
  66.     at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.3.0.jar:7.3.0]
  67.     at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.3.0.jar:7.3.0]
  68.     at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.3.0.jar:7.3.0]
  69.     at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.3.0.jar:7.3.0]
  70.     at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.3.0.jar:7.3.0]
  71.     ... 8 more

解决办法：
根据异常日志提示说：【Set fielddata=true on [event.dataset]】意思是说在，[event.dataset]上设置fielddata=true

修改方式如下：
1.在kibana的开发工具页面上执行

PUT filebeat-7.3.0/_mapping
{
  "properties": {
    "event.dataset": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

2.在任意一台Elasticsearch服务器控制台中执行
cURL:

curl -X PUT "IP:9200/filebeat-7.3.0/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "event.dataset": {
      "type":     "text",
      "fielddata": true
    }
  }
}
'

之后如果在次出现如下错误：
【Set fielddata=true on [log.level]】
执行如下操作解决
1.在kibana的开发工具页面上执行

PUT filebeat-7.3.0/_mapping
{
  "properties": {
    "log.level": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

cURL:
2.在任意一台Elasticsearch服务器控制台中执行

curl -X PUT "IP:9200/filebeat-7.3.0/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "log.level": {
      "type":     "text",
      "fielddata": true
    }
  }
}
'

cluster_block_exception:index [logstash-2020.07.07-000008] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

原因是磁盘空间不足85%时，默认值为85%
当Elasticsearch服务器存储达到%85就会开启只度模式无法写入数据。如下图：
image.png
解决方案：

  1. 加存储。
  2. 删除索引。
  3. 备份数据。


如果删除索引后logstash会重新创建索引此时索引文件名称为：logstash，正常的索引文件名称应该是：logstash-2020.11.12-000001，当然也可以正常使用，但是其他一些功能就无法使用了，比如索引无法根据日期来创建索引文件始终就是这一个索引文件，在索引生命周期功能中无法设置，不会生效。

解决方案：
1.在删除索引文件之前先停止logstash服务然后在删除索引文件，此时生成的索引文件名称就是根据日期来创建的了，索引的其他功能也都生效了

2.取消索引只读模式，前提是保证磁盘空间小于%85

PUT /索引名称/_settings?pretty
{
  "index.blocks.read_only_allow_delete": null
}

[ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>”http://10.144.107.22:9200/_bulk"}

logstash数据output到Elasticsearch时出现如下错误.

[ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>”http://xx.xx.xxx.xx:9200/_bulk"}

可能是索引别名重复，在删除索引时没用删除别名。

解决方案：
**

  1. 查询索引别名

    GET /xxx-*/_alias/*

  2. **删除别名

    DELETE /xxx-*/_alias/logstash