Question

我正在尝试使用 ELK Stack 分析日志首先，我将日志转换为JSON字符串，并将生成的Json传递给Elasticsearch服务器。 Kibana从弹性搜索服务器中挑选json字符串。

当我在Kibana中单击“可视化”按钮时，我在弹性搜索服务器控制台中收到以下错误

错误：

[2016-09-16 11:48:34,073][DEBUG][action.search            ] [Knickknack] All shards failed for phase: [query]
RemoteTransportException[[Knickknack][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"size":0,"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"gte":1474005813952,"lte":1474006713952,"format":"epoch_millis"}}}],"must_not":[]}}}},"aggs":{"3":{"terms":{"field":"contactId","size":20,"order":{"_count":"desc"}}}}}]]; nested: IllegalStateException[
Field data loading is forbidden on [contactId]];
Caused by: SearchParseException[failed to parse search source [{"size":0,"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"gte":1474005813952,"lte":1474006713952,"format":"epoch_millis"}}}],"must_not":[]}}}},"aggs":{"3":{"terms":{"field":"contactId","size":20,"order":{"_count":"desc"}}}}}]]; nested: IllegalStateException[Field data loading is forbidden on [contactId]];
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:654)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on [contactId]
        at org.elasticsearch.index.fielddata.IndexFieldDataService$1.build(IndexFieldDataService.java:74)
        at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:275)
        at org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:209)
        at org.elasticsearch.search.aggregations.bucket.terms.TermsParser.parse(TermsParser.java:76)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:198)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:103)
        at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:838)
        ... 12 more

Logstash配置文件：

input { file {

path => ["C:/Users/Desktop/ELK_LOGS/CustomLogs.log"]
start_position => beginning
ignore_older => 0

}
}

filter {
  grok { match => { "message" => "%{SYSLOG5424SD} %{WORD:thread}: %{QS} %{WORD:queue} %{WORD:queue}: %{QS}] %{TIMESTAMP_ISO8601:logTime} %{NAGIOSTIME:contactId} %{SYSLOG5424SD:logType} %{JAVACLASS} %{GREEDYDATA:audit_message}"} } 
 kv {   
  source => "audit_message"
  field_split => ", "
  target => "journeyLevelData"
  remove_field => [ "audit_message","message","host","thread","nagios_epoch" ]
  }
}

output {

   elasticsearch { hosts => ["localhost:9200"] 
                    manage_template => true
                    template_overwrite => true
                    template => "C:/Installed_Softwares/ELASTIC_SEARCH/elasticsearch-2.3.4/config/eserve_type.json"
                    template_name => "eserve_type"
                 }
}

模板：eserve_type.json

{
    "template": "eserve_type",
    "settings": {
         "number_of_shards" : 1
    },

  "mappings" : {
  "rendition": {
            "_timestamp": {
                "enabled": true,
                "store" : true
            },
    "properties" : {
      "logTime" : {"type" : "date", "store" : true, "analyzer": "keyword" },   
      "contactId" : {"type" : "string", "store" : true, "analyzer": "keyword" },   
      "logType" : {"type" : "string", "store" : true, "analyzer": "keyword" },    
      "JAVACLASS" : {"type" : "string", "store" : true, "analyzer": "keyword" },    
      "journeyLevelData" : {"type" : "string", "store" : true, "analyzer": "keyword" }
    }
  }
}
}

示例日志

[[ACTIVE] ExecuteThread：'6'表示队列：'weblogic.kernel.Default（self-tuning）'] 2016-08-17 01：38：32,113 [608442882] [INFO] com.bt.eserve。 controller JOURNEY_NAME = ALERTS，FUNCTION_NAME = ADD_ALERTS，DATA = VP51330570 VP51330571 VP53330571

ELK Stack - Elastic Search - Logstash - Kibana

0 个答案: