Logstash未在弹性搜索中创建索引

Question

它在控制台上没有显示任何错误。这是我正在运行的命令（在Windows 10上）-

logstash --verbose -f logstash-sample.conf

这是我的logstash-sample.conf文件-

input {
  file {
    path => "C:\Users\17739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"
    start_position => "beginning"
  }
}

filter {
   csv{
      separator => ","
      columns => ["altitude", "availableBikes", "availableDocks", "city", "id", "is_renting", "kioskType", "landMark", "lastCommunicationTime", "latitude", "location", "longitude", "postalCode", "stAddress1", "stAddress2", "stationName", "status", "statusKey", "statusValue", "testStation", "totalDocks"]
      }
}
output {
  elasticsearch { 
  hosts => ["localhost:9200"]
  index => "divvy_stations_status"
  document_type => "status"
   }
  stdout {
  codec => rubydebug
  }
}

这是Logstash控制台的输出-

C:\Users\17739\Documents\IIT\CSP586\logstash-6.6.2\bin>logstash --verbose -f logstash-sample.conf
Sending Logstash logs to C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-17T12:56:36,728][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-17T12:56:36,745][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-17T12:56:41,603][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch index=>"divvy_stations_status", id=>"f84c43181aab6f7bf9e89c0412ada5b5ead116534f6661194800152751a28e87", hosts=>[//localhost:9200], document_type=>"status", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1264be19-323c-4896-8214-929f15a74251", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-03-17T12:56:43,234][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-17T12:56:43,548][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-03-17T12:56:43,695][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-03-17T12:56:43,735][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-17T12:56:43,739][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-03-17T12:56:43,768][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-03-17T12:56:43,782][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-03-17T12:56:43,801][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-03-17T12:56:44,323][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/data/plugins/inputs/file/.sincedb_6f34c293ff88e0ad3c31e4a0f32e43d9", :path=>["C:\\Users\\17739\\Documents\\IIT\\CSP586\\tutorial\\project\\ChicagoSocialHub\\backend-build-divvy-status\\divvy_stations_status.csv"]}
[2019-03-17T12:56:44,369][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2e991954 run>"}
[2019-03-17T12:56:44,440][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-03-17T12:56:44,441][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-17T12:56:44,786][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

运行logstash --verbose -f logstash-sample.conf命令后，我期望这个新索引divvy_stations_status在弹性搜索索引中可见（也已通过kibana选中，也未显示在此处）。那是真的期望吗？这是我的 输出-

Answer 1

@sapy

当您尝试在Windows平台上运行logstash时，

您必须将文件路径更改为

C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv

即，使用正斜杠代替反斜杠。 第二个

您需要指定

sincedb_path => "NUL"

在输入插件中

它跟踪受监视的日志文件的当前位置。

在开发logstash conf文件时，这两个是好的做法。

Answer 2

解决方案是在文件路径中提供forward slash /而不是backward slash \。这很奇怪，因为在Windows中，我们始终对路径分隔符使用正斜杠。而且更怪异的是，它从未引发过诸如“找不到文件或其他内容”之类的错误

因此在logstash-sample.conf文件中，使用以下-

path => "C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv"