Logstash文件/ csv过滤器无法正常工作

时间:2016-05-11 14:06:01

标签: csv logstash

我尝试使用logstash解析csv文件以加载elasticsearch。此过滤器适用于logstash 1.4,但在升级到logstash 2.3之后没有。执行时没有给出错误,既不是在调试模式下,也不是在工作中。

过滤器如下

input {
file {
        path => "/usr/local/scripts/get_billinginfo/csv/aws-billing-detailed-line-items-with-resources-and-tags-2016-04.csv"
        start_position => beginning
        type => "billing_info"
        sincedb_path => "/dev/null"
        ignore_older => "2592000000"
    }
}



filter {
        if [type] == "billing_info" {
                csv {
                columns => [
                "InvoiceID",
                "PayerAccountId",
                "LinkedAccountId",
                "RecordType",
                "RecordId",
                "ProductName",
                "RateId",
                "SubscriptionId",
                "PricingPlanId",
                "UsageType",
                "Operation",
                "AvailabilityZone",
                "ReservedInstance",
                "ItemDescription",
                "UsageStartDate",
                "UsageEndDate",
                "UsageQuantity",
                "Rate",
                "Cost",
                "ResourceId",
                "aws_autoscaling_groupName",
                "aws_cloudformation_stack-id",
                "aws_cloudformation_stack-name",
                "user_Name",
                "user_Project"
                ]
                separator => ","
                remove_field => ["message"]
                }
                 if [InvoiceID] == "InvoiceID" {
                        drop { }
                }
                 if [RecordType] == "Rounding" {
                        drop { }
                }
                 if [RecordType] == "StatementTotal" {
                        drop { }
                }
                 if [RecordType] == "InvoiceTotal" {
                        drop { }
                }

        date {
                match => ["UsageStartDate", "yyyy-MM-dd HH:mm:ss"]
        }
        mutate {
                convert => {
                        "UsageQuantity" => "float"
                        "Rate" => "float"
                        "Cost" => "float"
                        "ResourceId" => "float"
                }
                gsub => ["ProductName", " ", "_"]
                gsub => ["UsageType", "-", "_"]
                gsub => ["UsageType", "\.", "_"]
                gsub => ["aws_autoscaling_groupName", "-", "_"]
                gsub => ["aws_cloudformation_stack-id", "-", "_"]
                gsub => ["aws_cloudformation_stack-name", "-", "_"]
                gsub => ["user_Name", "-", "_"]
                gsub => ["user_Project", "-", "_"]

        }
        }
}

output {
        if [type] == "billing_info" {
        elasticsearch {
        index => "detailed-billing-aws-%{+YYYY.MM}"
                document_type => "billing_details"
                hosts => "somehost.somedomain.net"
                }

        }
}

如果我更改为stdout输出,则不会收到任何输出。触摸文件是为了保持新鲜感,我试图在执行期间在文件末尾添加信息...任何想法都适合我。

此logstash过滤器之前在1.4 logstash安装上运行。现在我已经更新到2.3并且无法正常工作。

调试输出如下

Reading config file {:config_file=>"/usr/local/scripts/get_aws_billinginfo/load_billinginfo.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@path = ["/usr/local/scripts/get_billinginfo/csv/aws-billing-detailed-line-items-with-resources-and-tags-2016-04.csv"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@type = "billing_info" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@sincedb_path = "/dev/null" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@ignore_older = 2592000000 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::File/@close_older = 3600 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"csv", :path=>"logstash/filters/csv", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Filters::CSV/@columns = ["InvoiceID", "PayerAccountId", "LinkedAccountId", "RecordType", "RecordId", "ProductName", "RateId", "SubscriptionId", "PricingPlanId", "UsageType", "Operation", "AvailabilityZone", "ReservedInstance", "ItemDescription", "UsageStartDate", "UsageEndDate", "UsageQuantity", "Rate", "Cost", "ResourceId", "aws_autoscaling_groupName", "aws_cloudformation_stack-id", "aws_cloudformation_stack-name", "user_Name", "user_Project"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@separator = "," {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@remove_field = ["message"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@source = "message" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@quote_char = "\"" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@autogenerate_column_names = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@skip_empty_columns = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::CSV/@convert = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"drop", :path=>"logstash/filters/drop", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Filters::Drop/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@percentage = 100 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@percentage = 100 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@percentage = 100 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Drop/@percentage = 100 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"date", :path=>"logstash/filters/date", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Filters::Date/@match = ["UsageStartDate", "yyyy-MM-dd HH:mm:ss"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@target = "@timestamp" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"mutate", :path=>"logstash/filters/mutate", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Filters::Mutate/@convert = {"UsageQuantity"=>"float", "Rate"=>"float", "Cost"=>"float", "ResourceId"=>"float"} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@gsub = ["ProductName", " ", "_", "UsageType", "-", "_", "UsageType", "\\.", "_", "aws_autoscaling_groupName", "-", "_", "aws_cloudformation_stack-id", "-", "_", "aws_cloudformation_stack-name", "-", "_", "user_Name", "-", "_", "user_Project", "-", "_"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Filters::Mutate/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"elasticsearch", :path=>"logstash/outputs/elasticsearch", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
starting agent {:level=>:info, :file=>"logstash/agent.rb", :line=>"207", :method=>"execute"}
starting pipeline {:id=>"main", :level=>:info, :file=>"logstash/agent.rb", :line=>"461", :method=>"start_pipeline"}
Settings: Default pipeline workers: 4
Registering file input {:path=>["/usr/local/scripts/get_billinginfo/csv/aws-billing-detailed-line-items-with-resources-and-tags-2016-04.csv"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"171", :method=>"register"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@index = "detailed-billing-aws-%{+YYYY.MM}" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@document_type = "billing_details" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@hosts = ["172.16.230.48"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@manage_template = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@template_name = "logstash" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@template_overwrite = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@parent = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@flush_size = 500 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@idle_flush_time = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@upsert = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@max_retries = 3 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@script = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@script_type = "inline" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@script_lang = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@script_var_name = "event" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@scripted_upsert = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@retry_max_interval = 2 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@retry_max_items = 500 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@retry_on_conflict = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@action = "index" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@path = "/" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
_globbed_files: /usr/local/scripts/get_billinginfo/csv/aws-billing-detailed-line-items-with-resources-and-tags-2016-04.csv: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@sniffing = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Outputs::ElasticSearch/@sniffing_delay = 5 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Normalizing http path {:path=>"/", :normalized=>"/", :level=>:debug, :file=>"logstash/outputs/elasticsearch/http_client_builder.rb", :line=>"14", :method=>"build"}
Elasticsearch HTTP client options {:hosts=>["http://172.16.230.48:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :level=>:debug, :file=>"logstash/outputs/elasticsearch/http_client.rb", :line=>"127", :method=>"build_client"}
Using mapping template from {:path=>nil, :level=>:info, :file=>"logstash/outputs/elasticsearch/template_manager.rb", :line=>"6", :method=>"install_template"}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info, :file=>"logstash/outputs/elasticsearch/template_manager.rb", :line=>"8", :method=>"install_template"}
Found existing Elasticsearch template. Skipping template management {:name=>"logstash", :level=>:debug, :file=>"logstash/outputs/elasticsearch/http_client.rb", :line=>"30", :method=>"template_install"}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["172.16.230.48"], :level=>:info, :file=>"logstash/outputs/elasticsearch/common.rb", :line=>"19", :method=>"register"}
Will start workers for output {:worker_count=>1, :class=>LogStash::Outputs::ElasticSearch, :level=>:debug, :file=>"logstash/output_delegator.rb", :line=>"77", :method=>"register"}
Adding type with date config {:type=>nil, :field=>"UsageStartDate", :format=>"yyyy-MM-dd HH:mm:ss", :level=>:debug, :file=>"logstash/filters/date.rb", :line=>"301", :method=>"setupMatcher"}
Starting pipeline {:id=>"main", :pipeline_workers=>4, :batch_size=>125, :batch_delay=>5, :max_inflight=>500, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"188", :method=>"start_workers"}
Pipeline main started {:file=>"logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}
Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x73031cf0 @operations_mutex=#<Mutex:0x7dc4ece0>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x79fb7780>, @submit_proc=#<Proc:0x57768928@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @logger=#<Cabin::Channel:0xd0fdcca @metrics=#<Cabin::Metrics:0x2f6aed9c @metrics_lock=#<Mutex:0x5cd61994>, @metrics={}, @channel=#<Cabin::Channel:0xd0fdcca ...>>, @subscriber_lock=#<Mutex:0x3f8ae4ef>, @level=:debug, @subscribers={12448=>#<Cabin::Subscriber:0x346e844b @output=#<Cabin::Outputs::IO:0x589821eb @io=#<IO:fd 1>, @lock=#<Mutex:0x305d0b8b>>, @options={}>}, @data={}>, @last_flush=2016-05-11 16:01:01 +0200, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x78b7799e>, @buffer=[], @flush_thread=#<Thread:0x38bbe61f run>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x73031cf0 @operations_mutex=#<Mutex:0x7dc4ece0>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x79fb7780>, @submit_proc=#<Proc:0x57768928@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @logger=#<Cabin::Channel:0xd0fdcca @metrics=#<Cabin::Metrics:0x2f6aed9c @metrics_lock=#<Mutex:0x5cd61994>, @metrics={}, @channel=#<Cabin::Channel:0xd0fdcca ...>>, @subscriber_lock=#<Mutex:0x3f8ae4ef>, @level=:debug, @subscribers={12448=>#<Cabin::Subscriber:0x346e844b @output=#<Cabin::Outputs::IO:0x589821eb @io=#<IO:fd 1>, @lock=#<Mutex:0x305d0b8b>>, @options={}>}, @data={}>, @last_flush=2016-05-11 16:01:02 +0200, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x78b7799e>, @buffer=[], @flush_thread=#<Thread:0x38bbe61f run>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x73031cf0 @operations_mutex=#<Mutex:0x7dc4ece0>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x79fb7780>, @submit_proc=#<Proc:0x57768928@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @logger=#<Cabin::Channel:0xd0fdcca @metrics=#<Cabin::Metrics:0x2f6aed9c @metrics_lock=#<Mutex:0x5cd61994>, @metrics={}, @channel=#<Cabin::Channel:0xd0fdcca ...>>, @subscriber_lock=#<Mutex:0x3f8ae4ef>, @level=:debug, @subscribers={12448=>#<Cabin::Subscriber:0x346e844b @output=#<Cabin::Outputs::IO:0x589821eb @io=#<IO:fd 1>, @lock=#<Mutex:0x305d0b8b>>, @options={}>}, @data={}>, @last_flush=2016-05-11 16:01:03 +0200, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x78b7799e>, @buffer=[], @flush_thread=#<Thread:0x38bbe61f run>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x73031cf0 @operations_mutex=#<Mutex:0x7dc4ece0>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x79fb7780>, @submit_proc=#<Proc:0x57768928@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @logger=#<Cabin::Channel:0xd0fdcca @metrics=#<Cabin::Metrics:0x2f6aed9c @metrics_lock=#<Mutex:0x5cd61994>, @metrics={}, @channel=#<Cabin::Channel:0xd0fdcca ...>>, @subscriber_lock=#<Mutex:0x3f8ae4ef>, @level=:debug, @subscribers={12448=>#<Cabin::Subscriber:0x346e844b @output=#<Cabin::Outputs::IO:0x589821eb @io=#<IO:fd 1>, @lock=#<Mutex:0x305d0b8b>>, @options={}>}, @data={}>, @last_flush=2016-05-11 16:01:04 +0200, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x78b7799e>, @buffer=[], @flush_thread=#<Thread:0x38bbe61f run>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}

任何帮助将不胜感激

1 个答案:

答案 0 :(得分:0)

最后,我找到了根本原因,这有点令人尴尬......但确实如此。

我在日志文件路径和logstash上输了一个错字,而不是带来错误或警告......只有在文件有一天出现时才会永远等待。即使启用了调试模式,也不会显示该文件不存在的信息。

感谢您的耐心等待。