Question

我试图在Logstash（版本2.1.3）

的帮助下处理Nginx访问日志

根据Nginx访问日志中的不同端点，我想在不同的队列中或有时在不同的RabbitMQ服务器中发送数据。

这是我的Logstash配置：

input {
  stdin {}
}
filter {

grok {
   match => { "message" => "(?<status>.*?)!~~!(?<req_tm>.*?)!~~!(?<time>.*?)!~~!(?<req_method>.*?)!~~!(?<req_uri>.*)" }
   tag_on_failure => ["first_grok_failed"]
}
if "/endpoint1" in [req_uri] {
      mutate { add_field => { "[queue]" => "endpoint_one" } }
      mutate { add_field => { "[rmqshost]" => "10.10.10.1" } }
}
else if "/endpoint2" in [req_uri] {
    mutate { add_field => { "[queue]" => "endpoint_two" } }
    mutate { add_field => { "[rmqshost]" => "10.10.10.2" } }
}
else {
    mutate { add_field => { "[queue]" => "endpoint_other" } }
    mutate { add_field => { "[rmqshost]" => "10.10.10.3" } }
}
}
output {
   rabbitmq {
            exchange => "%{[queue]}_exchange"
            exchange_type => "direct"
            host => "%{[rmqshost]}"
            key => "%{[queue]}_key"
            password => "mypassword"
            user=>"myuser"
            vhost=>"myvhost"
            durable=>false

    }

   stdout { 
    codec  => rubydebug 
   }
}

在上面配置的过滤器部分中，我添加了动态变量＆＃34; queue＆＃34;和＆＃34; rmqshost＆＃34;。
在输出部分，我尝试在rabbitmq插件中使用那些变量块。

我收到以下错误，显示＆＃34; rmqshost＆＃34;变量没有被取代。

Connection to %{[rmqshost]}:5672 refused: host unknown 
{:exception=>"MarchHare::ConnectionRefused", :backtrace=>
["/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-
java/lib/march_hare/session.rb:473:in `converting_rjc_exceptions_to_ruby'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:500:in `new_connection_impl'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:136:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:109:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare.rb:20:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-rabbitmq_connection-2.3.0-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:137:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-rabbitmq_connection-2.3.0-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:94:in `connect!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-rabbitmq-3.0.7-java/lib/logstash/outputs/rabbitmq.rb:40:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:192:in `start_outputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:102:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/agent.rb:165:in `execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:90:in `run'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:95:in `run'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=>:error}

我正在运行logstash，如下所示：

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-filter.conf

以下示例数据：

200!~~!0.004!~~!14/Apr/2017:05:15:27 +0000!~~!GET!~~!/endpoint1?key1=val1
200!~~!0.004!~~!14/Apr/2017:05:17:25 +0000!~~!GET!~~!/endpoint2?key1=val2

Answer 1

由于我今天偶然发现了这个问题，我也对此进行了一些挖掘。 sprintf替换仅适用于rabbitmq输出插件中的关键字段。

@hare_info.exchange.publish(message, :routing_key => event.sprintf(@key), :properties => symbolize(@message_properties.merge(:persistent => @persistent)))

实际连接是使用https://github.com/logstash-plugins/logstash-mixin-rabbitmq_connection建立的，它不提供通过变量替换主机的选项：

:hosts => @host

因此目前不支持此功能，只有关键字段可能会被logstash变量替换。

Logstash动态变量不能与rabbitmq插件一起使用

1 个答案: