Question

我有一个在grokcontructor中成功测试过的nginx patteer但是当它添加到logstash 1.5.3时，日志最终会以_grokparsefailure结束

以下是我的access.log示例：

207.46.13.34 - - [14/Aug/2015:18:33:50 -0400] "GET /tag/dnssec/ HTTP/1.1" 200 1961 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"

这是nignx模式：

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} \[%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:answer} %{NUMBER:byte} "%{URI:referrer}" %{QS:referee} %{QS:agent}

我的logstash.conf看起来像这样：

input {
        lumberjack {
                port => 5000
                type => "logs"
                ssl_certificate => "/etc/pki/tls/certs/z0z0.tk.crt"
                ssl_key => "/etc/pki/tls/private/z0z0.tk.key"
        }
}
filter {
        if [type] == "nginx-access" {
                grok {
                        match => { "message" => "${NGINXACCESS}" }
                }
                geoip {
                        source => "clientip"
                        target => "geoip"
                        database => "/etc/logstash/GeoLiteCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
                }
                mutate {
                        convert => [ "[geoip][coordinates]", "float" ]
                }
        }
}
output {
        stdout {
                codec => rubydebug
        }
        elasticsearch {
                host => "172.17.0.5"
                cluster => "clustername"
                flush_size => 2000
        }
}

Answer 1

您尝试使用模式referrer将“ - ”匹配到字段URI中。不幸的是，“ - ”不是URI模式中的有效字符，它期望类似“http：// ...”

有一些模式示例匹配字符串或连字符（如内置COMMONAPACHELOG的一部分）：

 (?:%{NUMBER:bytes}|-)

你可以调整你的模式。

Answer 2

感谢Alain的建议我已经重新创建了模式但在/ opt / logstash / pattern / nginx中没有工作，所以我把它移到了logstash.conf，它有效，看起来像这样：

if [type] == "nginx-access" {
                grok {
                        match => { 'message' => '%{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|)\" %{NUMBER:answer} (?:%{NUMBER:byte}|-) (?:\"(?:%{URI:referrer}|-))\" (?:%{QS:referree}) %{QS:agent}' }
                }
}

logstash nginx模式将结果发送到_grokparsefailure

2 个答案: