Question

我最近开始涉足ELK处理日志文件。我见过的大多数示例都是关于处理单行日志，但我的日志文件具有以下结构

ObjMgrSqlLog    Detail  4   00000002576a1de0:0  2016-06-22 12:15:34 SELECT statement with ID: AECCAA8
SELECT /*+ ALL_ROWS */
      T1.CONFLICT_ID,
      T1.LAST_UPD,
      T1.CREATED,
      T1.LAST_UPD_BY,
      T1.CREATED_BY,
      T1.MODIFICATION_NUM,
      T1.ROW_ID,
      T1.NAME,
      T1.BU_ID,
      :1
   FROM 
       SIEBEL.S_RESP T1
   WHERE 
      (T1.ROW_ID = :2)
   ORDER BY
      T1.NAME
ObjMgrSqlLog    Detail  4   00000002576a1de0:0  2016-06-22 12:15:34 
Bind variable 1: ,,,SADMIN,00000002576a1de0:0,,Responsibility,Account Screen Homepage View
ObjMgrSqlLog    Detail  4   00000002576a1de0:0  2016-06-22 12:15:34 Bind variable 2: 0-30
ObjMgrSqlLog    Debug   5   00000002576a1de0:0  2016-06-22 12:15:34 User search spec: "0-30"
ObjMgrSqlLog    Debug   5   00000002576a1de0:0  2016-06-22 12:15:34 User sort spec: NameObjMgrSqlLog    Debug   5   00000002576a1de0:0  2016-06-22 12:15:34 System sort spec: ObjMgrSqlLog  Debug   5   00000002576a1de0:0  2016-06-22 12:15:34 ObjMgrSqlCursorLog  Prepare 5   00000002576a1de0:0  2016-06-22 12:15:34 Begin: PrepareStmt for Sql Cursor at aeccaa8ObjMgrSqlCursorLog  Prepare 5   00000002576a1de0:0  2016-06-22 12:15:34 End: PrepareStatement for Sql Cursor at aeccaa8ObjMgrSqlLog Detail  4   00000002576a1de0:0  2016-06-22 12:15:34 
***** SQL Statement Prepare Time for SQL Cursor with ID AECCAA8: 0.000 seconds *****

这些类型的部分都散布在整个日志文件中。我需要提取实际的SQL语句以及其他细节，例如执行时间，该时间包含在此部分的最后一行中

***** SQL Statement Prepare Time for SQL Cursor with ID AECCAA8: 0.000 seconds *****

我已经能够使用以下grok模式提取他的初始值，例如关卡，语句ID，日期

input {
    file {
        path => "c:/slog/test_log.txt"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{WORD:source}\t%{WORD:type}\t%{NUMBER:level}\t%{HEXID:id}"}
        patterns_dir => ["c:/elasticsearch/pattern_dir"]
    }
    mutate {
    remove_field => ["id"]
  }
}
output {
file {
    path => "c:/slog/test_output.txt"
}
}

以下是使用的自定义模式的详细信息：

SDATE (?:(\d\d){1,4}-\d{1,2}-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\s(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))
HEXID (%{BASE16NUM}:%{NUMBER})

我意识到我需要为SQL语句使用多行插件，但我无法理解如何开始。任何可以让我入手的多线模式指针都非常有用。

更新:: 我的最终目标是关注这个区块的标记

statement_id : AECCAA8
SQL SELECT /*+ ALL_ROWS */
      T1.CONFLICT_ID,
      T1.LAST_UPD,
      T1.CREATED,
      T1.LAST_UPD_BY,
      T1.CREATED_BY,
      T1.MODIFICATION_NUM,
      T1.ROW_ID,
      T1.NAME,
      T1.BU_ID,
      :1
   FROM 
       SIEBEL.S_RESP T1
   WHERE 
      (T1.ROW_ID = :2)
   ORDER BY
      T1.NAME
Bind Variables : [",,,SADMIN,00000002576a1de0:0,,Responsibility,Account Screen Homepage View","0-30"]
sql_time: 0.000

注意 SQL的格式化并不重要，我知道这可能会转换为单行文本

grok模式从日志文件中提取多行SQL

0 个答案: