Question

我已将filebeat配置为收集结构化日志输出（绿地项目，因此每个日志条目都是预定义格式的JSON文档）并将其直接发布到ELS。

示例日志文件摘录（请注意，additional是自由格式，所有其他属性都是固定的。此帖子格式相当，但每个顶级对象都在文件的一行中）：

{
    "TimeUtc": "2016-09-23T14:13:02.217520245Z",
    "ServiceKey": "MAAS_SVC",
    "Title": "Get All Campaigns - Start",
    "Additional": {
        "HTTPRequest": {
            "Method": "GET",
            "URL": {
                "Scheme": "",
                "Opaque": "",
                "User": null,
                "Host": "",
                "Path": "/admin/campaigns",
                "RawPath": "",
                "ForceQuery": false,
                "RawQuery": "",
                "Fragment": ""
            },
            "Proto": "HTTP/1.1",
            "ProtoMajor": 1,
            "ProtoMinor": 1,
            "Header": {
                "Accept": ["*/*"],
                "Accept-Encoding": ["gzip, deflate"],
                "Connection": ["keep-alive"],
                "Requestkey": ["78478050-47f0-4d0d-44e8-615d0599574a"],
                "User-Agent": ["python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"]
            },
            "Body": {
                "Closer": {
                    "Reader": null
                }
            },
            "ContentLength": 0,
            "TransferEncoding": null,
            "Close": false,
            "Host": "xxxxxxxxx",
            "Form": null,
            "PostForm": null,
            "MultipartForm": null,
            "Trailer": null,
            "RemoteAddr": "xxx.xxx.xxx.xxx",
            "RequestURI": "/admin/campaigns",
            "TLS": null,
            "Cancel": ,
            "Response": null
        }
    },
    "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
    "HostAddress": "xxxxxxxxx"
}

这导致filebeat向ELS发出以下请求：

{
    "@timestamp": "2016-10-12T13:53:21.597Z",
    "beat": {
        "hostname": "7bca0e28e69e",
        "name": "7bca0e28e69e"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "message": "{\"TimeUtc\":\"2016-09-23T14:13:02.217520245Z\",\"ServiceKey\":\"MAAS_SVC\",\"Title\":\"Get All Campaigns - Start\",\"Additional\":{\"HTTPRequest\":{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/admin/campaigns\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Connection\":[\"keep-alive\"],\"Requestkey\":[\"78478050-47f0-4d0d-44e8-615d0599574a\"],\"User-Agent\":[\"python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic\"]},\"Body\":{\"Closer\":{\"Reader\":null}},\"ContentLength\":0,\"TransferEncoding\":null,\"Close\":false,\"Host\":\"bistromath.marathon.mesos:40072\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.1.70:42854\",\"RequestURI\":\"/admin/campaigns\",\"TLS\":null,\"Cancel\":,\"Response\":null}},\"RequestKey\":\"78478050-47f0-4d0d-44e8-615d0599574a\",\"HostAddress\":\"ba47316c9c45\"}",
    "offset": 0,
    "source": "/filebeat/log-harvest/maas-service-single.log",
    "type": "log"
}

我可以阻止filebeat转义我的日志JSON，使其成为嵌套对象而不是字符串，还是需要修补filebeat？

Answer 1

可以在Filebeat 5.x中解析JSON消息，但不能在Filebeat 1.x中解析。可以在配置文件中指定json选项。

如果您只能使用Filebeat 1.x，那么您需要Logstash来解析message字段中的JSON数据。您将配置Filebeat - ＆gt; Logstash - ＆gt; Elasticsearch。

Filebeat 5.x配置：

filebeat:
  prospectors:
    - paths:
        - input.json
      json.message_key: Title
      json.keys_under_root: true
      json.add_error_key: true

output:
  console:
    pretty: true

示例输出：

{
  "@timestamp": "2016-10-12T22:40:16.338Z",
  "Additional": {
    "HTTPRequest": {
      "Body": {
        "Closer": {}
      },
      "Close": false,
      "ContentLength": 0,
      "Header": {
        "Accept": [
          "*/*"
        ],
        "Accept-Encoding": [
          "gzip, deflate"
        ],
        "Connection": [
          "keep-alive"
        ],
        "Requestkey": [
          "78478050-47f0-4d0d-44e8-615d0599574a"
        ],
        "User-Agent": [
          "python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"
        ]
      },
      "Host": "xxxxxxxxx",
      "Method": "GET",
      "Proto": "HTTP/1.1",
      "ProtoMajor": 1,
      "ProtoMinor": 1,
      "RemoteAddr": "xxx.xxx.xxx.xxx",
      "RequestURI": "/admin/campaigns",
      "URL": {
        "ForceQuery": false,
        "Fragment": "",
        "Host": "",
        "Opaque": "",
        "Path": "/admin/campaigns",
        "RawPath": "",
        "RawQuery": "",
        "Scheme": ""
      }
    }
  },
  "HostAddress": "xxxxxxxxx",
  "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
  "ServiceKey": "MAAS_SVC",
  "TimeUtc": "2016-09-23T14:13:02.217520245Z",
  "Title": "Get All Campaigns - Start",
  "beat": {
    "hostname": "host",
    "name": "host"
  },
  "input_type": "log",
  "offset": 919,
  "source": "input.json",
  "type": "log"
}

注意：您发布的JSON数据无效。 Cancel字段缺少值。我在通过Filebeat运行数据之前将其设置为null。

Answer 2

看起来Kibana 7.2 (June 2019)现在已经有了RBAC，并且 feature control

是否想从左侧导航中隐藏开发工具？仅向管理员显示堆栈监视？还是只允许某些用户访问仪表板和画布？功能控件可让您在Kibana UI中隐藏和限制应用程序和功能。

您可以根据用户的需求以及在安全性下根据用户的权限配置Kibana应用程序和功能。

这意味着不同的角色可以在同一空间访问不同的功能。超级用户可能具有创建和编辑可视化效果和仪表板的特权，而分析人员或管理人员可能具有具有只读特权的仪表板和画布。

使用filebeat将结构化日志数据直接推送到弹性搜索

2 个答案: