有没有办法将应用引擎中的日志加载到Google Cloud Platform上的BigQuery中?
我尝试使用联合查询在云存储中加载Stackdriver日志文件。但是,BigQuery无法加载Stackdriver编写的一些字段名称。
日志文件是换行符分隔的JSON,记录类似于
{
"insertId":"j594356785jpk",
"labels":{
"appengine.googleapis.com/instance_name":"aef-my-instance-20180204t220251-x59f",
"compute.googleapis.com/resource_id":"99999999999999999",
"compute.googleapis.com/resource_name":"c3453465db",
"compute.googleapis.com/zone":"us-central1-f"
},
"logName":"projects/my-project/logs/appengine.googleapis.com%2Fstdout",
"receiveTimestamp":"2018-02-08T02:59:59.972739505Z",
"resource":{
"labels":{
"module_id":"my-instance",
"project_id":"my-project",
"version_id":"20180204t220251"
},
"type":"gae_app"
},
"textPayload":"{\"json\":\"blob\"}\n",
"timestamp":"2018-02-08T02:59:54Z"
}
但BigQuery在此输入上返回错误:query: Invalid field name "compute.googleapis.com/zone". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long.
有没有办法将这种日志摄入BigQuery?
我特别感兴趣的是只提取textPayload
字段。
答案 0 :(得分:5)
您可以将 Stackdriver 日志导出到 BigQuery
在 Google Cloud Platform 控制台中, Stackdriver Logging >> 导出>> 创建导出。
创建过滤器resource.type="gae_app"
并填写右侧的接收器名称,然后选择 BigQuery 作为接收服务,然后选择要从中导出的数据集接收方目的地。 https://console.cloud.google.com/logs/exports
参考:https://cloud.google.com/logging/docs/export/configure_export_v2#dest-create