Question

在亚马逊Linux上更新yum后，Td-agent无法正常工作。

Td-agent 1.1.20-0有效，但1.1.21-0无效，s3中没有日志数据。

错误消息是

2015-01-15 06:36:40 +0900 [error]: failed to configure sub output s3: Other '' plugin already use same buffer_path: type = , buffer_path = /var/log/td-agent/buffer/s3
2015-01-15 06:36:40 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/buf_file.rb:94:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:188:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:471:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:47:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
2015-01-15 06:36:40 +0900 [error]: Cannot output messages with tag 'apache.access'

和td-agent.conf（整个配置）如下所示：

####
## Output descriptions:
##


## match tag=debug.** and dump to console
<match debug.**>
  type stdout
</match>

#httpd logs
<match apache.access>
  type forest
  subtype s3
  <template>
    aws_key_id MY_AWS_KEY
    aws_sec_key MY_AWS_SECRET_KEY
    s3_endpoint s3.amazonaws.com
    s3_bucket my_s3_bucket_name
    path httpd/logs/
    buffer_path /var/log/td-agent/buffer/s3
    time_slice_format %Y-%m-%d/${hostname}-%H
    time_slice_wait 10m
    buffer_chunk_limit 256m
  </template>
</match>

#not matched
<match app.**>
  type stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  type forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
#<source>
#  type http
#  port 8888
#</source>

## live debugging agent
<source>
  type debug_agent
  bind 127.0.0.1
  port 24230
</source>

## File input
## read apache logs continuously and tags td.apache.access
<source>
  type tail
  #format apache
  format ltsv
  time_format [%d/%b/%Y:%H:%M:%S %z]
  time_key time
  pos_file /var/log/td-agent/apache.access.pos
  path /path/to/apache_logfile.log
  tag apache.access
</source>

请告诉我如何解决

Answer 1

fluent-plugin-forest版本无关。这是时间和标签问题的数量。

Fluentd在配置阶段检查buffer_path重复。但是fluent-plugin-forest会在运行时创建插件这个奇怪的重复错误发生在运行时。

请将$ {tag}或其他占位符设置为buffer_path。

http://docs.fluentd.org/articles/buf_file

此外，fluent-plugin-forest现已受到限制。因此，我建议您在配置中设置flush_at_shutdown true。

https://github.com/tagomoris/fluent-plugin-forest/issues/15

Answer 2

不幸的是，亚马逊Linux并不正式支持td-agent。原因是Amazon Linux不断更改ABI（应用程序二进制接口）并且与CentOS / REHL不兼容。

您可以使用此脚本在Amazon Linux上创建程序包并构建程序包。＆GT; https://github.com/treasure-data/omnibus-td-agent

更新后td-agent无法正常工作

2 个答案: