filebeat到kafka：无法连接到代理

时间：2020-06-10 02:23:17

标签： apache-kafka apache-zookeeper hadoop2 filebeat

我在apache环境中是新手，目前正在尝试将日志数据从filebeat生产者发送到kafka代理。

环境：

  1. kafka 2.11（通过ambari安装）
  2. filebeat 7.4.2（Windows）

我试图将日志从filebeat发送到ambari，我启动了kafka服务器并创建了名为“ test”的主题，该主题列在--list中。我对卡夫卡经纪人的港口非常困惑。在一些教程中，我看到他们使用的是9092而不是2181。那么，现在，我应该使用哪个端口从filebeat发送日志？

这是我的filebeat.conf 

filebeat.inputs:
    - type: log
      paths:
       - C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
  hosts: ["x.x.x.x:9092"]
  topic: "test"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

结果

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)

2020-06-10T09:00:32.466+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)

2020-06-10T09:00:34.729+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)

2020-06-10T09:00:36.989+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.

2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092

这让我想知道我是否真的有9092端口。因此，我检查了server.properties。一些我最关心的问题：

port=6667
listeners=PLAINTEXT://x.x.x.x:6667

所以然后我再次尝试执行filebeat.conf并将端口9092更改为6667，这就是结果

2020-06-10T09:18:01.448+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667

2020-06-10T09:18:01.450+0700    INFO    kafka/log.go:53 producer/broker/1001 starting up

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [open] on test/0

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/leader/test/0 selected broker 1001

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.452+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:264     finished kafka batch
2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:278     Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [normal]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 abandoning broker 1001

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/broker/1001 shut down

问题

发生了什么事？应该使用哪个端口？每个端口有什么用？

任何回复都会非常感激。谢谢

更新

根据此source的正确来源是6667，因为kafka是通过ambari安装的。

1 个答案:

答案 0 :(得分：1)

可使用的端口没有限制，仅取决于可用性。 如您所说，在第一种情况下，代理可能已经在6667上启动，因此没有进程在9092上运行。

2020-06-10T09：18：01.451 + 0700 INFO kafka / log.go：53无法执行 连接到代理x.x.x.x：6667：拨打tcp：查找x.x.x.x：没有这样的主机

接下来，当您提及advertised.listeners属性时，应确保在advertised.listeners中提及的IP是分配给该计算机的IP。您不能分配1.1.1.1:9092（仅举一些例子）。

执行ifconfig（Linux），ipconfig（Windows），并在可从应用程序计算机访问的网络接口上查看计算机的IP。 在linux中，主要是为 eth0

必须从运行应用程序的计算机上访问此IP。 因此，运行您的应用程序的计算机应该能够解析该IP。您可能还需要检查Kafka代理与运行应用程序的计算机之间的网络连接。

相关问题
最新问题