Question

我正在使用Kafka Server 0.9和消费者kafka-client版本0.9以及kafka-producer 0.8.2。

每件事情都很有效，除非我得到很多关于协调员死于消费者的信息

2016-02-25 19:30:45.046  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.048  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.049  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.050  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.051  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.052  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.053  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.054  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.055  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.056  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.057  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.058  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.059  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.060  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.061  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.063  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.064  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.065  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.066  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.067  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.069  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.070  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.071  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.073  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.074  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.076  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.077  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.078  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.080  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.081  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.082  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.084  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.085  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.087  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.088  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.090  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.091  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.093  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094  INFO 10263 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.

我还注意到制作人每隔10分钟就会断开连接，如下所示

2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Fetching metadata from broker id:0,host:192.168.72.30,port:9092 with correlation id 41675 for 1 topic(s) Set(act)
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to 192.168.72.30:9092 for producing
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from 192.168.72.30:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Disconnecting from kafkauk.XXXXXXXXXX.co:9092
2016-03-12 15:55:36 INFO  [pool-1-thread-1] - Connected to kafkauk.XXXXXXXXXX.co:9092 for producing

这是我的制作人配置

metadata.broker.list=192.168.72.30:9092
serializer.class=kafka.serializer.StringEncoder
request.required.acks=1
linger.ms=2000
batch.size=500

和消费者配置

bootstrap.servers: kafkauk.xxxxxxxx.co:9092
group.id: cdrServer
client.id: cdrServer
enable.auto.commit: true
auto.commit.interval.ms: 1000
session.timeout.ms: 30000
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.StringDeserializer

我无法弄清楚这些是什么意思，我应该忽略它们，或者我在配置中遗漏了什么

我将kafka更改为消费者的调试级别后，我发现了以下

2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Node 2147483647 disconnected.
2016-03-13 18:21:55.586  INFO 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Marking the coordinator 2147483647 dead.
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Issuing group metadata request to broker 0
2016-03-13 18:21:55.586 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.NetworkClient   : Sending metadata request ClientRequest(expectResponse=true
, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=183025,client_id=cdrServer}, body={topics=[act]}), isInitiatedByNetworkCli
ent, createdTimeMs=1457893315586, sendTimeMs=0) to node 0
2016-03-13 18:21:55.591 DEBUG 5469 --- [      cdrServer] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 296 to Cluster(nodes = [N
ode(0, kafkauk.xxxxxxxxx.co, 9092)], partitions = [Partition(topic = act, partition = 0, leader = 0, replicas = [0,], isr = [0,]])
2016-03-13 18:21:55.592 DEBUG 5469 --- [      cdrServer] o.a.k.c.c.internals.AbstractCoordinator  : Group metadata response ClientResponse(receivedTimeMs=1457
893315592, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFu
tureCompletionHandler@1e2de777, request=RequestSend(header={api_key=10,api_version=0,correlation_id=183024,client_id=cdrServer}, body={group_id=cdrServer}), c
reatedTimeMs=1457893315586, sendTimeMs=1457893315586), responseBody={error_code=0,coordinator={node_id=0,host=kafkauk.xxxxxxxx.co,port=9092}})

我不确定这是一个网络问题，因为它确实每9分钟发生一次

更新

我发现这与

直接相关

connections.max.idle.ms: 300000

我放了什么然后我会断开此值

Answer 1

当消费者客户端和协调器之间存在网络通信错误时，标记协调器死机（当协调器死亡且组需要重新平衡时也会发生这种情况）。有多种情况（偏移提交请求，获取偏移等）可能导致此问题。我建议你研究造成这种情况的原因

Answer 2

我遇到了同样的问题。最后，在遵循Shannon关于TRACING日志的建议之后，我使用了：

logging.level.org.apache.kafka=TRACE

要了解我的客户正在尝试解析Euler:9092作为协调员......本地名称!! 因此，我在listeners文件中注释掉并更改了advertised.listeners和server.properties值。它现在正在运作！： - ）

Answer 3

在我的情况下，当我尝试手动分配分区时，消息在日志中。在我阅读了新消费者的api文档后，请注意：

消费者也可以使用assign（Collection）手动分配特定分区（类似于旧的＆＃34;简单＆＃34;消费者）。在这种情况下，将禁用动态分区分配和使用者组协调。

也就是说，如果你有这样的代码：

    KafkaConsumer<String, String> consumer = new KafkaConsumer(props);
    consumer.assign( Arrays.asList(
            new TopicPartition("topic", 0),
            new TopicPartition("topic", 1)
    ));

然后消息＆＃34;标记协调器2147483647死了＆＃34;总是放入我们的日志。

Answer 4

这基本上是你无法接触卡夫卡在我的情况下，我在流浪盒中运行Kafka，如果我启动VPN，它刷新了流浪的ip，因此它无法连接到它。
可能的解决方案：在这种情况下，请停止VPN并启动您的流浪者。

Answer 5

这也可能与漫长的垃圾收集停止世界阶段有关。就我而言，我在＆gt;之后遇到了这条消息。 10秒GC。

Answer 6

我今天遇到了这个问题并解决了它（暂时，我可以补充一下）。我已经就我是如何做到的，发布了一个答案here。

Kafka Consumer标记协调员2147483647死亡

6 个答案: