操作系统升级后,Kafka Brokers连续自动重启

时间:2019-01-14 13:23:50

标签: apache-kafka

我设置了一个基于3节点3的Zookeeper的Kafka。 我使用的版本是0.11.xx

最近,我们在两个生产集群的基础VM上进行了操作系统升级。

升级后,其中一个群集运行良好,但是另一个群集运行,我看到Kafka代理自动重新启动。

这种自动行为每分钟发生至少15-20次。

我会怀疑它是否存在配置问题,因为在另一个群集上实现了相同的更改(配置与有问题的群集相同),并且运行良好。

CPU和内存使用情况似乎正常,尽管与其他群集相比,CPU负载略有增加,但仍处于可控范围内

为什么会发生这种情况的任何具体原因?

我粘贴了错误日志,这些消息似乎一次又一次地出现

控制器日志

[2019-01-11 22:45:03,076] INFO [Controller 1]: Newly added brokers: , deleted brokers: 2, all live brokers: 1,3 (kafka.controller.KafkaController)
[2019-01-11 22:45:03,077] INFO Connection to 10.2.14.200 closed by remote host. down (kafka.controller.RequestSendThread)
Connection to 10.2.14.200 closed.ontroller-1-to-broker-2-send-thread]: Stopped (kafka.controller.RequestSendThread)
[2019-01-11 22:45:03,077] INFO [Controller-1-to-broker-2-send-thread]: Shutdown completed (kafka.controller.RequestSendThread)
Cleaning up ssh artifacts INFO [Controller 1]: Broker failure callback for 2 (kafka.controller.KafkaController)
[2019-01-11 22:45:03,109] INFO [Controller 1]: Removed ArrayBuffer(2) from list of shutting down brokers. (kafka.controller.KafkaController)
[2019-01-13 18:00:13,537] DEBUG [Controller 2]: Broker 1 has been elected as the controller, so stopping the election process. (kafka.controller.KafkaController)
[2019-01-13 18:00:31,822] INFO [controller-event-thread]: Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-01-13 18:00:31,822] INFO [controller-event-thread]: Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-01-13 18:00:31,822] INFO [controller-event-thread]: Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-01-13 18:00:31,823] DEBUG [Controller 2]: Controller resigning, broker id 2 (kafka.controller.KafkaController)
[2019-01-13 18:00:31,823] DEBUG [Controller 2]: De-registering IsrChangeNotificationListener (kafka.controller.KafkaController)
[2019-01-13 18:00:31,824] INFO [Partition state machine on Controller 2]: Stopped partition state machine (kafka.controller.PartitionStateMachine)
[2019-01-13 18:00:31,824] INFO [Replica state machine on controller 2]: Stopped replica state machine (kafka.controller.ReplicaStateMachine)
[2019-01-13 18:00:31,825] INFO [Controller 2]: Broker 2 resigned as the controller (kafka.controller.KafkaController)
[2019-01-13 18:00:44,477] INFO [controller-event-thread]: Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2019-01-13 18:00:44,513] DEBUG [Controller 2]: Broker 1 has been elected as the controller, so stopping the election process. (kafka.controller.KafkaController)*

Server.logs

[2019-01-12 15:59:55,782] INFO [Kafka Request Handler on Broker 2], shutting down (kafka.server.KafkaRequestHandlerPool)
[2019-01-12 15:59:55,784] INFO [Kafka Request Handler on Broker 2], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2019-01-12 15:59:55,786] INFO [ThrottledRequestReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,099] INFO [ThrottledRequestReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,099] INFO [ThrottledRequestReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,099] INFO [ThrottledRequestReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,103] INFO [ThrottledRequestReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,103] INFO [ThrottledRequestReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:56,103] INFO [ThrottledRequestReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:57,103] INFO [ThrottledRequestReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:57,103] INFO [ThrottledRequestReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-01-12 15:59:57,104] INFO [KafkaApi-2] Shutdown complete. (kafka.server.KafkaApis)
[2019-01-12 15:59:57,105] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2019-01-12 15:59:57,107] INFO EventThread shut down for session: 0x26832f429b62e06 (org.apache.zookeeper.ClientCnxn)

0 个答案:

没有答案