拓扑停止工作:SessionExpiredException:KeeperErrorCode = / supervisor的会话已过期

时间:2015-10-15 12:16:20

标签: apache-storm nimbus apache-zookeeper

我们正在使用storm进行处理,并在supervisior.log文件中获得了以下错误日志:

2015-10-14 11:40:05,012 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
2015-10-14 11:40:05,244 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
2015-10-14 11:40:05,309 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x15031faa157043d has expired, closing socket connection
2015-10-14 11:40:11,880 ERROR [timer] backtype.storm.daemon.supervisor - Error when processing event
java.lang.RuntimeException: org.apache.storm.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /supervisors
    at backtype.storm.util$wrap_in_runtime.invoke(util.clj:44) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$exists_node_QMARK_$fn__1826.invoke(zookeeper.clj:102) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.cluster$mk_distributed_cluster_state$reify__2073.set_ephemeral_node(cluster.clj:74) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.cluster$mk_storm_cluster_state$reify__2530.supervisor_heartbeat_BANG_(cluster.clj:358) ~[storm-core-0.9.5.jar:0.9.5]
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_22]
    at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_22]
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.5.1.jar:na]
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.5.1.jar:na]
    at backtype.storm.daemon.supervisor$fn__7444$exec_fn__1103__auto____7445$heartbeat_fn__7447.invoke(supervisor.clj:423) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$schedule_recurring$this__1807.invoke(timer.clj:99) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:50) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_22]
Caused by: org.apache.storm.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /supervisors
    at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:127) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:51) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$exists_node_QMARK_$fn__1826.invoke(zookeeper.clj:101) ~[storm-core-0.9.5.jar:0.9.5]
    ... 15 common frames omitted
2015-10-14 11:40:12,268 INFO  [CuratorFramework-0-EventThread] o.a.s.c.f.s.ConnectionStateManager - State change: SUSPENDED
2015-10-14 11:40:13,832 WARN  [CuratorFramework-0-EventThread] backtype.storm.cluster - Received event :disconnected::none: with disconnected Zookeeper.
2015-10-14 11:40:13,838 INFO  [CuratorFramework-0-EventThread] o.a.s.c.f.s.ConnectionStateManager - State change: LOST
2015-10-14 11:40:13,838 WARN  [CuratorFramework-0-EventThread] backtype.storm.cluster - Received event :expired::none: with disconnected Zookeeper.
2015-10-14 11:40:13,838 WARN  [CuratorFramework-0-EventThread] o.a.storm.curator.ConnectionState - Session expired event received
2015-10-14 11:40:13,844 INFO  [CuratorFramework-0-EventThread] org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@655daa9e
2015-10-14 11:40:14,245 ERROR [timer] backtype.storm.util - Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
    at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
    at backtype.storm.daemon.supervisor$supervisor_data$fn__7283.invoke(supervisor.clj:222) [storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:68) [storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_22]
2015-10-14 11:40:17,603 INFO  [CuratorFramework-0-EventThread] o.apache.storm.zookeeper.ClientCnxn - EventThread shut down
2015-10-14 11:40:25,719 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
2015-10-14 11:40:25,756 INFO  [Thread-3] backtype.storm.daemon.supervisor - Shutting down supervisor baae1d31-9953-4c32-b364-9f63b667217f
2015-10-14 11:40:25,781 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

Nimbus错误日志如下:

2015-10-14 11:39:43,819 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
2015-10-14 11:39:43,866 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
2015-10-14 11:39:50,568 INFO  [CuratorFramework-0-SendThread(127.0.0.1:2181)] o.apache.storm.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x15031faa1570430 has expired, closing socket connection
2015-10-14 11:39:56,568 ERROR [timer] backtype.storm.daemon.nimbus - Error when processing event
java.lang.RuntimeException: org.apache.storm.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /storms
    at backtype.storm.util$wrap_in_runtime.invoke(util.clj:44) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$get_children.invoke(zookeeper.clj:167) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.cluster$mk_distributed_cluster_state$reify__2073.get_children(cluster.clj:115) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.cluster$mk_storm_cluster_state$reify__2530.active_storms(cluster.clj:291) ~[storm-core-0.9.5.jar:0.9.5]
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_22]
    at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_22]
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.5.1.jar:na]
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298) ~[clojure-1.5.1.jar:na]
    at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:649) ~[storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na]
    at backtype.storm.daemon.nimbus$fn__3724$exec_fn__1103__auto____3725$fn__3730$fn__3731.invoke(nimbus.clj:909) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.daemon.nimbus$fn__3724$exec_fn__1103__auto____3725$fn__3730.invoke(nimbus.clj:908) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$schedule_recurring$this__1807.invoke(timer.clj:99) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:50) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_22]
Caused by: org.apache.storm.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /storms
    at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:127) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:51) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[storm-core-0.9.5.jar:0.9.5]
    at org.apache.storm.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.zookeeper$get_children.invoke(zookeeper.clj:166) ~[storm-core-0.9.5.jar:0.9.5]
    ... 16 common frames omitted
2015-10-14 11:39:56,663 ERROR [timer] backtype.storm.util - Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
    at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
    at backtype.storm.daemon.nimbus$nimbus_data$fn__3150.invoke(nimbus.clj:76) [storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:68) [storm-core-0.9.5.jar:0.9.5]
    at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42) [storm-core-0.9.5.jar:0.9.5]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_22]
2015-10-14 11:40:02,719 INFO  [Thread-3] backtype.storm.daemon.nimbus - Shutting down master

GC_Strom错误日志如下:

{Heap
 def new generation   total 78656K, used 24129K [0x000000007fff0000, 0x0000000085540000, 0x0000000095540000)
  eden space 69952K,  32% used [0x000000007fff0000, 0x000000008162f940, 0x0000000084440000)
  from space 8704K,  15% used [0x0000000084cc0000, 0x0000000084e10d60, 0x0000000085540000)
  to   space 8704K,   0% used [0x0000000084440000, 0x0000000084440000, 0x0000000084cc0000)
 tenured generation   total 174784K, used 64665K [0x0000000095540000, 0x000000009fff0000, 0x00000000bfff0000)
   the space 174784K,  36% used [0x0000000095540000, 0x00000000994666d0, 0x0000000099466800, 0x000000009fff0000)
 compacting perm gen  total 131072K, used 58359K [0x00000000bfff0000, 0x00000000c7ff0000, 0x00000000ffff0000)
   the space 131072K,  44% used [0x00000000bfff0000, 0x00000000c38edd40, 0x00000000c38ede00, 0x00000000c7ff0000)
No shared spaces configured.

由于这些错误,风暴拓扑结构尚未响应,风暴ui也无法正常工作。

任何人都可以帮忙解决此问题吗?

0 个答案:

没有答案