Cassandra bulkloader在数据上传期间偶尔会挂起

时间:2016-03-01 11:45:04

标签: cassandra bulkloader

我正在使用Cassandra bulkloader将数据上传到群集(用于上传数据的节点不是群集的一部分)。我观察到,在极少数情况下,bulkload过程会不确定地挂起 g数据上传。挂起是不可预测的,如果下次运行,相同的数据上传成功。

群集详细信息:20个节点Cassandra 2.1.11 EC2群集。 Bulkloader节点:2.1.2 Cassandra

以下是一个此类挂起的客户端和服务器日志

客户日志:

016-01-16 10:40:20,466 INFO [STREAM-OUT-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.hadoop.hdfs.DFSClient: Seek offset: 141643669
2016-01-16 10:40:20,525 INFO [STREAM-OUT-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.hadoop.hdfs.DFSClient: Seek offset: 152129429
2016-01-16 10:40:21,282 INFO [STREAM-IN-ip-172-31-32-55.us-west-2.compute.internal./172.31.32.55] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-55.us-west-2.compute.internal./172.31.32.55 is complete
2016-01-16 10:40:21,368 ERROR [STREAM-IN-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.cassandra.streaming.StreamSession: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Streaming error occurred
java.lang.NullPointerException
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:547)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:429)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:251)
at java.lang.Thread.run(Thread.java:744)
2016-01-16 10:40:21,369 INFO [STREAM-IN-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254 is complete
2016-01-16 10:40:21,780 INFO [STREAM-IN-ip-172-31-32-51.us-west-2.compute.internal./172.31.32.51] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-51.us-west-2.compute.internal./172.31.32.51 is complete
2016-01-16 10:40:31,841 INFO [STREAM-IN-ip-172-31-32-53.us-west-2.compute.internal./172.31.32.53] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-53.us-west-2.compute.internal./172.31.32.53 is complete
2016-01-16 10:40:34,527 INFO [STREAM-IN-ip-172-31-44-255.us-west-2.compute.internal./172.31.44.255] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-44-255.us-west-2.compute.internal./172.31.44.255 is complete

服务器日志(流失败的节点):

INFO [STREAM-INIT-/172.31.37.149:51647] 2016-01-16 10:40:21,360 StreamResultFuture.java:109 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff ID#0 Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/172.31.37.149:51647] 2016-01-16 10:40:21,360 StreamResultFuture.java:116 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff, ID#0 Received streaming plan for Bulk Load
INFO [STREAM-INIT-/172.31.37.149:51652] 2016-01-16 10:40:21,362 StreamResultFuture.java:116 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff, ID#0 Received streaming plan for Bulk Load
INFO [STREAM-IN-/172.31.37.148] 2016-01-16 10:40:21,368 StreamResultFuture.java:180 - Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with /172.31.37.148 is complete
WARN [STREAM-IN-/172.31.37.148] 2016-01-16 10:40:21,368 StreamResultFuture.java:207 - Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Stream failed <<<<<<<<<<<<< Last entry in log for the stream id
INFO [STREAM-IN-/172.31.37.149] 2016-01-16 10:40:21,407 StreamResultFuture.java:166 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff ID#0 Prepare completed. Receiving 1 files(29350007 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:230] 2016-01-16 10:40:22,159 StreamResultFuture.java:180 - Stream #507b0100-bc67-11e5-8fa7-e70ac8f0e359 Session with /172.31.37.148 is complete
INFO [StreamReceiveTask:230] 2016-01-16 10:40:22,159 StreamResultFuture.java:212 - Stream #507b0100-bc67-11e5-8fa7-e70ac8f0e359 All sessions completed
INFO [StreamReceiveTask:242] 2016-01-16 10:40:22,204 StreamResultFuture.java:180 - Stream #56ddad90-bc67-11e5-ae72-079f597e5002 Session with /172.31.37.150 is complete
INFO [StreamReceiveTask:242] 2016-01-16 10:40:22,205 StreamResultFuture.java:212 - Stream #56ddad90-bc67-11e5-ae72-079f597e5002 All sessions completed

0 个答案:

没有答案