Cassandra节点发生故障

时间:2015-04-03 06:56:40

标签: nullpointerexception cassandra

昨晚,我的node.js API抛出了错误

All host(s) tried for query failed.First host tried, 127.0.0.1:Host considered as DOWN.

正如错误所示,看起来cassandra节点已关闭,所以我深入了解cassandra日志并发现它们是cassandra引发的异常,但我无法弄清楚何时可能抛出此错误。

Cassandra记录

INFO  [MemtableFlushWriter:73] 2015-04-02 19:12:58,844 Memtable.java:370 - Completed flushing; nothing needed to be retained.  Commitlog position was ReplayPosition(segmentId=1427954898588, position=3473692)
INFO  [CompactionExecutor:60] 2015-04-02 19:18:28,208 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:61] 2015-04-02 19:28:28,209 CompactionManager.java:521 - No files to compact for user defined compaction
ERROR [CompactionExecutor:62] 2015-04-02 19:38:17,782 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:62,1,main]
java.lang.NullPointerException: null
    at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:236) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1089) ~[apache-cassandra-2.1.1.jar:2.1.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_67]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_67]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
INFO  [ScheduledTasks:1] 2015-04-02 19:38:22,991 ColumnFamilyStore.java:856 - Enqueuing flush of sstable_activity: 65753 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:74] 2015-04-02 19:38:22,992 Memtable.java:324 - Writing Memtable-sstable_activity@166239084(6642 serialized bytes, 2952 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:74] 2015-04-02 19:38:23,105 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-208-Data.db (3897 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3589687)
INFO  [CompactionExecutor:63] 2015-04-02 19:38:23,106 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-205-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-207-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-208-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-206-Data.db')]
INFO  [CompactionExecutor:63] 2015-04-02 19:38:23,221 CompactionTask.java:252 - Compacted 4 sstables to [/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-209,].  15,482 bytes to 3,897 (~25% of original) in 113ms = 0.032889MB/s.  328 total partitions merged to 82.  Partition merge counts were {4:82, }
INFO  [CompactionExecutor:62] 2015-04-02 19:38:28,210 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:64] 2015-04-02 19:48:28,213 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:65] 2015-04-02 19:58:28,214 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:66] 2015-04-02 20:08:28,215 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [CompactionExecutor:67] 2015-04-02 20:18:28,216 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [BatchlogTasks:1] 2015-04-02 20:19:58,913 ColumnFamilyStore.java:856 - Enqueuing flush of batchlog: 5848 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:75] 2015-04-02 20:19:58,914 Memtable.java:324 - Writing Memtable-batchlog@1542233641(4538 serialized bytes, 10 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:75] 2015-04-02 20:19:58,915 Memtable.java:370 - Completed flushing; nothing needed to be retained.  Commitlog position was ReplayPosition(segmentId=1427954898588, position=3789073)
INFO  [CompactionExecutor:68] 2015-04-02 20:28:28,217 CompactionManager.java:521 - No files to compact for user defined compaction
INFO  [ScheduledTasks:1] 2015-04-02 20:38:22,989 ColumnFamilyStore.java:856 - Enqueuing flush of compaction_history: 1214 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:76] 2015-04-02 20:38:22,990 Memtable.java:324 - Writing Memtable-compaction_history@1912511539(240 serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
INFO  [ScheduledTasks:1] 2015-04-02 20:38:22,992 ColumnFamilyStore.java:856 - Enqueuing flush of sstable_activity: 65753 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:77] 2015-04-02 20:38:22,993 Memtable.java:324 - Writing Memtable-sstable_activity@1348219988(6642 serialized bytes, 2952 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:76] 2015-04-02 20:38:23,049 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-107-Data.db (250 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3841853)
INFO  [MemtableFlushWriter:77] 2015-04-02 20:38:23,053 Memtable.java:363 - Completed flushing /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-210-Data.db (3916 bytes) for commitlog position ReplayPosition(segmentId=1427954898588, position=3841853)
INFO  [CompactionExecutor:69] 2015-04-02 20:38:28,217 CompactionManager.java:521 - No files to compact for user defined compaction

此点之后不会生成任何日志,可能是节点出现故障,服务器端apis开始抛出错误。

1 个答案:

答案 0 :(得分:2)

升级到Cassandra 2.1.5,请参阅CASSANDRA-8067