2017-02-26 17:57:24,372 - INFO - [main-EventThread:AbstractZkLedgerManager@171] - Received watched event WatchedEvent state:SyncConnected type:NodeDeleted path:/ledgers/00/0009/L7 109 from zookeeper based ledger manager. 2017-02-26 17:59:56,093 - ERROR - [BookieJournal-3181:Journal@1015] - Journal error java.lang.ArrayIndexOutOfBoundsException: 256 at com.yahoo.sketches.quantiles.DoublesUpdateImpl.zipSize2KBuffer(DoublesUpdateImpl.java:127) at com.yahoo.sketches.quantiles.DoublesUpdateImpl.inPlacePropagateCarry(DoublesUpdateImpl.java:92) at com.yahoo.sketches.quantiles.DoublesUpdateImpl.processFullBaseBuffer(DoublesUpdateImpl.java:46) at com.yahoo.sketches.quantiles.HeapDoublesSketch.update(HeapDoublesSketch.java:176) at org.apache.bokkeeper.stats.datasketches.DataSketchesOpStatsLogger.registerSuccessfulEvent(DataSketchesOpStatsLogger.java:59) at org.apache.bookkeeper.bookie.Journal.run(Journal.java:895) 2017-02-26 17:59:56,093 - INFO - [BookieJournal-3181:Journal@1024] - Journal exited loop! 2017-02-26 17:59:56,093 - ERROR - [Bookie-3181:Bookie@870] - Journal manager quits unexpectedly. 2017-02-26 17:59:56,094 - INFO - [Bookie-3181:Bookie@883] - Triggering shutdown of Bookie-3181 with exitCode 5 2017-02-26 17:59:56,096 - WARN - [metrics-1-1:DataSketchesMetricsProvider@76] - Failed to report stats: 128 java.lang.ArrayIndexOutOfBoundsException: 128 at com.yahoo.sketches.quantiles.DoublesAuxiliary.populateFromQuantilesSketch(DoublesAuxiliary.java:99) at com.yahoo.sketches.quantiles.DoublesAuxiliary.(DoublesAuxiliary.java:38) at com.yahoo.sketches.quantiles.DoublesSketch.constructAuxiliary(DoublesSketch.java:607) at com.yahoo.sketches.quantiles.DoublesSketch.getQuantile(DoublesSketch.java:195) at org.apache.bokkeeper.stats.datasketches.DataSketchesOpStatsLogger.getMedian(DataSketchesOpStatsLogger.java:121) at org.apache.bokkeeper.stats.datasketches.JsonFileReporter.lambda$report$9(JsonFileReporter.java:67) at java.util.concurrent.ConcurrentSkipListMap.forEach(ConcurrentSkipListMap.java:3252) at org.apache.bokkeeper.stats.datasketches.JsonFileReporter.report(JsonFileReporter.java:61) at org.apache.bokkeeper.stats.datasketches.DataSketchesMetricsProvider.lambda$null$5(DataSketchesMetricsProvider.java:74) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.apache.bokkeeper.stats.datasketches.DataSketchesMetricsProvider.lambda$start$6(DataSketchesMetricsProvider.java:72) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) 2017-02-26 17:59:56,097 - INFO - [BookieShutdownTrigger:Bookie@906] - Shutting down Bookie-3181 with exitCode 5 2017-02-26 17:59:56,097 - INFO - [main:BookieServer@597] - Stop stats provider 2017-02-26 17:59:56,097 - INFO - [main:BookieServer@363] - Shutting down BookieServer 2017-02-26 17:59:56,097 - INFO - [main:BookieNettyServer@192] - Shutting down BookieNettyServer 2017-02-26 17:59:56,103 - INFO - [bookie-reg-6-1:ZooKeeper@684] - Session: 0x85a613e5bba0248 closed 2017-02-26 17:59:56,104 - INFO - [BookieShutdownTrigger:Journal@1035] - Shutting down Journal 2017-02-26 17:59:56,104 - INFO - [bookie-reg-6-1-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x85a613e5bba0248 2017-02-26 17:59:56,105 - ERROR - [ForceWriteThread:Journal$ForceWriteThread@486] - ForceWrite thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at org.apache.bookkeeper.util.UnboundArrayBlockingQueue.take(UnboundArrayBlockingQueue.java:312) at org.apache.bookkeeper.bookie.Journal$ForceWriteThread.run(Journal.java:446) 2017-02-26 17:59:56,105 - INFO - [BookKeeperClientWorker-20-1:NetworkTopology@463] - Removing a node: /default-rack/10.64.103.187:3181 2017-02-26 17:59:56,108 - INFO - [BookieShutdownTrigger:SyncThread@170] - Shutting down SyncThread 2017-02-26 17:59:56,114 - INFO - [bookkeeper-io-17-8:PerChannelBookieClient@764] - Disconnected from bookie channel [id: 0x5f341598, L:/10.64.103.187:50614 ! R:10.64.103.187/10.64 .103.187:3181] 2017-02-26 17:59:56,126 - INFO - [bookie-io-15-3:AuthHandler$ServerSideHandler@59] - Channel open [id: 0x52524268, L:/10.64.103.187:3181 - R:/10.64.103.249:49448] 2017-02-26 17:59:56,139 - INFO - [BookieShutdownTrigger:GarbageCollectorThread@504] - Shutting down GarbageCollectorThread 2017-02-26 17:59:56,139 - WARN - [BookieShutdownTrigger:GarbageCollectorThread@515] - GC executor did not shut down in 60 seconds. Killing 2017-02-26 17:59:56,139 - INFO - [BookieShutdownTrigger:EntryLogger@1206] - Stopping EntryLogger 2017-02-26 17:59:56,142 - INFO - [BookieShutdownTrigger:EntryLogger$EntryLoggerAllocator@628] - Stopped entry logger preallocator. 2017-02-26 17:59:56,295 - INFO - [BookieShutdownTrigger:LedgerDirsManager@259] - Shutting down LedgerDirsMonitor 2017-02-26 17:59:56,296 - INFO - [LedgerDirsMonitorThread:LedgerDirsManager$LedgerDirsMonitor@336] - LedgerDirsMonitor thread is interrupted 2017-02-26 17:59:56,297 - INFO - [LedgerDirsMonitorThread:LedgerDirsManager$LedgerDirsMonitor@340] - LedgerDirsMonitorThread exited! 2017-02-26 17:59:56,297 - INFO - [main:AutoRecoveryMain@136] - Shutting down auto recovery: 0 2017-02-26 17:59:56,297 - INFO - [main:AutoRecoveryMain@140] - Shutting down AutoRecovery 2017-02-26 17:59:56,298 - INFO - [main:ReplicationWorker@390] - Shutting down replication worker 2017-02-26 17:59:56,298 - INFO - [main:ReplicationWorker@398] - Shutting down ReplicationWorker 2017-02-26 17:59:56,298 - INFO - [AuditorElector-10.64.103.187:3181:AuditorElector$2@217] - Shutting down AuditorElector 2017-02-26 17:59:56,299 - ERROR - [ReplicationWorker:ReplicationWorker@158] - UnavailableException while replicating fragments org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Interrupted while connecting zookeeper at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:492) at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:181) at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:147) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:486) ... 3 more 2017-02-26 17:59:56,300 - INFO - [ReplicationWorker:ReplicationWorker@163] - ReplicationWorker exited loop!