{"action":"opened","issue":{"url":"https://api.github.com/repos/apache/incubator-pulsar/issues/1456","repository_url":"https://api.github.com/repos/apache/incubator-pulsar","labels_url":"https://api.github.com/repos/apache/incubator-pulsar/issues/1456/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-pulsar/issues/1456/comments","events_url":"https://api.github.com/repos/apache/incubator-pulsar/issues/1456/events","html_url":"https://github.com/apache/incubator-pulsar/issues/1456","id":309144361,"number":1456,"title":"`bin/pulsar-admin persistent stats` doesn't work if there is no enough bookies available","user":{"login":"sijie","id":1217863,"avatar_url":"https://avatars2.githubusercontent.com/u/1217863?v=4","gravatar_id":"","url":"https://api.github.com/users/sijie","html_url":"https://github.com/sijie","followers_url":"https://api.github.com/users/sijie/followers","following_url":"https://api.github.com/users/sijie/following{/other_user}","gists_url":"https://api.github.com/users/sijie/gists{/gist_id}","starred_url":"https://api.github.com/users/sijie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sijie/subscriptions","organizations_url":"https://api.github.com/users/sijie/orgs","repos_url":"https://api.github.com/users/sijie/repos","events_url":"https://api.github.com/users/sijie/events{/privacy}","received_events_url":"https://api.github.com/users/sijie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-27T21:41:56Z","updated_at":"2018-03-27T21:41:56Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"#### Expected behavior\r\n\r\n`bin/pulsar-admin persistent stats` and `bin/pulsar-admin persistent stats-internal` should work even there is no enough bookie available. \r\n\r\n#### Actual behavior\r\n\r\n```\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: 21:34:28.120 [bookkeeper-ml-workers-OrderedScheduler-4-0] WARN  org.apache.pulsar.broker.service.ServerCnx - [/10.0.0.237:36224][persistent://benchmark/local/ns-JQGPJ5A/test-QMGnFoQ-0000-partition-82][sub-000] Failed to create consumer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:627) ~[org.apache.pulsar-pulsar-broker-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:254) ~[org.apache.pulsar-managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:247) ~[org.apache.pulsar-managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$null$1(ManagedLedgerImpl.java:344) ~[org.apache.pulsar-managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.0.0-incubating-SNAPSHOT.jar:2.0.0-incubating-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at org.apache.bookkeeper.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [org.apache.bookkeeper-bookkeeper-server-shaded-4.7.0-SNAPSHOT.jar:4.7.0-SNAPSHOT]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.21.Final.jar:4.1.21.Final]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: Caused by: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: ... 22 more\r\nMar 27 21:34:28 ip-10-0-0-95.ec2.internal pulsar[14056]: Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available\r\n```\r\n\r\n#### Steps to reproduce\r\n\r\n- bring a 3 node cluster\r\n- ingest data to fill up the cluster to make bookies readonly\r\n- run `bin/pulsar-admin persistent stats`\r\n\r\n#### System configuration\r\n**Pulsar version**: master\r\n"}}
{"actor":{"display_login":"sijie"}}