{"action":"created","issue":{"url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/29","repository_url":"https://api.github.com/repos/PeterGrace/opentsdb-docker","labels_url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/29/comments","events_url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/29/events","html_url":"https://github.com/PeterGrace/opentsdb-docker/issues/29","id":334184899,"node_id":"MDU6SXNzdWUzMzQxODQ4OTk=","number":29,"title":"Data seems to be getting corrupted after container removal","user":{"login":"dmelo","id":89710,"node_id":"MDQ6VXNlcjg5NzEw","avatar_url":"https://avatars2.githubusercontent.com/u/89710?v=4","gravatar_id":"","url":"https://api.github.com/users/dmelo","html_url":"https://github.com/dmelo","followers_url":"https://api.github.com/users/dmelo/followers","following_url":"https://api.github.com/users/dmelo/following{/other_user}","gists_url":"https://api.github.com/users/dmelo/gists{/gist_id}","starred_url":"https://api.github.com/users/dmelo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmelo/subscriptions","organizations_url":"https://api.github.com/users/dmelo/orgs","repos_url":"https://api.github.com/users/dmelo/repos","events_url":"https://api.github.com/users/dmelo/events{/privacy}","received_events_url":"https://api.github.com/users/dmelo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-06-20T17:45:53Z","updated_at":"2018-06-28T15:50:50Z","closed_at":"2018-06-28T15:50:50Z","author_association":"CONTRIBUTOR","body":"OpenTSDB stops working after a `docker-compose down & docker-compose up`. Only starts working again after I delete all volumes, after it can recreate clean ones. I think that perhaps some data is getting corrupted after the shutdown. Is there a way to check/fix this kind of problem on container start? It is not a good thing to have to wipe all the data after the container is removed.\r\n\r\n```\r\nopentsdb                  | 2018-06-20 17:37:19,208 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:47044\r\nopentsdb                  | 2018-06-20 17:37:19,209 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:47044\r\nopentsdb                  | 2018-06-20 17:37:19,213 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x1641e4572a60019 with negotiated timeout 5000 for client /127.0.0.1:47044\r\nopentsdb                  | 2018-06-20 17:37:19,222 INFO  [ProcessThread(sid:0 cport:2181):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x1641e4572a60019\r\nopentsdb                  | 2018-06-20 17:37:19,224 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:47044 which had sessionid 0x1641e4572a60019\r\nopentsdb                  | 2018-06-20 17:37:19,227 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:47048\r\nopentsdb                  | 2018-06-20 17:37:19,227 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:47048\r\nopentsdb                  | 2018-06-20 17:37:19,230 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x1641e4572a6001a with negotiated timeout 5000 for client /127.0.0.1:47048\r\nopentsdb                  | 2018-06-20 17:37:19,245 INFO  [ProcessThread(sid:0 cport:2181):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x1641e4572a6001a\r\nopentsdb                  | 2018-06-20 17:37:19,250 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:47048 which had sessionid 0x1641e4572a6001a\r\nopentsdb                  | 2018-06-20 17:37:19,254 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:47052\r\nopentsdb                  | 2018-06-20 17:37:19,256 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:47052\r\nopentsdb                  | 2018-06-20 17:37:19,274 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x1641e4572a6001b with negotiated timeout 5000 for client /127.0.0.1:47052\r\nopentsdb                  | 2018-06-20 17:37:19,296 INFO  [ProcessThread(sid:0 cport:2181):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x1641e4572a6001b\r\nopentsdb                  | 2018-06-20 17:37:19,301 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:47052 which had sessionid 0x1641e4572a6001b\r\nopentsdb                  | 2018-06-20 17:37:19,305 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:47056\r\nopentsdb                  | 2018-06-20 17:37:19,306 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:47056\r\nopentsdb                  | 2018-06-20 17:37:19,307 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x1641e4572a6001c with negotiated timeout 5000 for client /127.0.0.1:47056\r\nopentsdb                  | 2018-06-20 17:37:19,328 INFO  [ProcessThread(sid:0 cport:2181):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x1641e4572a6001c\r\nopentsdb                  | 2018-06-20 17:37:19,329 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:47056 which had sessionid 0x1641e4572a6001c\r\nopentsdb                  | Exception in thread \"main\" java.lang.RuntimeException: Initialization failed\r\nopentsdb                  | \tat net.opentsdb.tools.TSDMain.main(TSDMain.java:237)\r\nopentsdb                  | Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:\r\nopentsdb                  | \tat com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)\r\nopentsdb                  | \tat com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)\r\nopentsdb                  | \tat com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)\r\nopentsdb                  | \tat com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1284)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005)\r\nopentsdb                  | \tat org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2558)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:2420)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:2444)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:2427)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)\r\nopentsdb                  | \tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:4394)\r\nopentsdb                  | \tat org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)\r\nopentsdb                  | \tat org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)\r\nopentsdb                  | Caused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table=\"hbase:meta\", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table=\"hbase:meta\", region_name=\"hbase:meta,,1\", stop_key=\"\"), attempt=11, timeout=-1, hasTimedout=false)\r\nopentsdb                  | \tat org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2556)\r\nopentsdb                  | \t... 9 more\r\n```\r\n\r\n```\r\n$ docker --version\r\nDocker version 18.05.0-ce, build f150324\r\n$ docker-compose --version\r\ndocker-compose version 1.21.2, build a133471\r\n```\r\n\r\nAnd here is the docker-compose.yml that concerns the OpenTSDB:\r\n\r\n```\r\n    container_name: opentsdb\r\n    image: petergrace/opentsdb-docker:latest\r\n    restart: always\r\n    networks:\r\n      - dbnet\r\n    ports:\r\n      - 4242:4242\r\n      - 60030:60030\r\n    volumes:\r\n      - type: ${OPENTSDB_FSTYPE:-volume}\r\n        source: \"${OPENTSDB_FSSOURCE}\"\r\n        target: /data/hbase\r\n      - tsdb_tmp:/tmp\r\n```\r\n\r\nI'm using a bind volume."},"comment":{"url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/comments/401082426","html_url":"https://github.com/PeterGrace/opentsdb-docker/issues/29#issuecomment-401082426","issue_url":"https://api.github.com/repos/PeterGrace/opentsdb-docker/issues/29","id":401082426,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTA4MjQyNg==","user":{"login":"PeterGrace","id":967365,"node_id":"MDQ6VXNlcjk2NzM2NQ==","avatar_url":"https://avatars2.githubusercontent.com/u/967365?v=4","gravatar_id":"","url":"https://api.github.com/users/PeterGrace","html_url":"https://github.com/PeterGrace","followers_url":"https://api.github.com/users/PeterGrace/followers","following_url":"https://api.github.com/users/PeterGrace/following{/other_user}","gists_url":"https://api.github.com/users/PeterGrace/gists{/gist_id}","starred_url":"https://api.github.com/users/PeterGrace/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PeterGrace/subscriptions","organizations_url":"https://api.github.com/users/PeterGrace/orgs","repos_url":"https://api.github.com/users/PeterGrace/repos","events_url":"https://api.github.com/users/PeterGrace/events{/privacy}","received_events_url":"https://api.github.com/users/PeterGrace/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T15:50:50Z","updated_at":"2018-06-28T15:50:50Z","author_association":"OWNER","body":"Thanks a bunch for your help, I've merged this PR."}}
{"actor":{"display_login":"PeterGrace"}}