{"action":"created","issue":{"url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/74","repository_url":"https://api.github.com/repos/openzipkin/docker-zipkin","labels_url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/74/comments","events_url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/74/events","html_url":"https://github.com/openzipkin/docker-zipkin/issues/74","id":118677615,"number":74,"title":"errors from zipkin-query on restart","user":{"login":"gadams00","id":2694081,"avatar_url":"https://avatars.githubusercontent.com/u/2694081?v=3","gravatar_id":"","url":"https://api.github.com/users/gadams00","html_url":"https://github.com/gadams00","followers_url":"https://api.github.com/users/gadams00/followers","following_url":"https://api.github.com/users/gadams00/following{/other_user}","gists_url":"https://api.github.com/users/gadams00/gists{/gist_id}","starred_url":"https://api.github.com/users/gadams00/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gadams00/subscriptions","organizations_url":"https://api.github.com/users/gadams00/orgs","repos_url":"https://api.github.com/users/gadams00/repos","events_url":"https://api.github.com/users/gadams00/events{/privacy}","received_events_url":"https://api.github.com/users/gadams00/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2015-11-24T18:47:51Z","updated_at":"2016-04-02T07:00:25Z","closed_at":null,"body":"running docker-zipkin the first time works for me, but if I stop via CTRL-C, then run docker-compose up again, there seems to be a race condition between the zipkin-query and cassandra containers. I'm running \r\ngreg@greg-elitebook ~/git/docker-zipkin $ docker-compose --version\r\ndocker-compose version: 1.5.1\r\ngreg@greg-elitebook ~/git/docker-zipkin $ docker --version\r\nDocker version 1.9.1, build a34a1d5\r\n\r\n\r\n```\r\ngreg@greg-elitebook ~/git/docker-zipkin $ docker-compose up\r\nStarting dockerzipkin_cassandra_1\r\nStarting dockerzipkin_collector_1\r\nStarting dockerzipkin_query_1\r\nStarting dockerzipkin_web_1\r\nAttaching to dockerzipkin_cassandra_1, dockerzipkin_collector_1, dockerzipkin_query_1, dockerzipkin_web_1\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I\r\ncassandra_1 | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I\r\ncollector_1 | Cassandra contact points: 172.17.0.2\r\ncollector_1 | ** Starting zipkin collector...\r\ncollector_1 | Nov 24, 2015 6:43:53 PM com.twitter.zipkin.collector.Main$ main\r\ncollector_1 | INFO: Loading configuration\r\nquery_1     | Cassandra contact points: 172.17.0.2\r\nquery_1     | ** Starting zipkin query...\r\nweb_1       | ** Starting zipkin web...\r\ncassandra_1 | INFO  18:43:53 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap\r\ncassandra_1 | INFO  18:43:53 Global memtable on-heap threshold is enabled at 979MB\r\ncassandra_1 | INFO  18:43:53 Global memtable off-heap threshold is enabled at 979MB\r\ncassandra_1 | INFO  18:43:53 Hostname: d3685e64b5b2\r\ncassandra_1 | INFO  18:43:53 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_60\r\ncassandra_1 | INFO  18:43:53 Heap size: 4106223616/4106223616\r\ncassandra_1 | INFO  18:43:53 Code Cache Non-heap memory: init = 2555904(2496K) used = 3041920(2970K) committed = 3080192(3008K) max = 251658240(245760K)\r\ncassandra_1 | INFO  18:43:53 Metaspace Non-heap memory: init = 0(0K) used = 12597824(12302K) committed = 12976128(12672K) max = -1(-1K)\r\ncassandra_1 | INFO  18:43:53 Compressed Class Space Non-heap memory: init = 0(0K) used = 1542160(1506K) committed = 1703936(1664K) max = 1073741824(1048576K)\r\ncassandra_1 | INFO  18:43:53 Par Eden Space Heap memory: init = 671088640(655360K) used = 93952808(91750K) committed = 671088640(655360K) max = 671088640(655360K)\r\ncassandra_1 | INFO  18:43:53 Par Survivor Space Heap memory: init = 83886080(81920K) used = 0(0K) committed = 83886080(81920K) max = 83886080(81920K)\r\ncassandra_1 | INFO  18:43:53 CMS Old Gen Heap memory: init = 3351248896(3272704K) used = 0(0K) committed = 3351248896(3272704K) max = 3351248896(3272704K)\r\ncassandra_1 | INFO  18:43:53 Classpath: /cassandra/bin/../conf:/cassandra/bin/../build/classes/main:/cassandra/bin/../build/classes/thrift:/cassandra/bin/../lib/ST4-4.0.8.jar:/cassandra/bin/../lib/airline-0.6.jar:/cassandra/bin/../lib/antlr-runtime-3.5.2.jar:/cassandra/bin/../lib/apache-cassandra-2.2.3.jar:/cassandra/bin/../lib/apache-cassandra-clientutil-2.2.3.jar:/cassandra/bin/../lib/apache-cassandra-thrift-2.2.3.jar:/cassandra/bin/../lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/cassandra/bin/../lib/commons-cli-1.1.jar:/cassandra/bin/../lib/commons-codec-1.2.jar:/cassandra/bin/../lib/commons-lang3-3.1.jar:/cassandra/bin/../lib/commons-math3-3.2.jar:/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:/cassandra/bin/../lib/crc32ex-0.1.1.jar:/cassandra/bin/../lib/disruptor-3.0.1.jar:/cassandra/bin/../lib/ecj-4.4.2.jar:/cassandra/bin/../lib/guava-16.0.jar:/cassandra/bin/../lib/high-scale-lib-1.0.6.jar:/cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/cassandra/bin/../lib/jamm-0.3.0.jar:/cassandra/bin/../lib/javax.inject.jar:/cassandra/bin/../lib/jbcrypt-0.3m.jar:/cassandra/bin/../lib/jcl-over-slf4j-1.7.7.jar:/cassandra/bin/../lib/jna-4.0.0.jar:/cassandra/bin/../lib/joda-time-2.4.jar:/cassandra/bin/../lib/json-simple-1.1.jar:/cassandra/bin/../lib/libthrift-0.9.2.jar:/cassandra/bin/../lib/log4j-over-slf4j-1.7.7.jar:/cassandra/bin/../lib/logback-classic-1.1.3.jar:/cassandra/bin/../lib/logback-core-1.1.3.jar:/cassandra/bin/../lib/lz4-1.3.0.jar:/cassandra/bin/../lib/metrics-core-3.1.0.jar:/cassandra/bin/../lib/metrics-logback-3.1.0.jar:/cassandra/bin/../lib/netty-all-4.0.23.Final.jar:/cassandra/bin/../lib/ohc-core-0.3.4.jar:/cassandra/bin/../lib/ohc-core-j8-0.3.4.jar:/cassandra/bin/../lib/reporter-config-base-3.0.0.jar:/cassandra/bin/../lib/reporter-config3-3.0.0.jar:/cassandra/bin/../lib/sigar-1.6.4.jar:/cassandra/bin/../lib/slf4j-api-1.7.7.jar:/cassandra/bin/../lib/snakeyaml-1.11.jar:/cassandra/bin/../lib/snappy-java-1.1.1.7.jar:/cassandra/bin/../lib/stream-2.5.2.jar:/cassandra/bin/../lib/super-csv-2.1.0.jar:/cassandra/bin/../lib/thrift-server-0.3.7.jar:/cassandra/bin/../lib/jsr223/*/*.jar:/cassandra/bin/../lib/jamm-0.3.0.jar\r\ncassandra_1 | WARN  18:43:53 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.\r\ncassandra_1 | WARN  18:43:53 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.\r\ncassandra_1 | WARN  18:43:53 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release\r\ncassandra_1 | INFO  18:43:53 Initializing SIGAR library\r\ncassandra_1 | WARN  18:43:53 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true \r\ncassandra_1 | INFO  18:43:54 Initializing system.sstable_activity\r\nweb_1       | Nov 24, 2015 6:43:54 PM com.twitter.finagle.http.HttpMuxer$$anonfun$4 apply\r\nweb_1       | INFO: HttpMuxer[/admin/metrics.json] = com.twitter.finagle.stats.MetricsExporter(<function1>)\r\nweb_1       | Nov 24, 2015 6:43:54 PM com.twitter.finagle.http.HttpMuxer$$anonfun$4 apply\r\nweb_1       | INFO: HttpMuxer[/admin/per_host_metrics.json] = com.twitter.finagle.stats.HostMetricsExporter(<function1>)\r\nweb_1       | I 1124 18:43:55.035 THREAD1: /admin => com.twitter.server.handler.SummaryHandler\r\nweb_1       | I 1124 18:43:55.036 THREAD1: /admin/server_info => com.twitter.finagle.Filter$$anon$2\r\nweb_1       | I 1124 18:43:55.036 THREAD1: /admin/contention => com.twitter.finagle.Filter$$anon$2\r\nweb_1       | I 1124 18:43:55.067 THREAD1: /admin/lint.json => com.twitter.server.handler.LintHandler\r\nweb_1       | I 1124 18:43:55.067 THREAD1: /admin/lint => com.twitter.server.handler.LintHandler\r\nweb_1       | I 1124 18:43:55.067 THREAD1: /admin/threads => com.twitter.server.handler.ThreadsHandler\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/threads.json => com.twitter.server.handler.ThreadsHandler\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/announcer => com.twitter.finagle.Filter$$anon$2\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/dtab => com.twitter.finagle.Filter$$anon$2\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/pprof/heap => com.twitter.server.handler.HeapResourceHandler\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/pprof/profile => com.twitter.server.handler.ProfileResourceHandler\r\nweb_1       | I 1124 18:43:55.068 THREAD1: /admin/pprof/contention => com.twitter.server.handler.ProfileResourceHandler\r\nweb_1       | I 1124 18:43:55.069 THREAD1: /admin/shutdown => com.twitter.server.handler.ShutdownHandler\r\nweb_1       | I 1124 18:43:55.069 THREAD1: /admin/tracing => com.twitter.server.handler.TracingHandler\r\nweb_1       | I 1124 18:43:55.070 THREAD1: /admin/events/ => com.twitter.server.handler.EventRecordingHandler\r\nweb_1       | I 1124 18:43:55.070 THREAD1: /admin/events => com.twitter.server.handler.EventsHandler\r\nweb_1       | I 1124 18:43:55.069 THREAD1: /admin/ping => com.twitter.server.handler.ReplyHandler\r\nweb_1       | I 1124 18:43:55.082 THREAD1: /admin/clients/ => com.twitter.server.handler.ClientRegistryHandler\r\nweb_1       | I 1124 18:43:55.082 THREAD1: /admin/metrics => com.twitter.server.handler.MetricQueryHandler\r\nweb_1       | I 1124 18:43:55.083 THREAD1: /admin/servers/ => com.twitter.server.handler.ServerRegistryHandler\r\nweb_1       | I 1124 18:43:55.083 THREAD1: /admin/files/ => com.twitter.server.handler.ResourceHandler\r\nweb_1       | I 1124 18:43:55.083 THREAD1: /admin/registry.json => com.twitter.server.handler.RegistryHandler\r\nweb_1       | I 1124 18:43:55.086 THREAD1: Serving admin http on 0.0.0.0/0.0.0.0:9990\r\nweb_1       | I 1124 18:43:55.070 THREAD1: /admin/logging => com.twitter.server.handler.LoggingHandler\r\nweb_1       | I 1124 18:43:55.105 THREAD1: Finagle version 6.30.0 (rev=745578b931893c432e51da623287144e548cc489) built at 20151015-163818\r\ncassandra_1 | INFO  18:43:55 Initializing key cache with capacity of 100 MBs.\r\ncassandra_1 | INFO  18:43:55 Initializing row cache with capacity of 0 MBs\r\ncassandra_1 | INFO  18:43:55 Initializing counter cache with capacity of 50 MBs\r\ncassandra_1 | INFO  18:43:55 Scheduling counter cache save to every 7200 seconds (going to save all keys).\r\ncassandra_1 | INFO  18:43:55 Initializing system.hints\r\ncassandra_1 | INFO  18:43:55 Initializing system.compaction_history\r\ncassandra_1 | INFO  18:43:55 Initializing system.peers\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_columnfamilies\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_functions\r\ncassandra_1 | INFO  18:43:55 Initializing system.IndexInfo\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_columns\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_triggers\r\ncassandra_1 | INFO  18:43:55 Initializing system.local\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_usertypes\r\ncassandra_1 | INFO  18:43:55 Initializing system.batchlog\r\ncassandra_1 | INFO  18:43:55 Initializing system.available_ranges\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_aggregates\r\nweb_1       | I 1124 18:43:55.856 THREAD1: Tracer: com.twitter.finagle.zipkin.thrift.SamplingTracer\r\ncassandra_1 | INFO  18:43:55 Initializing system.paxos\r\ncassandra_1 | INFO  18:43:55 Initializing system.peer_events\r\ncassandra_1 | INFO  18:43:55 Initializing system.size_estimates\r\ncassandra_1 | INFO  18:43:55 Initializing system.compactions_in_progress\r\ncassandra_1 | INFO  18:43:55 Initializing system.schema_keyspaces\r\ncassandra_1 | INFO  18:43:55 Initializing system.range_xfers\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.span_duration_index\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.span_names\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.service_span_name_index\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.traces\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.service_names\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.annotations_index\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.service_name_index\r\ncassandra_1 | INFO  18:43:56 Initializing zipkin.dependencies\r\ncassandra_1 | INFO  18:43:56 Initializing system_distributed.parent_repair_history\r\ncassandra_1 | INFO  18:43:56 Initializing system_distributed.repair_history\r\ncassandra_1 | INFO  18:43:57 Initializing system_auth.role_permissions\r\ncassandra_1 | INFO  18:43:57 Initializing system_auth.resource_role_permissons_index\r\ncassandra_1 | INFO  18:43:57 Initializing system_auth.roles\r\ncassandra_1 | INFO  18:43:57 Initializing system_auth.role_members\r\ncassandra_1 | INFO  18:43:57 Initializing system_traces.sessions\r\ncassandra_1 | INFO  18:43:57 Initializing system_traces.events\r\ncassandra_1 | INFO  18:43:57 Completed loading (5 ms; 18 keys) KeyCache cache\r\ncassandra_1 | INFO  18:43:57 Replaying /cassandra/bin/../data/commitlog/CommitLog-5-1448390330992.log, /cassandra/bin/../data/commitlog/CommitLog-5-1448390330993.log\r\ncassandra_1 | INFO  18:43:57 Log replay complete, 50 replayed mutations\r\ncassandra_1 | INFO  18:43:57 Cassandra version: 2.2.3\r\ncassandra_1 | INFO  18:43:57 Thrift API version: 20.1.0\r\ncassandra_1 | INFO  18:43:57 CQL supported versions: 3.3.1 (default: 3.3.1)\r\ncassandra_1 | INFO  18:43:57 Initializing index summary manager with a memory pool size of 195 MB and a resize interval of 60 minutes\r\ncassandra_1 | INFO  18:43:57 Loading persisted ring state\r\ncassandra_1 | INFO  18:43:57 Starting up server gossip\r\ncassandra_1 | INFO  18:43:58 Starting Messaging Service on port 7000\r\ncassandra_1 | INFO  18:43:58 Using saved tokens [-1005564115229926819, -1031388840942697915, -1140177759946272161, -1234894594486811292, -1319265271788665595, -138419727228993398, -1438488610580630879, -1467300421285800562, -1500979997895511158, -1527227139147930528, -1597677766682107354, -1680246583249656280, -1759569376306831297, -1886863938717898264, -1915444714871394472, -1941718827689321284, -1945549122583837521, -1988177828870265436, -2006823838377827242, -2015661610815019761, -203150784164783199, -232319634009484686, -2329178707022255157, -2468309370217404916, -2484180943248657365, -2493560870485416552, -2604969093969860409, -264165822098668675, -2652051126711331013, -2825253391556337775, -2865675486685358444, -2885509087629765661, -2901654682882230829, -3189988182670232324, -3211082813971045390, -3328821490372770264, -3450257621897176901, -3521892111449220960, -3529005200857128131, -3616917933807864311, -3662555467247311560, -3814877711255926613, -3827818588403504120, -383199181167156152, -3874969728861943427, -391778865861433965, -3953206110024963507, -3975871099936343678, -4031497698427188629, -4169539845335155755, -4230627352442276744, -4252697509408379575, -4265596633160491795, -4295850472617529562, -4314163580812533645, -4490701447068485709, -4498581258988746709, -4510802848706198818, -468296742883305191, -4743113761369541150, -4797766586967944090, -4858427494707131345, -4953439611408527234, -4964873030617709098, -499618857633495221, -5063299734091843755, -5097497474452600066, -5174608723394955949, -5199969215573461000, -5235935721563199887, -5258523133481301883, -5321338943239664697, -5327589863354562433, -5331105831697007613, -5350717043844727127, -5403110763593115140, -5477538524762841323, -5628700789104949071, -5665553611309661473, -5668378625704686822, -5684959214911440416, -5731506353477798349, -5745159541269084455, -5764151763324161623, -5860000432802954051, -5884584869910241688, -5931753402771690589, -6005515591075704542, -6056461875380387740, -6146895113397396470, -6202938791209599981, -6283291885528227274, -6294885392849881130, -6459932482383291223, -6510272536808791220, -6686070995861025835, -6733864767165695131, -6979447822132752009, -7025350512042012262, -7037711667112305861, -7192807868039939863, -7304922347754676553, -7317994537095716480, -7504470487573424824, -7540437933841039656, -7585019537315182494, -7615119993476880643, -7654366429113096509, -7665696570607827754, -7677229989226531948, -7757137231472799247, -7864750582618713223, -7879382705861331817, -7969910356790100166, -7976189026866853214, -8005733849920754341, -8010829751862478299, -8242875137666423264, -8366349421534279056, -8366417854386344521, -8372630667772120795, -848963721729225177, -8519686189665052488, -8529887340960546533, -8581706541807184844, -8611568339315275418, -8639659110278218834, -8717930400390840553, -8795964657943783472, -9023236624645665795, -9038613909643480190, 102493519130026594, 1132390368587006010, 1261773697180410996, 1295312291986507234, 1341555725832256101, 1365562907580461914, 1376129316613965386, 1417448108384101870, 1442688230117897720, 1445381690004948080, 1449754924233693365, 1521641812581443366, 1533942725734713714, 1823901245119840487, 1881850108226940416, 1942367362982992633, 1965464524368161498, 2040653532865781230, 2081070439520758729, 2136203650683774861, 2146860054623753170, 2159188444224482520, 2189769731447385573, 2227761793054104209, 2399893933007220982, 2477229567059781919, 2548411734936869688, 2566989217596906028, 2608127297146576822, 2645146418662925226, 2684019735951965985, 2769637135884842459, 2778131528209199549, 2786634025780005845, 2883133595611972675, 2887955843343853128, 2921649703543449314, 2925336738469290488, 3012361259931620620, 315432289038035741, 3283155597130038075, 32943021033311943, 3326034676794255913, 3394584700574625844, 3424784935573969205, 3484273594334599429, 3509861980493647391, 3542731586109855106, 3753429344514702774, 3758212052996951136, 3825031363453798350, 3856850180566496224, 387357659372729960, 3916668055303715801, 3998083175214435544, 4047657921614052582, 4127722185242098876, 4164726062173243155, 4191084950211457142, 4419351923481706149, 4544371175045732685, 4553470645781284939, 4621433520171367849, 4636934512444417378, 479802361490771936, 4896329276955400801, 4909262106230914570, 4915650133965933870, 4936849944512430722, 4943429037807379253, 4950220537540022914, 4955608684855501357, 5038317835294379956, 5084939880511473499, 5138801618436967754, 5178229768592584042, 5196265782716029459, 5385701088814435596, 5484206106211166207, 5641427516530796089, 5785768836431380923, 5967740725601898475, 6085256284604842815, 6280833800483301396, 6347254769779083877, 6470992405022367387, 6524679145454616930, 6558577248954428750, 67415427235322533, 6799635844425693331, 6954170631416445483, 6968498588603083333, 6992014806268275687, 7033527454586228032, 7180294399267576938, 7243901880514582658, 7276009111330435898, 7333347155394230173, 7381528148968253219, 7440998720004987812, 7608747304788481589, 7617595434615156027, 7837357184125427940, 786798651458609277, 7873364885481494891, 7897608331932477575, 7936638055583272943, 7963032253037729525, 7972887586328029777, 8036252568729900619, 8096259999764829816, 8263394550110915420, 8321684450304552867, 8341552198764308209, 839323606425787292, 8490880459061284347, 8687521445051494667, 8735449087074099568, 9049392903176569878, 9074434209677064269, 9094331475407042891, 9116717370420347831, 91206972177929461, 9164130852107420041, 963313241461612111]\r\ncassandra_1 | INFO  18:43:58 Node /172.17.0.2 state jump to normal\r\ncassandra_1 | INFO  18:43:58 Waiting for gossip to settle before accepting client requests...\r\nquery_1     | Nov 24, 2015 6:43:59 PM java.util.logging.LogManager$RootLogger log\r\nquery_1     | SEVERE: Flag zipkin.store.cassandra.keyspace read before parse.\r\nquery_1     | Nov 24, 2015 6:43:59 PM java.util.logging.LogManager$RootLogger log\r\nquery_1     | SEVERE: Flag zipkin.store.cassandra.spanTTL read before parse.\r\nquery_1     | Nov 24, 2015 6:43:59 PM java.util.logging.LogManager$RootLogger log\r\nquery_1     | SEVERE: Flag zipkin.store.cassandra.indexTTL read before parse.\r\nquery_1     | Nov 24, 2015 6:43:59 PM java.util.logging.LogManager$RootLogger log\r\nquery_1     | SEVERE: Flag zipkin.store.cassandra.maxTraceCols read before parse.\r\ncollector_1 | Nov 24, 2015 6:43:59 PM com.twitter.ostrich.admin.BackgroundProcess start\r\ncollector_1 | INFO: Starting LatchedStatsListener\r\ncollector_1 | Nov 24, 2015 6:43:59 PM com.twitter.ostrich.admin.BackgroundProcess start\r\ncollector_1 | INFO: Starting TimeSeriesCollector\r\ncollector_1 | Nov 24, 2015 6:43:59 PM com.twitter.ostrich.admin.AdminHttpService start\r\ncollector_1 | INFO: Admin HTTP interface started on port 9900.\r\ncollector_1 | Nov 24, 2015 6:43:59 PM com.twitter.zipkin.collector.builder.CollectorServiceBuilder$$anonfun$apply$1 apply\r\ncollector_1 | INFO: Building store: <function0>\r\ncollector_1 | Nov 24, 2015 6:44:00 PM java.util.logging.LogManager$RootLogger log\r\ncollector_1 | SEVERE: Flag zipkin.store.cassandra.keyspace read before parse.\r\nquery_1     | Nov 24, 2015 6:44:00 PM com.twitter.finagle.http.HttpMuxer$$anonfun$4 apply\r\nquery_1     | INFO: HttpMuxer[/stats.json] = com.twitter.finagle.stats.OstrichExporter(<function1>)\r\nquery_1     | 18:44:00.037 [main] INFO  c.t.z.builder.QueryServiceBuilder - Process started\r\ncollector_1 | Nov 24, 2015 6:44:00 PM java.util.logging.LogManager$RootLogger log\r\ncollector_1 | SEVERE: Flag zipkin.store.cassandra.spanTTL read before parse.\r\ncollector_1 | Nov 24, 2015 6:44:00 PM java.util.logging.LogManager$RootLogger log\r\ncollector_1 | SEVERE: Flag zipkin.store.cassandra.indexTTL read before parse.\r\ncollector_1 | Nov 24, 2015 6:44:00 PM java.util.logging.LogManager$RootLogger log\r\ncollector_1 | SEVERE: Flag zipkin.store.cassandra.maxTraceCols read before parse.\r\ncollector_1 | Nov 24, 2015 6:44:00 PM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp\r\ncollector_1 | INFO: Finagle version 6.30.0 (rev=745578b931893c432e51da623287144e548cc489) built at 20151015-163818\r\nquery_1     | I 1124 18:44:00.148 THREAD1: /admin => com.twitter.server.handler.SummaryHandler\r\nquery_1     | I 1124 18:44:00.148 THREAD1: /admin/server_info => com.twitter.finagle.Filter$$anon$2\r\nquery_1     | I 1124 18:44:00.151 THREAD1: /admin/contention => com.twitter.finagle.Filter$$anon$2\r\nquery_1     | I 1124 18:44:00.161 THREAD1: /admin/lint.json => com.twitter.server.handler.LintHandler\r\nquery_1     | I 1124 18:44:00.160 THREAD1: /admin/lint => com.twitter.server.handler.LintHandler\r\nquery_1     | I 1124 18:44:00.161 THREAD1: /admin/threads.json => com.twitter.server.handler.ThreadsHandler\r\nquery_1     | I 1124 18:44:00.161 THREAD1: /admin/threads => com.twitter.server.handler.ThreadsHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/pprof/heap => com.twitter.server.handler.HeapResourceHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/pprof/profile => com.twitter.server.handler.ProfileResourceHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/pprof/contention => com.twitter.server.handler.ProfileResourceHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/ping => com.twitter.server.handler.ReplyHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/shutdown => com.twitter.server.handler.ShutdownHandler\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/tracing => com.twitter.server.handler.TracingHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/events => com.twitter.server.handler.EventsHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/events/ => com.twitter.server.handler.EventRecordingHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/logging => com.twitter.server.handler.LoggingHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/metrics => com.twitter.server.handler.MetricQueryHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/clients/ => com.twitter.server.handler.ClientRegistryHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/servers/ => com.twitter.server.handler.ServerRegistryHandler\r\nquery_1     | I 1124 18:44:00.163 THREAD1: /admin/files/ => com.twitter.server.handler.ResourceHandler\r\nquery_1     | I 1124 18:44:00.164 THREAD1: /admin/registry.json => com.twitter.server.handler.RegistryHandler\r\nquery_1     | I 1124 18:44:00.192 THREAD1: Serving admin http on 0.0.0.0/0.0.0.0:9901\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/dtab => com.twitter.finagle.Filter$$anon$2\r\nquery_1     | I 1124 18:44:00.162 THREAD1: /admin/announcer => com.twitter.finagle.Filter$$anon$2\r\nquery_1     | I 1124 18:44:00.403 THREAD1: Finagle version 6.30.0 (rev=745578b931893c432e51da623287144e548cc489) built at 20151015-163818\r\nquery_1     | I 1124 18:44:00.838 THREAD1: Tracer: com.twitter.finagle.zipkin.thrift.SamplingTracer\r\nquery_1     | 18:44:01.238 [main] INFO  c.t.f.http.routing.FileResolver - Local file mode enabled\r\nquery_1     | 18:44:01.494 [main] INFO  c.t.z.builder.QueryServiceBuilder - Resolving Finagle clients before warmup\r\nquery_1     | 18:44:01.497 [main] INFO  com.twitter.finagle - zipkin-tracer resolved to Addr.Bound, current size=1\r\nquery_1     | 18:44:01.500 [main] INFO  c.t.z.builder.QueryServiceBuilder - Done resolving clients: [zipkin-tracer].\r\nquery_1     | 18:44:01.501 [main] INFO  c.t.s.internal.FinagleBuildRevision$ - Resolved Finagle build revision: (rev=745578b931893c432e51da623287144e548cc489)\r\nquery_1     | 18:44:01.753 [main] INFO  c.t.z.builder.QueryServiceBuilder - Warming up.\r\nquery_1     | 18:44:01.973 [main] INFO  c.t.finatra.http.routing.HttpRouter - Adding routes\r\nquery_1     | POST    /api/v1/spans\r\nquery_1     | GET     /api/v1/spans\r\nquery_1     | GET     /api/v1/services\r\nquery_1     | GET     /api/v1/traces\r\nquery_1     | GET     /api/v1/trace/:id\r\nquery_1     | GET     /api/v1/dependencies\r\nquery_1     | 18:44:01.988 [main] INFO  c.t.z.builder.QueryServiceBuilder - http server started on port: 9411\r\nquery_1     | 18:44:02.045 [main] ERROR  - Flag zipkin.store.cassandra.keyspace read before parse.\r\nquery_1     | 18:44:02.123 [main] INFO  com.datastax.driver.core.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.\r\nquery_1     | Exception in thread \"main\" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.17.0.2:9042 (com.datastax.driver.core.TransportException: [/172.17.0.2:9042] Cannot connect))\r\nquery_1     | \tat com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:227)\r\nquery_1     | \tat com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:86)\r\nquery_1     | \tat com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1409)\r\nquery_1     | \tat com.datastax.driver.core.Cluster.init(Cluster.java:160)\r\nquery_1     | \tat com.datastax.driver.core.Cluster.connectAsync(Cluster.java:338)\r\nquery_1     | \tat com.datastax.driver.core.Cluster.connectAsync(Cluster.java:311)\r\nquery_1     | \tat com.datastax.driver.core.Cluster.connect(Cluster.java:250)\r\nquery_1     | \tat org.twitter.zipkin.storage.cassandra.Repository$Schema.ensureExists(Repository.java:906)\r\nquery_1     | \tat org.twitter.zipkin.storage.cassandra.Repository.<init>(Repository.java:111)\r\nquery_1     | \tat com.twitter.zipkin.cassandra.CassandraSpanStoreFactory$class.com$twitter$zipkin$cassandra$CassandraSpanStoreFactory$$lazyRepository(CassandraSpanStoreFactory.scala:48)\r\nquery_1     | \tat Evaluator__f4344435a0639cf971e61d313b032bc3562c9740_1949618564$Factory$2$.com$twitter$zipkin$cassandra$CassandraSpanStoreFactory$$lazyRepository$lzycompute((inline):27)\r\nquery_1     | \tat Evaluator__f4344435a0639cf971e61d313b032bc3562c9740_1949618564$Factory$2$.com$twitter$zipkin$cassandra$CassandraSpanStoreFactory$$lazyRepository((inline):27)\r\nquery_1     | \tat com.twitter.zipkin.cassandra.CassandraSpanStoreFactory$$anon$1.repository$lzycompute(CassandraSpanStoreFactory.scala:52)\r\nquery_1     | \tat com.twitter.zipkin.cassandra.CassandraSpanStoreFactory$$anon$1.repository(CassandraSpanStoreFactory.scala:52)\r\nquery_1     | \tat com.twitter.zipkin.storage.cassandra.CassandraSpanStore$$anonfun$apply$11.apply(CassandraSpanStore.scala:235)\r\nquery_1     | \tat com.twitter.zipkin.storage.cassandra.CassandraSpanStore$$anonfun$apply$11.apply(CassandraSpanStore.scala:230)\r\nquery_1     | \tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\r\nquery_1     | \tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\r\nquery_1     | \tat scala.collection.immutable.List.foreach(List.scala:381)\r\nquery_1     | \tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\r\nquery_1     | \tat scala.collection.immutable.List.map(List.scala:285)\r\nquery_1     | \tat com.twitter.zipkin.storage.cassandra.CassandraSpanStore.apply(CassandraSpanStore.scala:230)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.SpanStoreZipkinTracer.logSpans(SpanStoreZipkinTracer.scala:49)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.RawZipkinTracer$$anonfun$3.apply(RawZipkinTracer.scala:108)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.RawZipkinTracer$$anonfun$3.apply(RawZipkinTracer.scala:108)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.DeadlineSpanMap.update(DeadlineSpanMap.scala:51)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.RawZipkinTracer.annotate(RawZipkinTracer.scala:310)\r\nquery_1     | \tat com.twitter.finagle.zipkin.thrift.RawZipkinTracer.record(RawZipkinTracer.scala:225)\r\nquery_1     | \tat com.twitter.zipkin.query.BootstrapTrace$$anonfun$complete$1.apply(BootstrapTrace.scala:26)\r\nquery_1     | \tat com.twitter.zipkin.query.BootstrapTrace$$anonfun$complete$1.apply(BootstrapTrace.scala:26)\r\nquery_1     | \tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\nquery_1     | \tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\nquery_1     | \tat com.twitter.zipkin.query.BootstrapTrace$.complete(BootstrapTrace.scala:26)\r\nquery_1     | \tat com.twitter.zipkin.builder.QueryServiceBuilder.postWarmup(QueryServiceBuilder.scala:47)\r\nquery_1     | \tat com.twitter.inject.app.App$class.main(App.scala:50)\r\nquery_1     | \tat com.twitter.zipkin.query.ZipkinQueryServer.com$twitter$inject$server$TwitterServer$$super$main(ZipkinQueryServer.scala:37)\r\nquery_1     | \tat com.twitter.inject.server.TwitterServer$class.main(TwitterServer.scala:40)\r\nquery_1     | \tat com.twitter.zipkin.query.ZipkinQueryServer.main(ZipkinQueryServer.scala:37)\r\nquery_1     | \tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\nquery_1     | \tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\nquery_1     | \tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\nquery_1     | \tat java.lang.reflect.Method.invoke(Method.java:497)\r\nquery_1     | \tat com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168)\r\nquery_1     | \tat com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)\r\nquery_1     | \tat scala.Option.foreach(Option.scala:257)\r\nquery_1     | \tat com.twitter.app.App$class.nonExitingMain(App.scala:167)\r\nquery_1     | \tat com.twitter.zipkin.query.ZipkinQueryServer.nonExitingMain(ZipkinQueryServer.scala:37)\r\nquery_1     | \tat com.twitter.zipkin.query.Main$.main(Main.scala:44)\r\nquery_1     | \tat com.twitter.zipkin.query.Main.main(Main.scala)\r\ndockerzipkin_query_1 exited with code 1\r\ncassandra_1 | INFO  18:44:06 No gossip backlog; proceeding\r\ncassandra_1 | INFO  18:44:06 Netty using native Epoll event loop\r\ncassandra_1 | INFO  18:44:06 Using Netty Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c, netty-codec=netty-codec-4.0.23.Final.208198c, netty-codec-http=netty-codec-http-4.0.23.Final.208198c, netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, netty-common=netty-common-4.0.23.Final.208198c, netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]\r\ncassandra_1 | INFO  18:44:06 Starting listening for CQL clients on /0.0.0.0:9042...\r\ncassandra_1 | INFO  18:44:06 Binding thrift service to /0.0.0.0:9160\r\ncassandra_1 | INFO  18:44:06 Listening for thrift clients...\r\n```"},"comment":{"url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/comments/204659609","html_url":"https://github.com/openzipkin/docker-zipkin/issues/74#issuecomment-204659609","issue_url":"https://api.github.com/repos/openzipkin/docker-zipkin/issues/74","id":204659609,"user":{"login":"abesto","id":59982,"avatar_url":"https://avatars.githubusercontent.com/u/59982?v=3","gravatar_id":"","url":"https://api.github.com/users/abesto","html_url":"https://github.com/abesto","followers_url":"https://api.github.com/users/abesto/followers","following_url":"https://api.github.com/users/abesto/following{/other_user}","gists_url":"https://api.github.com/users/abesto/gists{/gist_id}","starred_url":"https://api.github.com/users/abesto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abesto/subscriptions","organizations_url":"https://api.github.com/users/abesto/orgs","repos_url":"https://api.github.com/users/abesto/repos","events_url":"https://api.github.com/users/abesto/events{/privacy}","received_events_url":"https://api.github.com/users/abesto/received_events","type":"User","site_admin":false},"created_at":"2016-04-02T07:00:25Z","updated_at":"2016-04-02T07:00:25Z","body":"This can be handled from the Docker side of things. See https://github.com/docker/compose/issues/374 for an infinitely long discussion, started years ago and still going, on how. One way could be adding a `wait_for_storage` bash function to each storage profile, and calling it from `query/run.sh`. For Cassandra, it would look something like this:\r\n\r\n```bash\r\ntries=60\r\nhost=???\r\nport=???\r\necho \"Waiting at most $timeout seconds for Cassandra to start accepting CQL clients on port $port...\"\r\nwhile [ $tries -ge 0 ] && ! nc -z $host $port; do\r\n    echo \"Waiting $tries more seconds for port $host:$port to open up...\"\r\n    tries=$(expr $tries - 1)\r\n    sleep 1\r\ndone\r\n\r\nif [ $tries -eq -1 ]; then\r\n    echo \"$host$port is still not open, bailing out\"\r\n    exit 1\r\nfi\r\n```"}}