{"action":"created","issue":{"url":"https://api.github.com/repos/mesos/storm/issues/155","repository_url":"https://api.github.com/repos/mesos/storm","labels_url":"https://api.github.com/repos/mesos/storm/issues/155/labels{/name}","comments_url":"https://api.github.com/repos/mesos/storm/issues/155/comments","events_url":"https://api.github.com/repos/mesos/storm/issues/155/events","html_url":"https://github.com/mesos/storm/issues/155","id":164467819,"number":155,"title":"Storm executors not starting with mesos+docker","user":{"login":"Spritekin","id":13467740,"avatar_url":"https://avatars.githubusercontent.com/u/13467740?v=3","gravatar_id":"","url":"https://api.github.com/users/Spritekin","html_url":"https://github.com/Spritekin","followers_url":"https://api.github.com/users/Spritekin/followers","following_url":"https://api.github.com/users/Spritekin/following{/other_user}","gists_url":"https://api.github.com/users/Spritekin/gists{/gist_id}","starred_url":"https://api.github.com/users/Spritekin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Spritekin/subscriptions","organizations_url":"https://api.github.com/users/Spritekin/orgs","repos_url":"https://api.github.com/users/Spritekin/repos","events_url":"https://api.github.com/users/Spritekin/events{/privacy}","received_events_url":"https://api.github.com/users/Spritekin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mesos/storm/labels/docker","name":"docker","color":"22B8EB"},{"url":"https://api.github.com/repos/mesos/storm/labels/help%20wanted","name":"help wanted","color":"159818"},{"url":"https://api.github.com/repos/mesos/storm/labels/question","name":"question","color":"cc317c"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-07-08T06:33:23Z","updated_at":"2016-07-13T03:19:19Z","closed_at":"2016-07-12T01:23:50Z","body":"Hi, \r\nI'm running Mesos 0.28.0 on CoreOS 1010.5 so everything in my environment is dockerized.\r\nI installed Storm using Marathon. I first used mesos/storm:0.1.4-0.10.1-0.28.2-jdk8 (yes I know different mesos versions but please please keep reading to the end). The Service and UI started fine. This is my Marathon json .\r\n```\r\n{\r\n  \"id\": \"storm\",\r\n  \"cpus\": 1,\r\n  \"mem\": 2048,\r\n  \"instances\": 1,\r\n  \"acceptedResourceRoles\": [ \"*\" ],\r\n  \"ports\": [7777,7778],\r\n  \"requirePorts\": true,\r\n  \"cmd\": \"/opt/storm/bin/run-with-marathon.sh\",\r\n  \"container\": {\r\n    \"type\": \"DOCKER\",\r\n    \"docker\": {\r\n      \"image\": \"mesos/storm:0.1.4-0.10.1-0.28.2-jdk8\",\r\n      \"network\": \"HOST\"\r\n    }\r\n  },\r\n  \"env\": {\r\n      \"MESOS_MASTER_ZK\": \"mesos-zk-dev.aws.cmcrc.com\",\r\n      \"STORM_NIMBUS_OPTS\": \"-c mesos.container.docker.image=mesos/storm:0.1.4-0.10.1-0.28.2-jdk8 -c topology.mesos.worker.mem.mb=2048 -c topology.mesos.executor.mem.mb=2048\"     \r\n  }\r\n}\r\n```\r\n\r\nThen I submit the word-count sample. All goes fine:\r\n```\r\n523  [main] INFO  b.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /opt/storm/storm-local/nimbus/inbox/stormjar-fc97bafa-2476-46c7-b3c3-d4554fc53de0.jar\r\n523  [main] INFO  b.s.StormSubmitter - Submitting topology word-count in distributed mode with conf {\"storm.zookeeper.topology.auth.scheme\":\"digest\",\"storm.zookeeper.topology.auth.payload\":\"-8550566346632356358:-5001853306070624651\",\"nimbus.host\":\"172.31.42.110\",\"topology.workers\":3,\"nimbus.thrift.port\":7778,\"topology.debug\":true}\r\n629  [main] INFO  b.s.StormSubmitter - Finished submitting topology: word-count\r\n```\r\n\r\n\r\nThe problem is that Nimbus tries to start the workers, then they fail to start. This is the supervisor log:\r\n```\r\n2016-07-08 06:07:11.756 b.s.u.Utils [INFO] Using defaults.yaml from resources\r\n2016-07-08 06:07:11.808 b.s.u.Utils [INFO] Using storm.yaml from resources\r\n2016-07-08 06:07:12.615 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT\r\n2016-07-08 06:07:12.615 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=localhost\r\n2016-07-08 06:07:12.615 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_91\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/storm-mesos-0.1.5-SNAPSHOT-storm0.10.1-mesos0.28.2.jar:/opt/storm/lib/slf4j-api-1.7.7.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/log4j-slf4j-impl-2.1.jar:/opt/storm/lib/log4j-core-2.1.jar:/opt/storm/lib/log4j-api-2.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/clojure-1.6.0.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/hadoop-auth-2.4.0.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/storm-core-0.10.1.jar:/opt/storm/conf\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=4.5.0-coreos-r1\r\n2016-07-08 06:07:12.616 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=root\r\n2016-07-08 06:07:12.617 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/root\r\n2016-07-08 06:07:12.617 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/opt/storm\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=localhost\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.8.0_91\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre\r\n2016-07-08 06:07:12.623 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/opt/storm/lib/reflectasm-1.07-shaded.jar:/opt/storm/lib/asm-4.0.jar:/opt/storm/lib/storm-mesos-0.1.5-SNAPSHOT-storm0.10.1-mesos0.28.2.jar:/opt/storm/lib/slf4j-api-1.7.7.jar:/opt/storm/lib/kryo-2.21.jar:/opt/storm/lib/log4j-slf4j-impl-2.1.jar:/opt/storm/lib/log4j-core-2.1.jar:/opt/storm/lib/log4j-api-2.1.jar:/opt/storm/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/lib/minlog-1.2.jar:/opt/storm/lib/clojure-1.6.0.jar:/opt/storm/lib/servlet-api-2.5.jar:/opt/storm/lib/hadoop-auth-2.4.0.jar:/opt/storm/lib/disruptor-2.10.4.jar:/opt/storm/lib/storm-core-0.10.1.jar:/opt/storm/conf\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=4.5.0-coreos-r1\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=root\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/root\r\n2016-07-08 06:07:12.624 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/opt/storm\r\n2016-07-08 06:07:12.789 b.s.u.Utils [INFO] Using defaults.yaml from resources\r\n2016-07-08 06:07:12.795 b.s.u.Utils [INFO] Using storm.yaml from resources\r\n2016-07-08 06:07:12.822 b.s.d.supervisor [INFO] Starting Supervisor with conf {\"topology.builtin.metrics.bucket.size.secs\" 60, \"nimbus.childopts\" \"-Xmx1024m\", \"ui.filter.params\" nil, \"storm.cluster.mode\" \"distributed\", \"topology.mesos.worker.cpu\" 1.0, \"mesos.framework.role\" \"*\", \"storm.messaging.netty.client_worker_threads\" 1, \"supervisor.run.worker.as.user\" false, \"topology.max.task.parallelism\" nil, \"zmq.threads\" 1, \"storm.group.mapping.service\" \"backtype.storm.security.auth.ShellBasedGroupsMapping\", \"transactional.zookeeper.root\" \"/transactional\", \"topology.sleep.spout.wait.strategy.time.ms\" 1, \"drpc.invocations.port\" 3773, \"topology.multilang.serializer\" \"backtype.storm.multilang.JsonSerializer\", \"storm.messaging.netty.server_worker_threads\" 1, \"topology.max.error.report.per.interval\" 5, \"storm.thrift.transport\" \"backtype.storm.security.auth.SimpleTransportPlugin\", \"zmq.hwm\" 0, \"storm.principal.tolocal\" \"backtype.storm.security.auth.DefaultPrincipalToLocal\", \"supervisor.worker.shutdown.sleep.secs\" 1, \"storm.zookeeper.retry.times\" 5, \"ui.actions.enabled\" true, \"zmq.linger.millis\" 5000, \"supervisor.enable\" true, \"topology.stats.sample.rate\" 0.05, \"storm.messaging.netty.min_wait_ms\" 100, \"storm.zookeeper.port\" 2181, \"supervisor.heartbeat.frequency.secs\" 5, \"topology.enable.message.timeouts\" true, \"drpc.worker.threads\" 64, \"drpc.queue.size\" 128, \"mesos.container.docker.image\" \"docker-registry.aws.cmcrc.com:5000/storm:0.1.4-0.10.1-0.28.0-jdk8\", \"drpc.https.keystore.password\" \"\", \"logviewer.port\" 8000, \"nimbus.reassign\" true, \"topology.executor.send.buffer.size\" 1024, \"topology.mesos.executor.cpu\" 0.1, \"topology.spout.wait.strategy\" \"backtype.storm.spout.SleepSpoutWaitStrategy\", \"ui.host\" \"0.0.0.0\", \"storm.nimbus.retry.interval.millis\" 2000, \"nimbus.inbox.jar.expiration.secs\" 3600, \"dev.zookeeper.path\" \"/tmp/dev-storm-zookeeper\", \"topology.acker.executors\" nil, \"topology.fall.back.on.java.serialization\" true, \"storm.zookeeper.servers\" [\"mesos-zk-dev.aws.cmcrc.com\"], \"nimbus.thrift.threads\" 64, \"logviewer.cleanup.age.mins\" 10080, \"topology.worker.childopts\" nil, \"topology.classpath\" nil, \"supervisor.monitor.frequency.secs\" 3, \"nimbus.credential.renewers.freq.secs\" 600, \"topology.skip.missing.kryo.registrations\" false, \"drpc.authorizer.acl.filename\" \"drpc-auth-acl.yaml\", \"storm.group.mapping.service.cache.duration.secs\" 120, \"topology.testing.always.try.serialize\" false, \"nimbus.monitor.freq.secs\" 10, \"supervisor.supervisors\" [], \"topology.tasks\" nil, \"topology.bolts.outgoing.overflow.buffer.enable\" false, \"storm.messaging.netty.socket.backlog\" 500, \"topology.workers\" 1, \"storm.local.dir\" \"storm-local\", \"worker.childopts\" \"-Xmx384m\", \"storm.auth.simple-white-list.users\" [], \"topology.message.timeout.secs\" 30, \"topology.state.synchronization.timeout.secs\" 60, \"mesos.framework.name\" \"Storm\", \"topology.tuple.serializer\" \"backtype.storm.serialization.types.ListDelegateSerializer\", \"supervisor.supervisors.commands\" [], \"logviewer.childopts\" \"-Xmx128m\", \"topology.environment\" nil, \"topology.debug\" false, \"storm.messaging.netty.max_retries\" 300, \"ui.childopts\" \"-Xmx768m\", \"supervisor.autostart.logviewer\" true, \"storm.zookeeper.session.timeout\" 20000, \"drpc.childopts\" \"-Xmx768m\", \"drpc.http.creds.plugin\" \"backtype.storm.security.auth.DefaultHttpCredentialsPlugin\", \"storm.zookeeper.connection.timeout\" 15000, \"storm.zookeeper.auth.user\" nil, \"storm.meta.serialization.delegate\" \"backtype.storm.serialization.GzipThriftSerializationDelegate\", \"topology.max.spout.pending\" nil, \"nimbus.supervisor.timeout.secs\" 60, \"nimbus.task.timeout.secs\" 30, \"drpc.port\" 3772, \"storm.zookeeper.retry.intervalceiling.millis\" 30000, \"nimbus.thrift.port\" 7778, \"storm.auth.simple-acl.admins\" [], \"mesos.master.url\" \"zk://mesos-zk-dev.aws.cmcrc.com:2181/mesos\", \"storm.nimbus.retry.times\" 5, \"supervisor.worker.start.timeout.secs\" 120, \"storm.zookeeper.retry.interval\" 1000, \"logs.users\" nil, \"transactional.zookeeper.port\" nil, \"drpc.max_buffer_size\" 1048576, \"topology.mesos.worker.mem.mb\" 2048, \"task.credentials.poll.secs\" 30, \"drpc.https.keystore.type\" \"JKS\", \"topology.worker.receiver.thread.count\" 1, \"supervisor.slots.ports\" [6700 6701 6702 6703], \"topology.transfer.buffer.size\" 1024, \"topology.worker.shared.thread.pool.size\" 4, \"drpc.authorizer.acl.strict\" false, \"nimbus.file.copy.expiration.secs\" 600, \"topology.executor.receive.buffer.size\" 1024, \"nimbus.task.launch.secs\" 120, \"storm.local.mode.zmq\" false, \"storm.messaging.netty.buffer_size\" 5242880, \"worker.heartbeat.frequency.secs\" 1, \"ui.http.creds.plugin\" \"backtype.storm.security.auth.DefaultHttpCredentialsPlugin\", \"storm.zookeeper.root\" \"/storm\", \"topology.tick.tuple.freq.secs\" nil, \"drpc.https.port\" -1, \"task.refresh.poll.secs\" 10, \"mesos.framework.checkpoint\" true, \"task.heartbeat.frequency.secs\" 3, \"storm.messaging.netty.max_wait_ms\" 1000, \"nimbus.impersonation.authorizer\" \"backtype.storm.security.auth.authorizer.ImpersonationAuthorizer\", \"drpc.http.port\" 3774, \"topology.error.throttle.interval.secs\" 10, \"storm.messaging.transport\" \"backtype.storm.messaging.netty.Context\", \"topology.mesos.executor.mem.mb\" 2048, \"storm.messaging.netty.authentication\" false, \"topology.kryo.factory\" \"backtype.storm.serialization.DefaultKryoFactory\", \"worker.gc.childopts\" \"\", \"nimbus.topology.validator\" \"backtype.storm.nimbus.DefaultTopologyValidator\", \"nimbus.cleanup.inbox.freq.secs\" 600, \"ui.users\" nil, \"transactional.zookeeper.servers\" nil, \"supervisor.worker.timeout.secs\" 30, \"storm.zookeeper.auth.password\" nil, \"supervisor.childopts\" \"-Xmx400m\", \"ui.filter\" nil, \"ui.header.buffer.bytes\" 4096, \"topology.disruptor.wait.timeout.millis\" 1000, \"storm.nimbus.retry.intervalceiling.millis\" 60000, \"topology.trident.batch.emit.interval.millis\" 500, \"topology.disruptor.wait.strategy\" \"com.lmax.disruptor.BlockingWaitStrategy\", \"storm.auth.simple-acl.users\" [], \"drpc.invocations.threads\" 64, \"java.library.path\" \"/usr/local/lib:/opt/local/lib:/usr/lib\", \"ui.port\" 8080, \"storm.log.dir\" \"/mnt/mesos/sandbox/logs\", \"storm.messaging.netty.transfer.batch.size\" 262144, \"logviewer.appender.name\" \"A1\", \"nimbus.thrift.max_buffer_size\" 1048576, \"nimbus.host\" \"localhost\", \"storm.auth.simple-acl.users.commands\" [], \"drpc.request.timeout.secs\" 600}\r\n2016-07-08 06:07:12.861 s.m.MesosSupervisor [INFO] Waiting for executor to initialize...\r\n2016-07-08 06:07:12.862 s.m.MesosSupervisor [WARN] shutdown not implemented in executor null, so cowardly refusing to kill tasks\r\n```\r\n\r\nSo I went to the mesos master logs and mesos slave logs and docker logs. This doesn't look like a OOM problem. It just tries to start then immediately crashes. Mesos master reports an error 127:\r\n```\r\nI0708 06:16:16.388259    14 http.cpp:312] HTTP GET for /master/state.json from 172.31.33.142:49846 with User-Agent='Go-http-client/1.1'\r\nI0708 06:16:17.610108     8 master.cpp:4865] Executor 'word-count-1-1467957966' of framework 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-0013 on slave 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-S1 at slave(1)@172.31.42.110:5051 (172.31.42.110): terminated with signal Unknown signal 127\r\nI0708 06:16:17.610179     8 master.cpp:6516] Removing executor 'word-count-1-1467957966' with resources cpus(*):0.1; mem(*):2048 of framework 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-0013 on slave 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-S1 at slave(1)@172.31.42.110:5051 (172.31.42.110)\r\nI0708 06:16:17.615766     8 master.cpp:4763] Status update TASK_FAILED (UUID: 5ef78429-f501-4a63-8248-69c4dd32f108) for task 172.31.42.110-82-1467958575.262 of framework 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-0013 from slave 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-S1 at slave(1)@172.31.42.110:5051 (172.31.42.110)\r\nI0708 06:16:17.615805     8 master.cpp:4811] Forwarding status update TASK_FAILED (UUID: 5ef78429-f501-4a63-8248-69c4dd32f108) for task 172.31.42.110-82-1467958575.262 of framework 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-0013\r\nI0708 06:16:17.615880     8 master.cpp:6421] Updating the state of task 172.31.42.110-82-1467958575.262 of framework 66768c79-9ef7-44b3-8ee4-f3ec4124eb85-0013 (latest state: TASK_FAILED, status update state: TASK_FAILED)\r\n```\r\n\r\nDocker logs say it was asked to shutdown:\r\n```\r\nI0708 06:18:34.760038   102 exec.cpp:143] Version: 0.28.2\r\nI0708 06:18:34.762950   122 exec.cpp:390] Executor asked to shutdown\r\nKilled\r\n```\r\n\r\nDocker inspect doesn't report anything wrong. No OOM. Just:\r\n\"ExitCode\": 137,\r\n\r\nAll parameters in storm.yaml seems as expected.\r\n\r\nFor a moment I thought it was due to different mesos versions (I run 0.28.0) and the executor looks for a file /usr/lib/libmesos-0.28.0.so, so I also tried compiling the image to 0.10.1-0.28.0 (because the 0.10.0-0.28.0 in docker hub has a bad storm UI) but the problem was the same.\r\n\r\nAnd unfortunately I couldn't find a 0.28.2 image in the mesosphere docker hub repo and I don't like building my own mesos images, as I prefer sticking to the \"official\" ones.\r\n\r\nSo now I have no idea what's going on. Mesos keeps trying to start the executors every 2 mins but of course it keeps failing. I can't see any other problem.\r\n\r\nHas anyone any idea how to solve this?\r\n\r\nThanks!\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mesos/storm/issues/comments/232245113","html_url":"https://github.com/mesos/storm/issues/155#issuecomment-232245113","issue_url":"https://api.github.com/repos/mesos/storm/issues/155","id":232245113,"user":{"login":"erikdw","id":1111441,"avatar_url":"https://avatars.githubusercontent.com/u/1111441?v=3","gravatar_id":"","url":"https://api.github.com/users/erikdw","html_url":"https://github.com/erikdw","followers_url":"https://api.github.com/users/erikdw/followers","following_url":"https://api.github.com/users/erikdw/following{/other_user}","gists_url":"https://api.github.com/users/erikdw/gists{/gist_id}","starred_url":"https://api.github.com/users/erikdw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikdw/subscriptions","organizations_url":"https://api.github.com/users/erikdw/orgs","repos_url":"https://api.github.com/users/erikdw/repos","events_url":"https://api.github.com/users/erikdw/events{/privacy}","received_events_url":"https://api.github.com/users/erikdw/received_events","type":"User","site_admin":false},"created_at":"2016-07-13T03:19:19Z","updated_at":"2016-07-13T03:19:19Z","body":"@Spritekin : thanks for closing it for now.  Let us know what you find.  I spent a bit of time starting to get a coreos + mesos setup going in a VM, but most of the tutorial stuff online for that is very dated.  So put it on the backburner for now."}}
{"actor":{"display_login":"erikdw"}}