{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/176","repository_url":"https://api.github.com/repos/cloudfoundry/diego-release","labels_url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/176/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/176/comments","events_url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/176/events","html_url":"https://github.com/cloudfoundry/diego-release/issues/176","id":165694726,"number":176,"title":"post-start script of rep fail for a single node cloud foundry deployment","user":{"login":"gossion","id":2063764,"avatar_url":"https://avatars.githubusercontent.com/u/2063764?v=3","gravatar_id":"","url":"https://api.github.com/users/gossion","html_url":"https://github.com/gossion","followers_url":"https://api.github.com/users/gossion/followers","following_url":"https://api.github.com/users/gossion/following{/other_user}","gists_url":"https://api.github.com/users/gossion/gists{/gist_id}","starred_url":"https://api.github.com/users/gossion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gossion/subscriptions","organizations_url":"https://api.github.com/users/gossion/orgs","repos_url":"https://api.github.com/users/gossion/repos","events_url":"https://api.github.com/users/gossion/events{/privacy}","received_events_url":"https://api.github.com/users/gossion/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry/diego-release/labels/in%20progress","name":"in progress","color":"3461AD"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-15T01:45:17Z","updated_at":"2016-07-22T17:20:32Z","closed_at":null,"body":"\r\nI was trying to deploy all cf & diego jobs in a single node, and got an error saying that post-start script of rep faill.\r\n\r\nbosh task log:\r\n```\r\nGetting deployment properties from director...\r\nUnable to get properties list from director, trying without it...\r\nCannot get current deployment information from director, possibly a new deployment\r\n\r\nDeploying\r\n---------\r\n\r\nDirector task 512\r\n  Started preparing deployment > Preparing deployment. Done (00:00:01)\r\n\r\n  Started preparing package compilation > Finding packages to compile. Done (00:00:00)\r\n\r\n  Started creating missing vms\r\n  Started creating missing vms > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0)\r\n  Started creating missing vms > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444)\r\n     Done creating missing vms > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0) (00:02:26)\r\n     Done creating missing vms > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444) (00:02:49)\r\n     Done creating missing vms (00:02:49)\r\n\r\n  Started updating job postgres_z1 > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0) (canary). Done (00:01:25)\r\n  Started updating job cf_z1 > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444) (canary). Failed: Action Failed get_task: Task 4e432186-719b-407f-618f-43a9db41aa96 result: 1 of 4 post-start scripts failed. Failed Jobs: rep. Successful Jobs: bbs, uaa, cloud_controller_ng. (00:16:23)\r\n\r\nError 450001: Action Failed get_task: Task 4e432186-719b-407f-618f-43a9db41aa96 result: 1 of 4 post-start scripts failed. Failed Jobs: rep. Successful Jobs: bbs, uaa, cloud_controller_ng.\r\n\r\nTask 512 error\r\n```\r\n\r\nmonit summary indicated that all jobs are running:\r\n```\r\nThe Monit daemon 5.2.5 uptime: 18m\r\n\r\nProcess 'consul_agent'              running\r\nProcess 'nats'                      running\r\nProcess 'nats_stream_forwarder'     running\r\nProcess 'metron_agent'              running\r\nProcess 'etcd'                      running\r\nProcess 'etcd_metrics_server'       running\r\nProcess 'rpc_nfsd'                  running\r\nProcess 'rpc_mountd'                running\r\nProcess 'cloud_controller_ng'       running\r\nProcess 'cloud_controller_worker_local_1' running\r\nProcess 'cloud_controller_worker_local_2' running\r\nProcess 'nginx_cc'                  running\r\nProcess 'cloud_controller_worker_1' running\r\nProcess 'cloud_controller_clock'    running\r\nFile 'nfs_mounter'                  accessible\r\nProcess 'route_registrar'           running\r\nProcess 'consul_template'           running\r\nFile 'haproxy_config'               accessible\r\nProcess 'haproxy'                   running\r\nProcess 'doppler'                   running\r\nProcess 'loggregator_trafficcontroller' running\r\nProcess 'uaa'                       running\r\nProcess 'gorouter'                  running\r\nProcess 'statsd-injector'           running\r\nProcess 'syslog_drain_binder'       running\r\nProcess 'bbs'                       running\r\nProcess 'auctioneer'                running\r\nProcess 'converger'                 running\r\nProcess 'rep'                       running\r\nProcess 'garden'                    running\r\nProcess 'stager'                    running\r\nProcess 'nsync_listener'            running\r\nProcess 'nsync_bulker'              running\r\nProcess 'tps_listener'              running\r\nProcess 'tps_watcher'               running\r\nProcess 'cc_uploader'               running\r\nProcess 'ssh_proxy'                 running\r\nProcess 'file_server'               running\r\nProcess 'route_emitter'             running\r\nSystem 'system_localhost'           running\r\n```\r\n\r\nBut from post-start log of rep, it failed for \"timout out pinging rep\". \r\n```\r\nThu Jul 14 06:26:18 UTC 2016: Pinging rep...\r\nThu Jul 14 06:26:19 UTC 2016: Attempt 1 failed. Trying again...\r\nThu Jul 14 06:26:20 UTC 2016: Attempt 2 failed. Trying again...\r\nThu Jul 14 06:26:21 UTC 2016: Attempt 3 failed. Trying again...\r\nThu Jul 14 06:26:22 UTC 2016: Attempt 4 failed. Trying again...\r\n...\r\nThu Jul 14 06:31:14 UTC 2016: Attempt 291 failed. Trying again...\r\nThu Jul 14 06:31:15 UTC 2016: Attempt 292 failed. Trying again...\r\nThu Jul 14 06:31:16 UTC 2016: Attempt 293 failed. Trying again...\r\nThu Jul 14 06:31:17 UTC 2016: Attempt 294 failed. Trying again...\r\nThu Jul 14 06:31:18 UTC 2016: Timed out pinging rep. Failing post-start.\r\n```\r\n\r\nFrom /var/vcap/monit/monit.log, I can see rep has restarted several times.\r\n```\r\n[UTC Jul 14 06:13:48] info     : monit: generated unique Monit id da648ecabcdb851f427aa42e45db1b20 and stored to '/root/.monit.id'\r\n[UTC Jul 14 06:13:48] info     : Starting monit daemon with http interface at [127.0.0.1:2822]\r\n[UTC Jul 14 06:13:48] info     : Starting monit HTTP server at [127.0.0.1:2822]\r\n[UTC Jul 14 06:13:48] info     : monit HTTP server started\r\n[UTC Jul 14 06:13:48] info     : 'system_localhost' Monit started\r\n[UTC Jul 14 06:19:46] info     : Reinitializing monit daemon\r\n[UTC Jul 14 06:19:46] info     : Awakened by the SIGHUP signal\r\n[UTC Jul 14 06:19:46] info     : Reinitializing monit - Control file '/var/vcap/bosh/etc/monitrc'\r\n[UTC Jul 14 06:19:46] info     : Shutting down monit HTTP server\r\n[UTC Jul 14 06:19:46] info     : monit HTTP server stopped\r\n[UTC Jul 14 06:19:46] warning  : /var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc'\r\n[UTC Jul 14 06:19:46] info     : Starting monit HTTP server at [127.0.0.1:2822]\r\n[UTC Jul 14 06:19:46] info     : monit HTTP server started\r\n[UTC Jul 14 06:19:46] info     : 'system_localhost' Monit reloaded\r\n[UTC Jul 14 06:22:33] info     : Reinitializing monit daemon\r\n[UTC Jul 14 06:22:33] info     : Awakened by the SIGHUP signal\r\n[UTC Jul 14 06:22:33] info     : Reinitializing monit - Control file '/var/vcap/bosh/etc/monitrc'\r\n[UTC Jul 14 06:22:33] info     : Shutting down monit HTTP server\r\n[UTC Jul 14 06:22:33] info     : monit HTTP server stopped\r\n[UTC Jul 14 06:22:33] info     : Starting monit HTTP server at [127.0.0.1:2822]\r\n[UTC Jul 14 06:22:33] info     : monit HTTP server started\r\n[UTC Jul 14 06:22:33] info     : 'system_localhost' Monit reloaded\r\n[UTC Jul 14 06:22:33] error    : 'consul_agent' process is not running\r\n[UTC Jul 14 06:22:33] info     : 'consul_agent' trying to restart\r\n[UTC Jul 14 06:22:33] info     : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl\r\n[UTC Jul 14 06:22:38] info     : start service 'route_emitter' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'file_server' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'ssh_proxy' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cc_uploader' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'tps_watcher' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'tps_listener' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nsync_bulker' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nsync_listener' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'stager' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'garden' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'rep' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'converger' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'auctioneer' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'bbs' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'syslog_drain_binder' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'statsd-injector' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'gorouter' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'uaa' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'loggregator_trafficcontroller' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'doppler' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'haproxy' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'haproxy_config' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'consul_template' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'route_registrar' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nfs_mounter' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_clock' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_1' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nginx_cc' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_local_2' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_local_1' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_ng' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'rpc_mountd' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'rpc_nfsd' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'etcd_metrics_server' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'etcd' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'metron_agent' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nats_stream_forwarder' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'nats' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:38] info     : start service 'consul_agent' on user request\r\n[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened\r\n[UTC Jul 14 06:22:40] info     : 'consul_template' start: /var/vcap/jobs/haproxy/bin/consul_template_ctl\r\n[UTC Jul 14 06:22:41] info     : 'nats' start: /var/vcap/jobs/nats/bin/nats_ctl\r\n[UTC Jul 14 06:22:42] info     : 'nats_stream_forwarder' start: /var/vcap/jobs/nats_stream_forwarder/bin/nats_stream_forwarder_ctl\r\n[UTC Jul 14 06:22:43] info     : 'nats' start action done\r\n[UTC Jul 14 06:22:43] info     : 'nats_stream_forwarder' start action done\r\n[UTC Jul 14 06:22:43] info     : 'metron_agent' start: /var/vcap/jobs/metron_agent/bin/metron_agent_ctl\r\n[UTC Jul 14 06:22:44] info     : 'metron_agent' start action done\r\n[UTC Jul 14 06:22:44] info     : 'etcd' start: /usr/bin/timeout\r\n[UTC Jul 14 06:22:52] info     : 'etcd' start action done\r\n[UTC Jul 14 06:22:53] info     : 'etcd_metrics_server' start: /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl\r\n[UTC Jul 14 06:22:54] info     : 'etcd_metrics_server' start action done\r\n[UTC Jul 14 06:22:54] info     : 'rpc_nfsd' start: /var/vcap/jobs/debian_nfs_server/bin/rpc_nfsd_ctl\r\n[UTC Jul 14 06:23:00] info     : 'rpc_mountd' start: /var/vcap/jobs/debian_nfs_server/bin/rpc_mountd_ctl\r\n[UTC Jul 14 06:23:01] info     : 'rpc_nfsd' start action done\r\n[UTC Jul 14 06:23:01] info     : 'rpc_mountd' start action done\r\n[UTC Jul 14 06:23:01] info     : 'cloud_controller_ng' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl\r\n[UTC Jul 14 06:23:31] error    : 'cloud_controller_ng' failed to start\r\n[UTC Jul 14 06:23:31] info     : 'nginx_cc' start: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl\r\n[UTC Jul 14 06:23:32] info     : 'cloud_controller_ng' start action done\r\n[UTC Jul 14 06:23:32] info     : 'cloud_controller_worker_local_1' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl\r\n[UTC Jul 14 06:23:33] info     : 'cloud_controller_worker_local_1' start action done\r\n[UTC Jul 14 06:23:33] info     : 'cloud_controller_worker_local_2' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl\r\n[UTC Jul 14 06:23:34] info     : 'cloud_controller_worker_local_2' start action done\r\n[UTC Jul 14 06:23:34] info     : 'nginx_cc' start action done\r\n[UTC Jul 14 06:23:34] info     : 'cloud_controller_worker_1' start: /var/vcap/jobs/cloud_controller_worker/bin/cloud_controller_worker_ctl\r\n[UTC Jul 14 06:23:35] info     : 'cloud_controller_worker_1' start action done\r\n[UTC Jul 14 06:23:35] info     : 'cloud_controller_clock' start: /var/vcap/jobs/cloud_controller_clock/bin/cloud_controller_clock_ctl\r\n[UTC Jul 14 06:23:36] info     : 'cloud_controller_clock' start action done\r\n[UTC Jul 14 06:23:36] info     : 'nfs_mounter' start: /var/vcap/jobs/nfs_mounter/bin/nfs_mounter_ctl\r\n[UTC Jul 14 06:23:36] info     : 'nfs_mounter' start action done\r\n[UTC Jul 14 06:23:36] info     : 'route_registrar' start: /var/vcap/jobs/route_registrar/bin/route_registrar_ctl\r\n[UTC Jul 14 06:23:37] info     : 'route_registrar' start action done\r\n[UTC Jul 14 06:23:38] info     : 'consul_template' start action done\r\n[UTC Jul 14 06:23:38] info     : 'haproxy' start: /var/vcap/jobs/haproxy/bin/haproxy_ctl\r\n[UTC Jul 14 06:23:39] info     : 'haproxy' start action done\r\n[UTC Jul 14 06:23:39] info     : 'doppler' start: /var/vcap/jobs/doppler/bin/doppler_ctl\r\n[UTC Jul 14 06:23:40] info     : 'doppler' start action done\r\n[UTC Jul 14 06:23:40] info     : 'loggregator_trafficcontroller' start: /var/vcap/jobs/loggregator_trafficcontroller/bin/loggregator_trafficcontroller_ctl\r\n[UTC Jul 14 06:23:41] info     : 'loggregator_trafficcontroller' start action done\r\n[UTC Jul 14 06:23:41] info     : 'uaa' start: /var/vcap/jobs/uaa/bin/uaa_ctl\r\n[UTC Jul 14 06:23:42] info     : 'uaa' start action done\r\n[UTC Jul 14 06:23:42] info     : 'gorouter' start: /var/vcap/jobs/gorouter/bin/gorouter_ctl\r\n[UTC Jul 14 06:24:07] info     : 'gorouter' start action done\r\n[UTC Jul 14 06:24:07] info     : 'statsd-injector' start: /var/vcap/jobs/statsd-injector/bin/statsd-injector-ctl\r\n[UTC Jul 14 06:24:08] info     : 'statsd-injector' start action done\r\n[UTC Jul 14 06:24:08] info     : 'syslog_drain_binder' start: /var/vcap/jobs/syslog_drain_binder/bin/syslog_drain_binder_ctl\r\n[UTC Jul 14 06:24:09] info     : 'syslog_drain_binder' start action done\r\n[UTC Jul 14 06:24:09] info     : 'bbs' start: /var/vcap/jobs/bbs/bin/bbs_ctl\r\n[UTC Jul 14 06:24:10] info     : 'bbs' start action done\r\n[UTC Jul 14 06:24:10] info     : 'auctioneer' start: /var/vcap/jobs/auctioneer/bin/auctioneer_ctl\r\n[UTC Jul 14 06:24:11] info     : 'auctioneer' start action done\r\n[UTC Jul 14 06:24:11] info     : 'converger' start: /var/vcap/jobs/converger/bin/converger_ctl\r\n[UTC Jul 14 06:24:12] info     : 'converger' start action done\r\n[UTC Jul 14 06:24:12] info     : 'garden' start: /var/vcap/jobs/garden/bin/garden_ctl\r\n[UTC Jul 14 06:24:14] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl\r\n[UTC Jul 14 06:24:44] error    : 'rep' failed to start\r\n[UTC Jul 14 06:24:44] info     : 'garden' start action done\r\n[UTC Jul 14 06:24:44] info     : 'stager' start: /var/vcap/jobs/stager/bin/stager_ctl\r\n[UTC Jul 14 06:24:45] info     : 'stager' start action done\r\n[UTC Jul 14 06:24:45] info     : 'nsync_listener' start: /var/vcap/jobs/nsync/bin/nsync_listener_ctl\r\n[UTC Jul 14 06:24:46] info     : 'nsync_listener' start action done\r\n[UTC Jul 14 06:24:46] info     : 'nsync_bulker' start: /var/vcap/jobs/nsync/bin/nsync_bulker_ctl\r\n[UTC Jul 14 06:24:47] info     : 'nsync_bulker' start action done\r\n[UTC Jul 14 06:24:47] info     : 'tps_listener' start: /var/vcap/jobs/tps/bin/tps_listener_ctl\r\n[UTC Jul 14 06:24:48] info     : 'tps_listener' start action done\r\n[UTC Jul 14 06:24:48] info     : 'tps_watcher' start: /var/vcap/jobs/tps/bin/tps_watcher_ctl\r\n[UTC Jul 14 06:24:49] info     : 'tps_watcher' start action done\r\n[UTC Jul 14 06:24:49] info     : 'cc_uploader' start: /var/vcap/jobs/cc_uploader/bin/cc_uploader_ctl\r\n[UTC Jul 14 06:24:50] info     : 'cc_uploader' start action done\r\n[UTC Jul 14 06:24:50] info     : 'ssh_proxy' start: /var/vcap/jobs/ssh_proxy/bin/ssh_proxy_ctl\r\n[UTC Jul 14 06:24:51] info     : 'ssh_proxy' start action done\r\n[UTC Jul 14 06:24:52] info     : 'file_server' start: /var/vcap/jobs/file_server/bin/file_server_ctl\r\n[UTC Jul 14 06:24:53] info     : 'file_server' start action done\r\n[UTC Jul 14 06:24:53] info     : 'route_emitter' start: /var/vcap/jobs/route_emitter/bin/route_emitter_ctl\r\n[UTC Jul 14 06:24:54] info     : 'route_emitter' start action done\r\n[UTC Jul 14 06:24:54] info     : 'haproxy_config' start action done\r\n[UTC Jul 14 06:24:54] info     : 'rep' start action done\r\n[UTC Jul 14 06:24:54] info     : Awakened by User defined signal 1\r\n[UTC Jul 14 06:24:54] info     : 'consul_agent' start action done\r\n[UTC Jul 14 06:24:54] info     : 'consul_agent' process is running with pid 11565\r\n[UTC Jul 14 06:24:54] error    : 'cloud_controller_ng' process is not running\r\n[UTC Jul 14 06:24:54] info     : 'cloud_controller_ng' trying to restart\r\n[UTC Jul 14 06:24:54] info     : 'nginx_cc' stop: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl\r\n[UTC Jul 14 06:24:55] info     : 'cloud_controller_ng' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl\r\n[UTC Jul 14 06:25:03] info     : 'cloud_controller_ng' started\r\n[UTC Jul 14 06:25:03] info     : 'nginx_cc' start: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl\r\n[UTC Jul 14 06:25:16] error    : 'haproxy_config' timestamp was changed for /var/vcap/jobs/haproxy/config/haproxy.conf\r\n[UTC Jul 14 06:25:16] info     : 'haproxy_config' exec: /var/vcap/jobs/haproxy/bin/haproxy_ctl\r\n[UTC Jul 14 06:25:16] error    : 'rep' process is not running\r\n[UTC Jul 14 06:25:16] info     : 'rep' trying to restart\r\n[UTC Jul 14 06:25:16] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl\r\n[UTC Jul 14 06:25:18] info     : 'rep' started\r\n[UTC Jul 14 06:25:28] info     : 'cloud_controller_ng' process is running with pid 14525\r\n[UTC Jul 14 06:25:28] error    : 'haproxy' process PID changed from 13002 to 14800\r\n[UTC Jul 14 06:25:28] info     : 'haproxy_config' timestamp was not changed for /var/vcap/jobs/haproxy/config/haproxy.conf\r\n[UTC Jul 14 06:25:28] info     : 'rep' process is running with pid 14780\r\n[UTC Jul 14 06:25:38] info     : 'haproxy' process PID has not changed since last cycle\r\n[UTC Jul 14 06:32:09] error    : 'haproxy_config' timestamp was changed for /var/vcap/jobs/haproxy/config/haproxy.conf\r\n[UTC Jul 14 06:32:09] info     : 'haproxy_config' exec: /var/vcap/jobs/haproxy/bin/haproxy_ctl\r\n[UTC Jul 14 06:32:19] error    : 'haproxy' process PID changed from 14800 to 18242\r\n[UTC Jul 14 06:32:19] info     : 'haproxy_config' timestamp was not changed for /var/vcap/jobs/haproxy/config/haproxy.conf\r\n[UTC Jul 14 06:32:29] info     : 'haproxy' process PID has not changed since last cycle\r\n[UTC Jul 14 06:35:20] error    : 'rep' process is not running\r\n[UTC Jul 14 06:35:20] info     : 'rep' trying to restart\r\n[UTC Jul 14 06:35:20] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl\r\n[UTC Jul 14 06:35:31] info     : 'rep' process is running with pid 19090\r\n```\r\n\r\nThe result indicates that rep is not available before the post-start timeout (300 seconds).  So I tried to set [time out of rep post-start](https://github.com/cloudfoundry/diego-release/blob/develop/jobs/rep/templates/post-start.erb#L5) as 800, then I got my deployment pass. \r\n\r\nNot sure why rep need to restart, so my question is: is it reasonable to just change the timeout value of post-start for a single node deployment? If yes, I think a parameter such as diego.rep.post_start_timeout_in_seconds can be added to rep job spec; if not, I believe there is some other issues that I was not aware of, so could you please guide me how to debug the issue.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/comments/234603208","html_url":"https://github.com/cloudfoundry/diego-release/issues/176#issuecomment-234603208","issue_url":"https://api.github.com/repos/cloudfoundry/diego-release/issues/176","id":234603208,"user":{"login":"jenspinney","id":3200507,"avatar_url":"https://avatars.githubusercontent.com/u/3200507?v=3","gravatar_id":"","url":"https://api.github.com/users/jenspinney","html_url":"https://github.com/jenspinney","followers_url":"https://api.github.com/users/jenspinney/followers","following_url":"https://api.github.com/users/jenspinney/following{/other_user}","gists_url":"https://api.github.com/users/jenspinney/gists{/gist_id}","starred_url":"https://api.github.com/users/jenspinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jenspinney/subscriptions","organizations_url":"https://api.github.com/users/jenspinney/orgs","repos_url":"https://api.github.com/users/jenspinney/repos","events_url":"https://api.github.com/users/jenspinney/events{/privacy}","received_events_url":"https://api.github.com/users/jenspinney/received_events","type":"User","site_admin":false},"created_at":"2016-07-22T17:20:32Z","updated_at":"2016-07-22T17:20:32Z","body":"Hi @gossion, sorry about the delay. It seems like it's just taking a while to spin up a container, maybe because the VM is overloaded. Have you tried increasing the size of the VM? What does the resource utilization of the VM look like when you're seeing this?"}}
{"actor":{"display_login":"jenspinney"}}