{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/4063","repository_url":"https://api.github.com/repos/mesosphere/marathon","labels_url":"https://api.github.com/repos/mesosphere/marathon/issues/4063/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon/issues/4063/comments","events_url":"https://api.github.com/repos/mesosphere/marathon/issues/4063/events","html_url":"https://github.com/mesosphere/marathon/issues/4063","id":163607320,"number":4063,"title":"Unsuccessful leader election - [500 Internal Server on everything Marathon related -- DCOS 1.7 on Azure]","user":{"login":"ryankennedyio","id":11158006,"avatar_url":"https://avatars.githubusercontent.com/u/11158006?v=3","gravatar_id":"","url":"https://api.github.com/users/ryankennedyio","html_url":"https://github.com/ryankennedyio","followers_url":"https://api.github.com/users/ryankennedyio/followers","following_url":"https://api.github.com/users/ryankennedyio/following{/other_user}","gists_url":"https://api.github.com/users/ryankennedyio/gists{/gist_id}","starred_url":"https://api.github.com/users/ryankennedyio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryankennedyio/subscriptions","organizations_url":"https://api.github.com/users/ryankennedyio/orgs","repos_url":"https://api.github.com/users/ryankennedyio/repos","events_url":"https://api.github.com/users/ryankennedyio/events{/privacy}","received_events_url":"https://api.github.com/users/ryankennedyio/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mesosphere/marathon/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/mesosphere/marathon/labels/OKR%20Stability","name":"OKR Stability","color":"bfe5bf"},{"url":"https://api.github.com/repos/mesosphere/marathon/labels/service","name":"service","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/mesosphere/marathon/milestones/12","html_url":"https://github.com/mesosphere/marathon/milestones/Backlog","labels_url":"https://api.github.com/repos/mesosphere/marathon/milestones/12/labels","id":870039,"number":12,"title":"Backlog","description":"","creator":{"login":"ConnorDoyle","id":379372,"avatar_url":"https://avatars.githubusercontent.com/u/379372?v=3","gravatar_id":"","url":"https://api.github.com/users/ConnorDoyle","html_url":"https://github.com/ConnorDoyle","followers_url":"https://api.github.com/users/ConnorDoyle/followers","following_url":"https://api.github.com/users/ConnorDoyle/following{/other_user}","gists_url":"https://api.github.com/users/ConnorDoyle/gists{/gist_id}","starred_url":"https://api.github.com/users/ConnorDoyle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ConnorDoyle/subscriptions","organizations_url":"https://api.github.com/users/ConnorDoyle/orgs","repos_url":"https://api.github.com/users/ConnorDoyle/repos","events_url":"https://api.github.com/users/ConnorDoyle/events{/privacy}","received_events_url":"https://api.github.com/users/ConnorDoyle/received_events","type":"User","site_admin":false},"open_issues":284,"closed_issues":87,"state":"open","created_at":"2014-11-17T19:11:38Z","updated_at":"2016-07-26T10:13:12Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-07-04T05:50:10Z","updated_at":"2016-07-28T04:58:40Z","closed_at":null,"body":"Hi guys,\r\n\r\nRan into this fun one about half an hour ago. Wasn't doing anything I hadn't done before -- I think I was changing the ports in one of the apps loaded before scaling it up.\r\n\r\nAll the docker images that were running from Marathon still seem to be up.  DCOS is showing the correct CPU/Memory allocated. Mesos was giving me 503 errors for a few minutes but has fixed itself -- showing all slaves correctly but no active tasks or frameworks shown.\r\n\r\nAnyway, have forwarded my DCOS Master http to Localhost to view the UI.\r\n\r\n### DC/OS Dashboard\r\nhttp://localhost:8000/#/dashboard/ is fine\r\nDCOS Dashboard -> Services shows this:\r\n![screen shot 2016-07-04 at 3 10 08 pm](https://cloud.githubusercontent.com/assets/11158006/16551627/8edfff14-41f9-11e6-96a2-73aa8102f783.png)\r\n\r\n### Marathon Dashboard\r\nhttp://localhost:8000/service/marathon/ui/# shows this:\r\n![screen shot 2016-07-04 at 3 10 52 pm](https://cloud.githubusercontent.com/assets/11158006/16551624/88d14a6a-41f9-11e6-9f77-8a8ca017faef.png)\r\n\r\n\r\n### Mesos UI\r\nShowing this -- no active tasks/frameowkrs and apparently all resources are idle.\r\n![screen shot 2016-07-04 at 3 12 32 pm](https://cloud.githubusercontent.com/assets/11158006/16551665/f0bac11a-41f9-11e6-993d-96ee2289c48c.png)\r\n\r\n\r\nHowever if I go over to `Slaves`, we see:\r\n![screen shot 2016-07-04 at 3 14 05 pm](https://cloud.githubusercontent.com/assets/11158006/16551676/00fbb2aa-41fa-11e6-89a1-796401bdb215.png)\r\nThe \"Registered\" column is roughly when this 500 issue started.\r\n\r\n\r\n\r\n### Command  Line from Master\r\n`user@dcos-master-0:~$ dcos marathon app list`\r\n`Error decoding response from [https://localhost/service/marathon/v2/info]: HTTP 500: Internal Server Error`\r\n\r\n`user@dcos-master-0:~$ dcos marathon about`\r\n`Error decoding response from [https://localhost/service/marathon/v2/info]: HTTP 500: Internal Server Error`\r\n\r\n`user@dcos-master-0:~$ dcos package list`\r\n`Unexpected down stream http error: GET /v2/apps 503`\r\n\r\n\r\n\r\n#### Logs\r\n`$ journalctl --follow -u dcos-marathon` isn't giving me anything helpful:\r\n```\r\ndevops@dcos-master-01234567-0:~$ journalctl --follow -u dcos-marathon\r\n-- Logs begin at Sun 2016-07-03 19:34:07 UTC. --\r\nJul 04 05:46:34 dcos-master-01234567-0 java[26729]: [2016-07-04 05:46:34,258] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME from 172.16.0.5:8080 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder$:qtp325674467-37)\r\nJul 04 05:46:37 dcos-master-01234567-0 java[26729]: [2016-07-04 05:46:37,022] INFO 127.0.0.1 - - [04/Jul/2016:05:46:34 +0000] \"GET //127.0.0.1/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME HTTP/1.1\" 503 351 \"-\" \"Resty/HTTP-Simple 0.1.0 (Lua)\" 276 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$671419b5:qtp325674467-37)\r\nJul 04 05:46:43 dcos-master-01234567-0 java[26729]: [2016-07-04 05:46:43,446] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps from 172.16.0.5:8080 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder$:qtp325674467-35)\r\nJul 04 05:46:46 dcos-master-01234567-0 java[26729]: [2016-07-04 05:46:46,214] INFO 172.16.0.5 - - [04/Jul/2016:05:46:43 +0000] \"GET //master.mesos:8080/v2/apps HTTP/1.1\" 503 351 \"-\" \"-\"  276 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$671419b5:qtp325674467-35)\r\nJul 04 05:47:07 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:07,668] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME from 172.16.0.5:8080 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder$:qtp325674467-37)\r\nJul 04 05:47:10 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:10,433] INFO 127.0.0.1 - - [04/Jul/2016:05:47:07 +0000] \"GET //127.0.0.1/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME HTTP/1.1\" 503 351 \"-\" \"Resty/HTTP-Simple 0.1.0 (Lua)\" 276 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$671419b5:qtp325674467-37)\r\nJul 04 05:47:28 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:28,328] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME from 172.16.0.5:8080 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder$:qtp325674467-38)\r\nJul 04 05:47:31 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:31,094] INFO 127.0.0.1 - - [04/Jul/2016:05:47:28 +0000] \"GET //127.0.0.1/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME HTTP/1.1\" 503 351 \"-\" \"Resty/HTTP-Simple 0.1.0 (Lua)\" 276 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$671419b5:qtp325674467-38)\r\nJul 04 05:47:53 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:53,327] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME from 172.16.0.5:8080 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder$:qtp325674467-33)\r\nJul 04 05:47:56 dcos-master-01234567-0 java[26729]: [2016-07-04 05:47:56,097] INFO 127.0.0.1 - - [04/Jul/2016:05:47:53 +0000] \"GET //127.0.0.1/v2/apps?embed=apps.tasks&label=DCOS_SERVICE_NAME HTTP/1.1\" 503 351 \"-\" \"Resty/HTTP-Simple 0.1.0 (Lua)\" 277 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$671419b5:qtp325674467-33)\r\nJul 04 05:48:24 dcos-master-01234567-0 java[26729]: [2016-07-04 05:48:24,821] INFO Proxying request to GET http://172.16.0.9:8080/v2/apps?embed=apps.tasks from 17\r\n```"},"comment":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/comments/235801237","html_url":"https://github.com/mesosphere/marathon/issues/4063#issuecomment-235801237","issue_url":"https://api.github.com/repos/mesosphere/marathon/issues/4063","id":235801237,"user":{"login":"LinkMJB","id":8824103,"avatar_url":"https://avatars.githubusercontent.com/u/8824103?v=3","gravatar_id":"","url":"https://api.github.com/users/LinkMJB","html_url":"https://github.com/LinkMJB","followers_url":"https://api.github.com/users/LinkMJB/followers","following_url":"https://api.github.com/users/LinkMJB/following{/other_user}","gists_url":"https://api.github.com/users/LinkMJB/gists{/gist_id}","starred_url":"https://api.github.com/users/LinkMJB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LinkMJB/subscriptions","organizations_url":"https://api.github.com/users/LinkMJB/orgs","repos_url":"https://api.github.com/users/LinkMJB/repos","events_url":"https://api.github.com/users/LinkMJB/events{/privacy}","received_events_url":"https://api.github.com/users/LinkMJB/received_events","type":"User","site_admin":false},"created_at":"2016-07-28T04:58:40Z","updated_at":"2016-07-28T04:58:40Z","body":"I see this same problem when disk latency prevents timely heartbeat response as well (my current setup uses a common supporting RAID to support the entire DC/OS deployment--roadmap exists to expand/scale out to additional/separate hardware in the near future). It causes all services managed by Marathon to restart when this condition occurs. Glad to see there is a proposed fix. Looking forward to upgrading!"}}
{"actor":{"display_login":"LinkMJB"}}