{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/1808","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/1808/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/1808/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/1808/events","html_url":"https://github.com/hashicorp/nomad/issues/1808","id":182662325,"number":1808,"title":"Services restarting after exit code 0","user":{"login":"jshaw86","id":382292,"avatar_url":"https://avatars.githubusercontent.com/u/382292?v=3","gravatar_id":"","url":"https://api.github.com/users/jshaw86","html_url":"https://github.com/jshaw86","followers_url":"https://api.github.com/users/jshaw86/followers","following_url":"https://api.github.com/users/jshaw86/following{/other_user}","gists_url":"https://api.github.com/users/jshaw86/gists{/gist_id}","starred_url":"https://api.github.com/users/jshaw86/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jshaw86/subscriptions","organizations_url":"https://api.github.com/users/jshaw86/orgs","repos_url":"https://api.github.com/users/jshaw86/repos","events_url":"https://api.github.com/users/jshaw86/events{/privacy}","received_events_url":"https://api.github.com/users/jshaw86/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/nomad/labels/waiting-reply","name":"waiting-reply","color":"d4c5f9"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-10-12T23:52:15Z","updated_at":"2016-10-13T17:03:30Z","closed_at":"2016-10-13T17:03:30Z","body":"### Nomad version\r\nnomad v0.4.1\r\n\r\n### Operating system and Environment details\r\n14.04\r\n\r\n### Issue\r\nThere could be cases where a service wants to exit itself gracefully (returns 0). Nomad currently doesn't respect this exit code and tries to restart the process.\r\n\r\n### Reproduction steps\r\n1. Create a service that exits with 0 (I created a node process)\r\n2. Run the service as you would any other service\r\n3. Monitor the `nomad alloc-status`\r\n\r\n### Nomad Client logs (if appropriate)\r\nNOTE: I cherry picked these out of syslog for conciseness. \r\n```\r\nOct 12 23:41:35 vagrant-ubuntu-trusty-64 nomad[1960]: client: starting task runners for alloc '8813e53b-a2d0-4397-e698-7e639ebfe747'\r\nOct 12 23:41:35 vagrant-ubuntu-trusty-64 nomad[1960]: client: starting task context for 'exitcodezero' (alloc '8813e53b-a2d0-4397-e698-7e639ebfe747')\r\nOct 12 23:41:35 vagrant-ubuntu-trusty-64 nomad[1960]: plugin: starting plugin: /opt/nomad/nomad []string{\"/opt/nomad/nomad\", \"executor\", \"/opt/nomad/data/alloc/8813e53b-a2d0-4397-e698-7e639ebfe747/exitcodezero/exitcodezero-executor.out\"}\r\nOct 12 23:41:35 vagrant-ubuntu-trusty-64 nomad[1960]: plugin: waiting for RPC address for: /opt/nomad/nomad\r\nOct 12 23:41:43 vagrant-ubuntu-trusty-64 consul[1935]: agent: Synced service '_nomad-executor-8813e53b-a2d0-4397-e698-7e639ebfe747-exitcodezero-exitcodezero'\r\nOct 12 23:41:44 vagrant-ubuntu-trusty-64 nomad[1960]: client: task \"exitcodezero\" for alloc \"8813e53b-a2d0-4397-e698-7e639ebfe747\" completed successfully\r\n```\r\n\r\n```\r\nSummary\r\nTask Group    Queued  Starting  Running  Failed  Complete  Lost\r\nexitcodezero  0       22        3        0       0         0\r\n\r\nAllocations\r\nID        Eval ID   Node ID   Task Group    Desired  Status   Created At\r\n017cec2f  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n1214756b  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n16df915e  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n230ff063  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n26aa3d37  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n353d15ea  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n36682a20  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n39581d92  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n3c880d33  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n48a77b88  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n553826b7  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n554844b6  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n6253dfdc  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n6d4feea0  2598fff7  2b710653  exitcodezero  run      running  10/12/16 16:41:35 PDT\r\n76879f56  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n83b16ace  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n8813e53b  2598fff7  2b710653  exitcodezero  run      running  10/12/16 16:41:35 PDT\r\n947a72d1  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\n96eaf6ff  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\nb37bc73d  2598fff7  2b710653  exitcodezero  run      running  10/12/16 16:41:35 PDT\r\nbbbca837  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\ne0c7bfe8  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\ned0c6391  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\nf1955472  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\nfae9cae9  2598fff7  2b710653  exitcodezero  run      pending  10/12/16 16:41:35 PDT\r\nvagrant@blocks1:~/exit_code_zero$ nomad alloc-status 8813e53b\r\nID            = 8813e53b\r\nEval ID       = 2598fff7\r\nName          = exitcodezero.exitcodezero[22]\r\nNode ID       = 2b710653\r\nJob ID        = exitcodezero\r\nClient Status = pending\r\n\r\nTask \"exitcodezero\" is \"pending\"\r\nTask Resources\r\nCPU     Memory  Disk     IOPS  Addresses\r\n20 MHz  10 MiB  300 MiB  0     http: 127.0.0.1:47852\r\n\r\nRecent Events:\r\nTime                   Type        Description\r\n10/12/16 16:41:44 PDT  Restarting  Task restarting in 15.803725806s\r\n10/12/16 16:41:44 PDT  Terminated  Exit Code: 0\r\n10/12/16 16:41:43 PDT  Started     Task started by client\r\n10/12/16 16:41:35 PDT  Received    Task received by client\r\nvagrant@blocks1:~/exit_code_zero$ nomad alloc-status 8813e53b\r\nID            = 8813e53b\r\nEval ID       = 2598fff7\r\nName          = exitcodezero.exitcodezero[22]\r\nNode ID       = 2b710653\r\nJob ID        = exitcodezero\r\nClient Status = failed\r\n\r\nTask \"exitcodezero\" is \"dead\"\r\nTask Resources\r\nCPU     Memory  Disk     IOPS  Addresses\r\n20 MHz  10 MiB  300 MiB  0     http: 127.0.0.1:47852\r\n\r\nRecent Events:\r\nTime                   Type            Description\r\n10/12/16 16:42:00 PDT  Not Restarting  Error was unrecoverable\r\n10/12/16 16:42:00 PDT  Driver Failure  failed to start task 'exitcodezero' for alloc '8813e53b-a2d0-4397-e698-7e639ebfe747': Couldn't create destination file /opt/nomad/data/alloc/8813e53b-a2d0-4397-e698-7e639ebfe747/exitcodezero/usr/bin/blocks-usage: open /opt/nomad/data/alloc/8813e53b-a2d0-4397-e698-7e639ebfe747/exitcodezero/usr/bin/blocks-usage: text file busy\r\n10/12/16 16:41:44 PDT  Restarting      Task restarting in 15.803725806s\r\n10/12/16 16:41:44 PDT  Terminated      Exit Code: 0\r\n10/12/16 16:41:43 PDT  Started         Task started by client\r\n10/12/16 16:41:35 PDT  Received        Task received by client\r\n```\r\n\r\nNOTE: The fact that it runs into the chroot issue is besides the point, the issue in this case is that it tries to restart the task.\r\n\r\n### Job file (if appropriate)\r\n```\r\n{\r\n    \"Job\": {\r\n        \"Region\": \"vagrant-blu-1\",\r\n        \"ID\": \"exitcodezero\",\r\n        \"Name\": \"exitcodezero\",\r\n        \"Type\": \"service\",\r\n        \"Priority\": 50,\r\n        \"AllAtOnce\": false,\r\n        \"Datacenters\": [\r\n            \"dev-vagrant-blu-1\"\r\n        ],\r\n        \"Constraints\": null,\r\n        \"TaskGroups\": [\r\n            {\r\n                \"Name\": \"exitcodezero\",\r\n                \"Count\": 25,\r\n                \"Constraints\": null,\r\n                \"Tasks\": [\r\n                    {\r\n                        \"Name\": \"exitcodezero\",\r\n                        \"Driver\": \"exec\",\r\n                        \"User\": \"\",\r\n                        \"Config\": {\r\n                            \"args\": [\r\n                                \"/usr/lib/exitcodezero/process.js\"\r\n                            ],\r\n                            \"command\": \"/usr/bin/node\"\r\n                        },\r\n                        \"Constraints\": null,\r\n                        \"Env\": null,\r\n                        \"Services\": [\r\n                            {\r\n                                \"Id\": \"\",\r\n                                \"Name\": \"exitcodezero\",\r\n                                \"Tags\": null,\r\n                                \"PortLabel\": \"http\",\r\n                                \"Checks\": [\r\n                                    {\r\n                                        \"Id\": \"\",\r\n                                        \"Name\": \"ping_exitcodezero\",\r\n                                        \"Type\": \"http\",\r\n                                        \"Command\": \"\",\r\n                                        \"Args\": null,\r\n                                        \"Path\": \"/\",\r\n                                        \"Protocol\": \"\",\r\n                                        \"PortLabel\": \"\",\r\n                                        \"Interval\": 2000000000,\r\n                                        \"Timeout\": 2000000000\r\n                                    }\r\n                                ]\r\n                            }\r\n                        ],\r\n                        \"Resources\": {\r\n                            \"CPU\": 20,\r\n                            \"MemoryMB\": 10,\r\n                            \"DiskMB\": 300,\r\n                            \"IOPS\": 0,\r\n                            \"Networks\": [\r\n                                {\r\n                                    \"Public\": false,\r\n                                    \"CIDR\": \"\",\r\n                                    \"ReservedPorts\": null,\r\n                                    \"DynamicPorts\": [\r\n                                        {\r\n                                            \"Label\": \"http\",\r\n                                            \"Value\": 0\r\n                                        }\r\n                                    ],\r\n                                    \"IP\": \"\",\r\n                                    \"MBits\": 1\r\n                                }\r\n                            ]\r\n                        },\r\n                        \"Meta\": null,\r\n                        \"KillTimeout\": 5000000000,\r\n                        \"LogConfig\": {\r\n                            \"MaxFiles\": 10,\r\n                            \"MaxFileSizeMB\": 10\r\n                        },\r\n                        \"Artifacts\": null\r\n                    }\r\n                ],\r\n                \"RestartPolicy\": {\r\n                    \"Interval\": 60000000000,\r\n                    \"Attempts\": 2,\r\n                    \"Delay\": 15000000000,\r\n                    \"Mode\": \"delay\"\r\n                },\r\n                \"Meta\": null\r\n            }\r\n        ],\r\n        \"Update\": {\r\n            \"Stagger\": 0,\r\n            \"MaxParallel\": 0\r\n        },\r\n        \"Periodic\": null,\r\n        \"Meta\": null,\r\n        \"Status\": \"dead\",\r\n        \"StatusDescription\": \"\",\r\n        \"CreateIndex\": 105,\r\n        \"ModifyIndex\": 157,\r\n        \"JobModifyIndex\": 105\r\n    }\r\n}\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/comments/253574783","html_url":"https://github.com/hashicorp/nomad/issues/1808#issuecomment-253574783","issue_url":"https://api.github.com/repos/hashicorp/nomad/issues/1808","id":253574783,"user":{"login":"dadgar","id":1047914,"avatar_url":"https://avatars.githubusercontent.com/u/1047914?v=3","gravatar_id":"","url":"https://api.github.com/users/dadgar","html_url":"https://github.com/dadgar","followers_url":"https://api.github.com/users/dadgar/followers","following_url":"https://api.github.com/users/dadgar/following{/other_user}","gists_url":"https://api.github.com/users/dadgar/gists{/gist_id}","starred_url":"https://api.github.com/users/dadgar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadgar/subscriptions","organizations_url":"https://api.github.com/users/dadgar/orgs","repos_url":"https://api.github.com/users/dadgar/repos","events_url":"https://api.github.com/users/dadgar/events{/privacy}","received_events_url":"https://api.github.com/users/dadgar/received_events","type":"User","site_admin":false},"created_at":"2016-10-13T17:03:30Z","updated_at":"2016-10-13T17:03:30Z","body":"I am not sure it is though. The meaning is super unclear. If I have a service with count 3 and 2 have exited with code zero and there is a job update do I bring them back up? A service explicitly means that it should be up server traffic. \r\n\r\nI would use an external orchestrator to stop the job. "}}
{"actor":{"display_login":"dadgar"}}