{"action":"created","comment":{"url":"https://api.github.com/repos/PBSPro/pbspro/pulls/comments/212780940","pull_request_review_id":149488201,"id":212780940,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxMjc4MDk0MA==","diff_hunk":"@@ -1084,7 +1073,11 @@ svr_strtjob2(job *pjob, struct batch_request *preq)\n \t\t\tpjob->ji_qs.ji_jobid,\n \t\t\t\"Unable to Run Job, send to Mom failed\");\n \n-\t\tif (pjob->ji_qs.ji_substate == JOB_SUBSTATE_PROVISION)\n+\t\tif(pbs_errno == PBSE_NORELYMOM)\n+\t\t\tmark_node_down(pjob->ji_qs.ji_destin, \"could not send job to mom\");","path":"src/server/req_runjob.c","position":87,"original_position":87,"commit_id":"4242d0d2982f8395c415ac71ed10d952cf9e880c","original_commit_id":"4242d0d2982f8395c415ac71ed10d952cf9e880c","user":{"login":"arungrover","id":17924134,"node_id":"MDQ6VXNlcjE3OTI0MTM0","avatar_url":"https://avatars0.githubusercontent.com/u/17924134?v=4","gravatar_id":"","url":"https://api.github.com/users/arungrover","html_url":"https://github.com/arungrover","followers_url":"https://api.github.com/users/arungrover/followers","following_url":"https://api.github.com/users/arungrover/following{/other_user}","gists_url":"https://api.github.com/users/arungrover/gists{/gist_id}","starred_url":"https://api.github.com/users/arungrover/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arungrover/subscriptions","organizations_url":"https://api.github.com/users/arungrover/orgs","repos_url":"https://api.github.com/users/arungrover/repos","events_url":"https://api.github.com/users/arungrover/events{/privacy}","received_events_url":"https://api.github.com/users/arungrover/received_events","type":"User","site_admin":false},"body":"What happens when the server fails to connect to mom while staging in the files? It seems that svr_strtjob2 may not even get called in that case.","created_at":"2018-08-25T00:26:02Z","updated_at":"2018-08-25T00:28:22Z","html_url":"https://github.com/PBSPro/pbspro/pull/780#discussion_r212780940","pull_request_url":"https://api.github.com/repos/PBSPro/pbspro/pulls/780","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/comments/212780940"},"html":{"href":"https://github.com/PBSPro/pbspro/pull/780#discussion_r212780940"},"pull_request":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/780"}}},"pull_request":{"url":"https://api.github.com/repos/PBSPro/pbspro/pulls/780","id":208618369,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA4NjE4MzY5","html_url":"https://github.com/PBSPro/pbspro/pull/780","diff_url":"https://github.com/PBSPro/pbspro/pull/780.diff","patch_url":"https://github.com/PBSPro/pbspro/pull/780.patch","issue_url":"https://api.github.com/repos/PBSPro/pbspro/issues/780","number":780,"state":"open","locked":false,"title":"PP-1026: Stranded array subjobs","user":{"login":"Shrini-h","id":30307577,"node_id":"MDQ6VXNlcjMwMzA3NTc3","avatar_url":"https://avatars1.githubusercontent.com/u/30307577?v=4","gravatar_id":"","url":"https://api.github.com/users/Shrini-h","html_url":"https://github.com/Shrini-h","followers_url":"https://api.github.com/users/Shrini-h/followers","following_url":"https://api.github.com/users/Shrini-h/following{/other_user}","gists_url":"https://api.github.com/users/Shrini-h/gists{/gist_id}","starred_url":"https://api.github.com/users/Shrini-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shrini-h/subscriptions","organizations_url":"https://api.github.com/users/Shrini-h/orgs","repos_url":"https://api.github.com/users/Shrini-h/repos","events_url":"https://api.github.com/users/Shrini-h/events{/privacy}","received_events_url":"https://api.github.com/users/Shrini-h/received_events","type":"User","site_admin":false},"body":"<!--- Please review your changes in preview mode -->\r\n<!--- Provide a general summary of your changes in the Title above -->\r\n\r\n#### Bug/feature Description\r\n* *Sometimes instantiated subjobs can get stranded in queued state due to communication failure with execution node*\r\n* *Not easily reproducible, usually observed during communication hiccups in large clusters using large job arrays*\r\n* *Refer issue ID: [PP-1026](https://pbspro.atlassian.net/browse/PP-1026)*\r\n\r\n#### Affected Platform(s)\r\n* *All*\r\n\r\n#### Cause\r\n* *Instantiated subjobs in Q state. the subjobs were instantiated when the server first tried to run them, the find_jobs() in req_runjob() finds an entry for them, causing the server to reject them.*\r\n\r\n#### Solution Description\r\n* *Purge an instantiated Queued subjob and re-instantiate it afresh during runjob*\r\n\r\n#### Testing logs/output\r\n* *PTL test cannot be automated cause there is no way to create a stranded subjob through commands*\r\n* *Hence I devised below manual steps to verify the fix*\r\n\r\n1. Manually build the server by commenting out below lines (for force creating an instantiated Q subjob)\r\nhttps://github.com/PBSPro/pbspro/blob/882f33a732d82736aca098d8ee0a7e493287628c/src/server/req_runjob.c#L677-L681\r\n2. Comment out below line\r\nhttps://github.com/PBSPro/pbspro/blob/882f33a732d82736aca098d8ee0a7e493287628c/test/tests/functional/pbs_sched_subjob_badstate.py#L77\r\n3. Run PTL testcase\r\n`pbs_benchpress -t TestSchedSubjobBadstate`\r\n4. After the above test completes we can see\r\n from `/etc/init.d/pbs status` that `pbs_mom` is not running but `pbsnodes -av` shows the node as free\r\n from `qstat -t`  that all the subjob are in Q state,\r\n5. Manually Q sub some more regular jobs and array jobs. all will get queued. The first subjob of all the array are stranded subjobs.\r\n6. now start the pbs_mom.\r\n7. We should see that the stranded Q subjobs from the PTL test case and step 5 goes to Running\r\n8. Verify the server logs. We should see as seen in the sample attached below\r\n[strandedsubjob_logs.zip](https://github.com/PBSPro/pbspro/files/2315311/strandedsubjob_logs.zip)\r\nPTL test output attached below\r\n[ptl_test_results.zip](https://github.com/PBSPro/pbspro/files/2317518/ptl_test_results.zip)\r\n\r\n#### Checklist:\r\n<!--- Use the preview button to see the checkboxes/links properly. -->\r\n<!--- Go over the following points, and put an `x` (without spaces around it) in the boxes that apply. -->\r\n<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->\r\n- [x] I have joined the **[pbspro community forum](http://community.pbspro.org/)**.\r\n- [x] My pull request contains a **single, signed** commit. See **[setting up gpg signature](https://pbspro.atlassian.net/wiki/display/DG/Signing+Your+Git+Commits)**.\r\n- [x] My code follows the **[coding style](https://pbspro.atlassian.net/wiki/display/DG/Coding+Standards)** of this project.\r\n- [ ] My change requires project documentation. See **[required documentation checklist](https://pbspro.atlassian.net/wiki/display/DG/Checklist+for+Developing+Features+and+Bug+Fixes)** for details.\r\n   - [ ] I have added documentation in the **[project documentation area](https://pbspro.atlassian.net/wiki/display/PD)**.\r\n- [ ] I have added new **PTL test(s) to my commit**. (See **[using PTL for testing](https://pbspro.atlassian.net/wiki/display/DG/Using+PTL+for+Testing)**) *(or)*\r\n   - [x] I have added  **manual test(s) to this pull request and explained why PTL is not appropriate** for this case.\r\n- [x] All new and existing automated tests have passed. (See **[running automated PTL tests](https://pbspro.atlassian.net/wiki/display/DG/PTL+Quick+Start+Guide)**).\r\n- [x] I have attached **test logs to this pull request** as evidence of testing/verification.\r\n\r\n\r\n__***For further information please visit the [Developer Guide Home](https://pbspro.atlassian.net/wiki/display/DG/Developer+Guide+Home).***__\r\n","created_at":"2018-08-15T16:50:37Z","updated_at":"2018-08-25T00:28:22Z","closed_at":null,"merged_at":null,"merge_commit_sha":"545b286bfbc26d7103d789f401e2417bb1e78852","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/PBSPro/pbspro/pulls/780/commits","review_comments_url":"https://api.github.com/repos/PBSPro/pbspro/pulls/780/comments","review_comment_url":"https://api.github.com/repos/PBSPro/pbspro/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PBSPro/pbspro/issues/780/comments","statuses_url":"https://api.github.com/repos/PBSPro/pbspro/statuses/4242d0d2982f8395c415ac71ed10d952cf9e880c","head":{"label":"Shrini-h:dev_PP-1026","ref":"dev_PP-1026","sha":"4242d0d2982f8395c415ac71ed10d952cf9e880c","user":{"login":"Shrini-h","id":30307577,"node_id":"MDQ6VXNlcjMwMzA3NTc3","avatar_url":"https://avatars1.githubusercontent.com/u/30307577?v=4","gravatar_id":"","url":"https://api.github.com/users/Shrini-h","html_url":"https://github.com/Shrini-h","followers_url":"https://api.github.com/users/Shrini-h/followers","following_url":"https://api.github.com/users/Shrini-h/following{/other_user}","gists_url":"https://api.github.com/users/Shrini-h/gists{/gist_id}","starred_url":"https://api.github.com/users/Shrini-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shrini-h/subscriptions","organizations_url":"https://api.github.com/users/Shrini-h/orgs","repos_url":"https://api.github.com/users/Shrini-h/repos","events_url":"https://api.github.com/users/Shrini-h/events{/privacy}","received_events_url":"https://api.github.com/users/Shrini-h/received_events","type":"User","site_admin":false},"repo":{"id":97791084,"node_id":"MDEwOlJlcG9zaXRvcnk5Nzc5MTA4NA==","name":"pbspro","full_name":"Shrini-h/pbspro","owner":{"login":"Shrini-h","id":30307577,"node_id":"MDQ6VXNlcjMwMzA3NTc3","avatar_url":"https://avatars1.githubusercontent.com/u/30307577?v=4","gravatar_id":"","url":"https://api.github.com/users/Shrini-h","html_url":"https://github.com/Shrini-h","followers_url":"https://api.github.com/users/Shrini-h/followers","following_url":"https://api.github.com/users/Shrini-h/following{/other_user}","gists_url":"https://api.github.com/users/Shrini-h/gists{/gist_id}","starred_url":"https://api.github.com/users/Shrini-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shrini-h/subscriptions","organizations_url":"https://api.github.com/users/Shrini-h/orgs","repos_url":"https://api.github.com/users/Shrini-h/repos","events_url":"https://api.github.com/users/Shrini-h/events{/privacy}","received_events_url":"https://api.github.com/users/Shrini-h/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/Shrini-h/pbspro","description":"An HPC workload manager and job scheduler for desktops, clusters, and clouds.","fork":true,"url":"https://api.github.com/repos/Shrini-h/pbspro","forks_url":"https://api.github.com/repos/Shrini-h/pbspro/forks","keys_url":"https://api.github.com/repos/Shrini-h/pbspro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Shrini-h/pbspro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Shrini-h/pbspro/teams","hooks_url":"https://api.github.com/repos/Shrini-h/pbspro/hooks","issue_events_url":"https://api.github.com/repos/Shrini-h/pbspro/issues/events{/number}","events_url":"https://api.github.com/repos/Shrini-h/pbspro/events","assignees_url":"https://api.github.com/repos/Shrini-h/pbspro/assignees{/user}","branches_url":"https://api.github.com/repos/Shrini-h/pbspro/branches{/branch}","tags_url":"https://api.github.com/repos/Shrini-h/pbspro/tags","blobs_url":"https://api.github.com/repos/Shrini-h/pbspro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Shrini-h/pbspro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Shrini-h/pbspro/git/refs{/sha}","trees_url":"https://api.github.com/repos/Shrini-h/pbspro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Shrini-h/pbspro/statuses/{sha}","languages_url":"https://api.github.com/repos/Shrini-h/pbspro/languages","stargazers_url":"https://api.github.com/repos/Shrini-h/pbspro/stargazers","contributors_url":"https://api.github.com/repos/Shrini-h/pbspro/contributors","subscribers_url":"https://api.github.com/repos/Shrini-h/pbspro/subscribers","subscription_url":"https://api.github.com/repos/Shrini-h/pbspro/subscription","commits_url":"https://api.github.com/repos/Shrini-h/pbspro/commits{/sha}","git_commits_url":"https://api.github.com/repos/Shrini-h/pbspro/git/commits{/sha}","comments_url":"https://api.github.com/repos/Shrini-h/pbspro/comments{/number}","issue_comment_url":"https://api.github.com/repos/Shrini-h/pbspro/issues/comments{/number}","contents_url":"https://api.github.com/repos/Shrini-h/pbspro/contents/{+path}","compare_url":"https://api.github.com/repos/Shrini-h/pbspro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Shrini-h/pbspro/merges","archive_url":"https://api.github.com/repos/Shrini-h/pbspro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Shrini-h/pbspro/downloads","issues_url":"https://api.github.com/repos/Shrini-h/pbspro/issues{/number}","pulls_url":"https://api.github.com/repos/Shrini-h/pbspro/pulls{/number}","milestones_url":"https://api.github.com/repos/Shrini-h/pbspro/milestones{/number}","notifications_url":"https://api.github.com/repos/Shrini-h/pbspro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Shrini-h/pbspro/labels{/name}","releases_url":"https://api.github.com/repos/Shrini-h/pbspro/releases{/id}","deployments_url":"https://api.github.com/repos/Shrini-h/pbspro/deployments","created_at":"2017-07-20T04:39:41Z","updated_at":"2018-08-23T06:16:23Z","pushed_at":"2018-08-23T08:58:44Z","git_url":"git://github.com/Shrini-h/pbspro.git","ssh_url":"git@github.com:Shrini-h/pbspro.git","clone_url":"https://github.com/Shrini-h/pbspro.git","svn_url":"https://github.com/Shrini-h/pbspro","homepage":"www.pbspro.org","size":15937,"stargazers_count":1,"watchers_count":1,"language":"C","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"PBSPro:master","ref":"master","sha":"b8eb6ffb5eb68009aee1c87837f5325022326574","user":{"login":"PBSPro","id":19280436,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjgwNDM2","avatar_url":"https://avatars2.githubusercontent.com/u/19280436?v=4","gravatar_id":"","url":"https://api.github.com/users/PBSPro","html_url":"https://github.com/PBSPro","followers_url":"https://api.github.com/users/PBSPro/followers","following_url":"https://api.github.com/users/PBSPro/following{/other_user}","gists_url":"https://api.github.com/users/PBSPro/gists{/gist_id}","starred_url":"https://api.github.com/users/PBSPro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PBSPro/subscriptions","organizations_url":"https://api.github.com/users/PBSPro/orgs","repos_url":"https://api.github.com/users/PBSPro/repos","events_url":"https://api.github.com/users/PBSPro/events{/privacy}","received_events_url":"https://api.github.com/users/PBSPro/received_events","type":"Organization","site_admin":false},"repo":{"id":59082072,"node_id":"MDEwOlJlcG9zaXRvcnk1OTA4MjA3Mg==","name":"pbspro","full_name":"PBSPro/pbspro","owner":{"login":"PBSPro","id":19280436,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjgwNDM2","avatar_url":"https://avatars2.githubusercontent.com/u/19280436?v=4","gravatar_id":"","url":"https://api.github.com/users/PBSPro","html_url":"https://github.com/PBSPro","followers_url":"https://api.github.com/users/PBSPro/followers","following_url":"https://api.github.com/users/PBSPro/following{/other_user}","gists_url":"https://api.github.com/users/PBSPro/gists{/gist_id}","starred_url":"https://api.github.com/users/PBSPro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PBSPro/subscriptions","organizations_url":"https://api.github.com/users/PBSPro/orgs","repos_url":"https://api.github.com/users/PBSPro/repos","events_url":"https://api.github.com/users/PBSPro/events{/privacy}","received_events_url":"https://api.github.com/users/PBSPro/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PBSPro/pbspro","description":"An HPC workload manager and job scheduler for desktops, clusters, and clouds.","fork":false,"url":"https://api.github.com/repos/PBSPro/pbspro","forks_url":"https://api.github.com/repos/PBSPro/pbspro/forks","keys_url":"https://api.github.com/repos/PBSPro/pbspro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PBSPro/pbspro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PBSPro/pbspro/teams","hooks_url":"https://api.github.com/repos/PBSPro/pbspro/hooks","issue_events_url":"https://api.github.com/repos/PBSPro/pbspro/issues/events{/number}","events_url":"https://api.github.com/repos/PBSPro/pbspro/events","assignees_url":"https://api.github.com/repos/PBSPro/pbspro/assignees{/user}","branches_url":"https://api.github.com/repos/PBSPro/pbspro/branches{/branch}","tags_url":"https://api.github.com/repos/PBSPro/pbspro/tags","blobs_url":"https://api.github.com/repos/PBSPro/pbspro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PBSPro/pbspro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PBSPro/pbspro/git/refs{/sha}","trees_url":"https://api.github.com/repos/PBSPro/pbspro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PBSPro/pbspro/statuses/{sha}","languages_url":"https://api.github.com/repos/PBSPro/pbspro/languages","stargazers_url":"https://api.github.com/repos/PBSPro/pbspro/stargazers","contributors_url":"https://api.github.com/repos/PBSPro/pbspro/contributors","subscribers_url":"https://api.github.com/repos/PBSPro/pbspro/subscribers","subscription_url":"https://api.github.com/repos/PBSPro/pbspro/subscription","commits_url":"https://api.github.com/repos/PBSPro/pbspro/commits{/sha}","git_commits_url":"https://api.github.com/repos/PBSPro/pbspro/git/commits{/sha}","comments_url":"https://api.github.com/repos/PBSPro/pbspro/comments{/number}","issue_comment_url":"https://api.github.com/repos/PBSPro/pbspro/issues/comments{/number}","contents_url":"https://api.github.com/repos/PBSPro/pbspro/contents/{+path}","compare_url":"https://api.github.com/repos/PBSPro/pbspro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PBSPro/pbspro/merges","archive_url":"https://api.github.com/repos/PBSPro/pbspro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PBSPro/pbspro/downloads","issues_url":"https://api.github.com/repos/PBSPro/pbspro/issues{/number}","pulls_url":"https://api.github.com/repos/PBSPro/pbspro/pulls{/number}","milestones_url":"https://api.github.com/repos/PBSPro/pbspro/milestones{/number}","notifications_url":"https://api.github.com/repos/PBSPro/pbspro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PBSPro/pbspro/labels{/name}","releases_url":"https://api.github.com/repos/PBSPro/pbspro/releases{/id}","deployments_url":"https://api.github.com/repos/PBSPro/pbspro/deployments","created_at":"2016-05-18T04:31:29Z","updated_at":"2018-08-24T18:15:03Z","pushed_at":"2018-08-24T18:18:37Z","git_url":"git://github.com/PBSPro/pbspro.git","ssh_url":"git@github.com:PBSPro/pbspro.git","clone_url":"https://github.com/PBSPro/pbspro.git","svn_url":"https://github.com/PBSPro/pbspro","homepage":"www.pbspro.org","size":15766,"stargazers_count":189,"watchers_count":189,"language":"C","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":148,"mirror_url":null,"archived":false,"open_issues_count":18,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":148,"open_issues":18,"watchers":189,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/780"},"html":{"href":"https://github.com/PBSPro/pbspro/pull/780"},"issue":{"href":"https://api.github.com/repos/PBSPro/pbspro/issues/780"},"comments":{"href":"https://api.github.com/repos/PBSPro/pbspro/issues/780/comments"},"review_comments":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/780/comments"},"review_comment":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PBSPro/pbspro/pulls/780/commits"},"statuses":{"href":"https://api.github.com/repos/PBSPro/pbspro/statuses/4242d0d2982f8395c415ac71ed10d952cf9e880c"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"arungrover"}}