{"action":"created","issue":{"url":"https://api.github.com/repos/xcat2/xcat-core/issues/1940","repository_url":"https://api.github.com/repos/xcat2/xcat-core","labels_url":"https://api.github.com/repos/xcat2/xcat-core/issues/1940/labels{/name}","comments_url":"https://api.github.com/repos/xcat2/xcat-core/issues/1940/comments","events_url":"https://api.github.com/repos/xcat2/xcat-core/issues/1940/events","html_url":"https://github.com/xcat2/xcat-core/issues/1940","id":181145064,"number":1940,"title":"running packimage prevents nodes from booting.","user":{"login":"ralphbellofatto","id":4139514,"avatar_url":"https://avatars.githubusercontent.com/u/4139514?v=3","gravatar_id":"","url":"https://api.github.com/users/ralphbellofatto","html_url":"https://github.com/ralphbellofatto","followers_url":"https://api.github.com/users/ralphbellofatto/followers","following_url":"https://api.github.com/users/ralphbellofatto/following{/other_user}","gists_url":"https://api.github.com/users/ralphbellofatto/gists{/gist_id}","starred_url":"https://api.github.com/users/ralphbellofatto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ralphbellofatto/subscriptions","organizations_url":"https://api.github.com/users/ralphbellofatto/orgs","repos_url":"https://api.github.com/users/ralphbellofatto/repos","events_url":"https://api.github.com/users/ralphbellofatto/events{/privacy}","received_events_url":"https://api.github.com/users/ralphbellofatto/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-05T12:56:05Z","updated_at":"2016-10-07T18:59:04Z","closed_at":null,"body":"If we run the following command, which builds a 10GB diskless boot image and there was a previous valid image in that place, we are unable to boot the nodes that point to this image until the packimage completes.\r\n```\r\npackimage rh72_op5_compute\r\n```\r\n\r\nthe node eventually boots up but reports this:\r\n```\r\n[Wed Oct  5 08:27:55 2016][   14.747784] dracut-initqueue[1470]: http://10.1.0.1//install/netboot/rhels7.2/ppc64le/rh72_op5_compute/rootimg.cpio.gz:                                                    \r\n[Wed Oct  5 08:27:55 2016][   14.748376] dracut-initqueue[1470]: 2016-10-05 12:27:49 ERROR 403: Forbidden.                                                                                              \r\n[Wed Oct  5 08:28:22 2016][   41.753150] dracut-initqueue[1470]: http://10.1.0.1//install/netboot/rhels7.2/ppc64le/rh72_op5_compute/rootimg.cpio.gz:                                                    \r\n[Wed Oct  5 08:28:22 2016][   41.753823] dracut-initqueue[1470]: 2016-10-05 12:28:16 ERROR 403: Forbidden.                                                                                              \r\n[Wed Oct  5 08:28:49 2016][   68.758139] dracut-initqueue[1470]: http://10.1.0.1//install/netboot/rhels7.2/ppc64le/rh72_op5_compute/rootimg.cpio.gz:                                                    \r\n[Wed Oct  5 08:28:49 2016][   68.758699] dracut-initqueue[1470]: 2016-10-05 12:28:43 ERROR 403: Forbidden.                                                                                              \r\n```\r\n\r\nAnd there is nothing in the xcat node status to indicate that this is the hold up.\r\n\r\nSince package takes about 10 minutes, this starts to cause operational problems when nodes are being rebooted.\r\n\r\nI'm not sure what the expected behavior should be here.  There are a couple of possibilites.\r\n\r\n- have pack image put the put the new compressed files into a temporary directory and when done, remove the original files and move the new files in its place.  This would have the advantage of cutting down the time that the booting node is waiting for the packimage to complete.\r\n- detect the condition and change the lsdef status and the nodestat results during the boot sequence to indicate a delay is in progress and why.\r\n\r\n- when performing the pack image, warn the operator that what they are doing will prevent nodes from booting if they are currently provisioned with that diskless image.  This might encourage using an a/b approach to image directories and properly direct the administrator to avoid using one that is currently in play.\r\n\r\nRight now, the system generates a bit of confusion, this is more more because of the time it takes to pack a large image and causes us to have higher admin costs due to the confusion.\r\n\r\n\r\n- "},"comment":{"url":"https://api.github.com/repos/xcat2/xcat-core/issues/comments/252333934","html_url":"https://github.com/xcat2/xcat-core/issues/1940#issuecomment-252333934","issue_url":"https://api.github.com/repos/xcat2/xcat-core/issues/1940","id":252333934,"user":{"login":"whowutwut","id":10049070,"avatar_url":"https://avatars.githubusercontent.com/u/10049070?v=3","gravatar_id":"","url":"https://api.github.com/users/whowutwut","html_url":"https://github.com/whowutwut","followers_url":"https://api.github.com/users/whowutwut/followers","following_url":"https://api.github.com/users/whowutwut/following{/other_user}","gists_url":"https://api.github.com/users/whowutwut/gists{/gist_id}","starred_url":"https://api.github.com/users/whowutwut/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whowutwut/subscriptions","organizations_url":"https://api.github.com/users/whowutwut/orgs","repos_url":"https://api.github.com/users/whowutwut/repos","events_url":"https://api.github.com/users/whowutwut/events{/privacy}","received_events_url":"https://api.github.com/users/whowutwut/received_events","type":"User","site_admin":false},"created_at":"2016-10-07T18:59:04Z","updated_at":"2016-10-07T18:59:04Z","body":"@ralphbellofatto  Yes, the error makes sense because it's the packed image that is going to be served to the diskless compute nodes and if you are writing it, it's not complete.  \r\n\r\nHowever, I do think that this something that happens in the development/test environment but I don't think this is something that would occur on a production system, because a osimage that is being used in production, should not be overwritten.  If the admin is creating a new one, it would be done as a different osimage or on another system to test/verify before moving into production. \r\n\r\nAlthough that being said,  I do like the first option that you recommended as that would reduce the chance of this happening, which is also good. "}}
{"actor":{"display_login":"whowutwut"}}