{"action":"created","issue":{"url":"https://api.github.com/repos/os-autoinst/openQA/issues/535","repository_url":"https://api.github.com/repos/os-autoinst/openQA","labels_url":"https://api.github.com/repos/os-autoinst/openQA/issues/535/labels{/name}","comments_url":"https://api.github.com/repos/os-autoinst/openQA/issues/535/comments","events_url":"https://api.github.com/repos/os-autoinst/openQA/issues/535/events","html_url":"https://github.com/os-autoinst/openQA/pull/535","id":131396247,"number":535,"title":"treat kernels / initrds as assets, allow download of all assets","user":{"login":"AdamWill","id":916551,"avatar_url":"https://avatars.githubusercontent.com/u/916551?v=3","gravatar_id":"","url":"https://api.github.com/users/AdamWill","html_url":"https://github.com/AdamWill","followers_url":"https://api.github.com/users/AdamWill/followers","following_url":"https://api.github.com/users/AdamWill/following{/other_user}","gists_url":"https://api.github.com/users/AdamWill/gists{/gist_id}","starred_url":"https://api.github.com/users/AdamWill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AdamWill/subscriptions","organizations_url":"https://api.github.com/users/AdamWill/orgs","repos_url":"https://api.github.com/users/AdamWill/repos","events_url":"https://api.github.com/users/AdamWill/events{/privacy}","received_events_url":"https://api.github.com/users/AdamWill/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2016-02-04T16:13:26Z","updated_at":"2016-02-17T19:43:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/os-autoinst/openQA/pulls/535","html_url":"https://github.com/os-autoinst/openQA/pull/535","diff_url":"https://github.com/os-autoinst/openQA/pull/535.diff","patch_url":"https://github.com/os-autoinst/openQA/pull/535.patch"},"body":"this is to make it possible to do 'remote scheduling' (where\r\nthe scheduler runs on some machine which does not have access\r\nto the openQA 'factory' directory) of jobs that boot using a\r\nkernel and/or initrd and/or hdd image, just as the 'ISOURL'\r\nfeature made it possible for jobs that boot using an ISO. I had\r\nto change a few other things to be happy with it, though.\r\n\r\nThe convention now is you can request openQA download for any\r\nasset type, by adding _URL to the appropriate setting name. To\r\ndownload the ISO set 'ISO_URL', to download a kernel set\r\n'KERNEL_URL', and so on. The POST will error out if you try to\r\ndownload something that is *not* an asset type. This is because\r\nwe can't really guess where we should store non-assets.\r\n\r\nAs with the old ISOURL code, if you set the normal setting\r\n(ISO, KERNEL etc.), that value will be used as the filename of\r\nthe downloaded file, otherwise the original filename will be\r\nsplit out from the URL and used.\r\n\r\nOf course, in order to achieve the original goal, I had to make\r\nkernels and initrds be treated as assets :) In garretraziel's\r\ninitial patch they were not. We have another good reason to\r\ntreat them as assets, though - it will allow us to clean them\r\nup with the limit_assets gru task (this patch does not change\r\nthat yet, but it's necessary to make it possible). For now we\r\ntreat kernels and initrds as 'other' assets, they could be\r\ngiven a new asset type instead if we think that's a good idea.\r\nHDDs of course already have an asset type.\r\n\r\nIt seemed clean to factor the 'figure out the asset type for\r\na given setting' code in parse_assets_from_settings, because we\r\nneed to do that same thing in job_schedule_iso now, to decide\r\nwhere the asset should be downloaded to (we need to know its\r\ntype). So this adds 'asset_type_from_setting' to Utils, and\r\nhas both parse_assets_from_settings and job_schedule_iso use\r\nit.\r\n\r\nIt's also necessary to define OTHER_DIR (like HDD_DIR and\r\nISO_DIR) in Common.pm so isotovideo can use it. The code for\r\nattaching the appropriate path to the KERNEL and INITRD values\r\nis almost identical to the code for ISO, but it's not easy to\r\nreuse unfortunately.\r\n\r\nFinally I added a bit more handling of potential error cases\r\nin download_asset (renamed from download_iso) to try and avoid\r\ntriggering the Eternal Gru Failure Loop Of Pain (it now checks\r\nif it can write to the target path, and catches errors when\r\nmoving the temporary file to the final location, since we found\r\nout that was possible)."},"comment":{"url":"https://api.github.com/repos/os-autoinst/openQA/issues/comments/185371660","html_url":"https://github.com/os-autoinst/openQA/pull/535#issuecomment-185371660","issue_url":"https://api.github.com/repos/os-autoinst/openQA/issues/535","id":185371660,"user":{"login":"okurz","id":1693432,"avatar_url":"https://avatars.githubusercontent.com/u/1693432?v=3","gravatar_id":"","url":"https://api.github.com/users/okurz","html_url":"https://github.com/okurz","followers_url":"https://api.github.com/users/okurz/followers","following_url":"https://api.github.com/users/okurz/following{/other_user}","gists_url":"https://api.github.com/users/okurz/gists{/gist_id}","starred_url":"https://api.github.com/users/okurz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/okurz/subscriptions","organizations_url":"https://api.github.com/users/okurz/orgs","repos_url":"https://api.github.com/users/okurz/repos","events_url":"https://api.github.com/users/okurz/events{/privacy}","received_events_url":"https://api.github.com/users/okurz/received_events","type":"User","site_admin":false},"created_at":"2016-02-17T19:43:14Z","updated_at":"2016-02-17T19:43:14Z","body":"I would have preferred tests write like\r\n```perl\r\n-# check we have no gru download tasks to start with\r\n-my @tasks = $schema->resultset(\"GruTasks\")->search({taskname => 'download_asset'});\r\n-ok(scalar @tasks == 0);\r\nis(scalar @tasks = $schema->resultset(\"GruTasks\")->search({taskname => 'download_asset'}), 0,\r\n    'check we have no gru download tasks to start with');\r\n```\r\n\r\nbut better @aaannz just merges this before the swearing continues ;-)"}}