{"action":"created","issue":{"url":"https://api.github.com/repos/ipfs/jenkins/issues/93","repository_url":"https://api.github.com/repos/ipfs/jenkins","labels_url":"https://api.github.com/repos/ipfs/jenkins/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/ipfs/jenkins/issues/93/comments","events_url":"https://api.github.com/repos/ipfs/jenkins/issues/93/events","html_url":"https://github.com/ipfs/jenkins/issues/93","id":301001416,"node_id":"MDU6SXNzdWUzMDEwMDE0MTY=","number":93,"title":"Parallize aegir scripts","user":{"login":"VictorBjelkholm","id":459764,"node_id":"MDQ6VXNlcjQ1OTc2NA==","avatar_url":"https://avatars2.githubusercontent.com/u/459764?v=4","gravatar_id":"","url":"https://api.github.com/users/VictorBjelkholm","html_url":"https://github.com/VictorBjelkholm","followers_url":"https://api.github.com/users/VictorBjelkholm/followers","following_url":"https://api.github.com/users/VictorBjelkholm/following{/other_user}","gists_url":"https://api.github.com/users/VictorBjelkholm/gists{/gist_id}","starred_url":"https://api.github.com/users/VictorBjelkholm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VictorBjelkholm/subscriptions","organizations_url":"https://api.github.com/users/VictorBjelkholm/orgs","repos_url":"https://api.github.com/users/VictorBjelkholm/repos","events_url":"https://api.github.com/users/VictorBjelkholm/events{/privacy}","received_events_url":"https://api.github.com/users/VictorBjelkholm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"VictorBjelkholm","id":459764,"node_id":"MDQ6VXNlcjQ1OTc2NA==","avatar_url":"https://avatars2.githubusercontent.com/u/459764?v=4","gravatar_id":"","url":"https://api.github.com/users/VictorBjelkholm","html_url":"https://github.com/VictorBjelkholm","followers_url":"https://api.github.com/users/VictorBjelkholm/followers","following_url":"https://api.github.com/users/VictorBjelkholm/following{/other_user}","gists_url":"https://api.github.com/users/VictorBjelkholm/gists{/gist_id}","starred_url":"https://api.github.com/users/VictorBjelkholm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VictorBjelkholm/subscriptions","organizations_url":"https://api.github.com/users/VictorBjelkholm/orgs","repos_url":"https://api.github.com/users/VictorBjelkholm/repos","events_url":"https://api.github.com/users/VictorBjelkholm/events{/privacy}","received_events_url":"https://api.github.com/users/VictorBjelkholm/received_events","type":"User","site_admin":false},"assignees":[{"login":"VictorBjelkholm","id":459764,"node_id":"MDQ6VXNlcjQ1OTc2NA==","avatar_url":"https://avatars2.githubusercontent.com/u/459764?v=4","gravatar_id":"","url":"https://api.github.com/users/VictorBjelkholm","html_url":"https://github.com/VictorBjelkholm","followers_url":"https://api.github.com/users/VictorBjelkholm/followers","following_url":"https://api.github.com/users/VictorBjelkholm/following{/other_user}","gists_url":"https://api.github.com/users/VictorBjelkholm/gists{/gist_id}","starred_url":"https://api.github.com/users/VictorBjelkholm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VictorBjelkholm/subscriptions","organizations_url":"https://api.github.com/users/VictorBjelkholm/orgs","repos_url":"https://api.github.com/users/VictorBjelkholm/repos","events_url":"https://api.github.com/users/VictorBjelkholm/events{/privacy}","received_events_url":"https://api.github.com/users/VictorBjelkholm/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-02-28T12:12:46Z","updated_at":"2018-05-31T04:40:22Z","closed_at":null,"author_association":"MEMBER","body":"I made a quick test to see if it's viable to parallize the different aegir scripts (branch here: https://github.com/ipfs/jenkins-libs/blob/parallize/vars/javascript.groovy). Reason why is because ipfs/js-ipfs jobs takes ~20 minutes from start to finish. The tests in ipfs/js-ipfs first runs nodejs tests, then browser and after that webworkers. Experiment was to run build once for each platform + nodejs version, then use that build for use for each one of the tests + os + nodejs version.\r\n\r\n**Conclusion**: doesn't actually speed up builds significantly\r\n\r\n- because we need one job per os + per nodejs version + aegir command, we end with 12 jobs for running tests\r\n- this seems to slow down the jenkins pipeline because `stash/unstash` (which we use to pass around files) is not working well with large directories, making each job spending ~2 minutes just transfering files first\r\n- since tests are not isolated, we can only run one job per worker, making this parallization saturating the queue. We have 5 workers of each OS, which a parallel build of aegir scripts would require at least 6 of each OS (if we run one job). The queue gets full from just one test run\r\n- when running 12 jobs at the same time in one stage, the reporting back to the master node is delayed, leading to jobs finishing in 2 minutes, actually not finishing until 5 minutes after finishing\r\n- specific to ipfs/js-ipfs, `npm run test:node` is still the slowest one and slows down the complete reporting after pipeline finished. "},"comment":{"url":"https://api.github.com/repos/ipfs/jenkins/issues/comments/393400025","html_url":"https://github.com/ipfs/jenkins/issues/93#issuecomment-393400025","issue_url":"https://api.github.com/repos/ipfs/jenkins/issues/93","id":393400025,"node_id":"MDEyOklzc3VlQ29tbWVudDM5MzQwMDAyNQ==","user":{"login":"dryajov","id":1094341,"node_id":"MDQ6VXNlcjEwOTQzNDE=","avatar_url":"https://avatars1.githubusercontent.com/u/1094341?v=4","gravatar_id":"","url":"https://api.github.com/users/dryajov","html_url":"https://github.com/dryajov","followers_url":"https://api.github.com/users/dryajov/followers","following_url":"https://api.github.com/users/dryajov/following{/other_user}","gists_url":"https://api.github.com/users/dryajov/gists{/gist_id}","starred_url":"https://api.github.com/users/dryajov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dryajov/subscriptions","organizations_url":"https://api.github.com/users/dryajov/orgs","repos_url":"https://api.github.com/users/dryajov/repos","events_url":"https://api.github.com/users/dryajov/events{/privacy}","received_events_url":"https://api.github.com/users/dryajov/received_events","type":"User","site_admin":false},"created_at":"2018-05-31T04:40:22Z","updated_at":"2018-05-31T04:40:22Z","author_association":"MEMBER","body":"> Aegir should possibly have two hooks, one for the browser and another for webworker. For js-ipfs itself we can either start two different ipfsd-ctl servers, or share a single instance and keep a ref count.\r\n\r\nAgree, we need to start two ipfsd-ctl server if we want parallel browser and webworker runs."}}
{"actor":{"display_login":"dryajov"}}