{"action":"closed","number":1576,"pull_request":{"url":"https://api.github.com/repos/LLNL/spack/pulls/1576","id":82222451,"html_url":"https://github.com/LLNL/spack/pull/1576","diff_url":"https://github.com/LLNL/spack/pull/1576.diff","patch_url":"https://github.com/LLNL/spack/pull/1576.patch","issue_url":"https://api.github.com/repos/LLNL/spack/issues/1576","number":1576,"state":"closed","locked":false,"title":"[WIP] Overhaul Spack's CI Infrastructure","user":{"login":"adamjstewart","id":12021217,"avatar_url":"https://avatars.githubusercontent.com/u/12021217?v=3","gravatar_id":"","url":"https://api.github.com/users/adamjstewart","html_url":"https://github.com/adamjstewart","followers_url":"https://api.github.com/users/adamjstewart/followers","following_url":"https://api.github.com/users/adamjstewart/following{/other_user}","gists_url":"https://api.github.com/users/adamjstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjstewart/subscriptions","organizations_url":"https://api.github.com/users/adamjstewart/orgs","repos_url":"https://api.github.com/users/adamjstewart/repos","events_url":"https://api.github.com/users/adamjstewart/events{/privacy}","received_events_url":"https://api.github.com/users/adamjstewart/received_events","type":"User","site_admin":false},"body":"I would like to overhaul Spack's CI infrastructure. Here is what I have in mind.\r\n\r\n## Only test what has changed\r\n\r\nAs it is now, if I submit a PR that fixes a typo in the documentation, Travis runs `spack test`. Obviously, a documentation change can't possibly break Spack, so this test is unnecessary. If I submit a PR that adds a new package, we also run `spack test`. This test won't catch anything more than whether or not the file can be parsed by Spack. We can do better.\r\n\r\nI propose that we divide our tests into 4 test suites: packages, core, docs, and flake8. If you have any more testing groups you would like to see, let me know.\r\n\r\n### Package tests\r\n\r\nIf the only files that have changed are in `var/spack/repos/builtin/packages`, running `spack test` doesn't do us any good. We need package-specific testing. What I'm thinking is for every package.py file that is modified, or at least the leaf nodes, run a set of Spack commands:\r\n\r\n- spack info\r\n  - prevents problems like #1503\r\n- spack versions\r\n  - doesn't work for most packages, but we should fix that\r\n- spack checksum\r\n  - same problem as above\r\n- spack fetch\r\n  - try fetching every single version to make sure they are still available\r\n  - checks that url_for_version works for every version\r\n- spack patch\r\n  - try patching every available version\r\n  - checks that added patches actually work for all versions specified\r\n  - checks that new versions added to a package still work with the patch\r\n- spack graph --concretize\r\n  - currently breaks for any package of considerable size, see #1098 \r\n- spack spec\r\n  - run for every possible set of variants\r\n  - catches problems like #1552 \r\n- spack env\r\n- spack install --run-tests\r\n- spack activate\r\n  - only run for extensions\r\n  - checks whether or not files conflict\r\n\r\nNote that many of these tests will need to be skipped for packages that cannot be fetched from a URL. Also keep in mind that many of these tests would fail if we ran them on current packages. Sometimes this is fine, but sometimes it could be easy to fix. By actually testing installations through Travis, we can see if there are any missing dependencies that aren't found on a bare-bones Linux system.\r\n\r\n### Core tests\r\n\r\nOnly run `spack test` if core Spack libraries have been modified. This is defined as files within:\r\n\r\n- bin\r\n- etc\r\n- lib\r\n  - but exclude lib/spack/docs, see below\r\n- share\r\n  - does `spack test` currently cover this stuff?\r\n  - may want to add a new test suite specifically for this\r\n    - could source `setup-env.sh` and `setup-env.csh`\r\n\r\nObviously `spack test` isn't perfect, and there are many unit tests that could be added. This PR **will not** cover these. It will only cover when `spack test` is run.\r\n\r\n### Documentation tests\r\n\r\nIf you try building Spack's documentation, you'll notice that there are several warning and error messages. These messages should cause the test to fail. This prevents problems like #1140 (oops, my fault). We want developers to add documentation as frequently as possible, even if they don't know how to test their changes or aren't very familiar with reStructuredText.\r\n\r\nI'm not very familiar with all of the `make` options we use, but is there any target I should try building aside from the default?\r\n\r\nAs a part of this PR, I will attempt to resolve all current warning and error messages, so as not to repeat past mistakes with flake8 integration.\r\n\r\nThere isn't any way to automatically deploy new documentation to http://software.llnl.gov/spack/index.html when a documentation changing PR is accepted, is there??? This would fix an old issue, #892.\r\n\r\n~~Also, is there a way to make all Sphinx build warnings fatal? I wouldn't make it fatal permanently, just during testing.~~ You can use `make SPHINXOPTS=-W` to treat all warnings as fatal errors.\r\n\r\n### Flake8 tests\r\n\r\nI think these tests are pretty solid at this point. I may integrate #1510 into this PR.\r\n\r\n## macOS testing\r\n\r\nShould we start testing on osx? There is only one major hurdle that I see. [Travis does not currently support Python on osx](https://docs.travis-ci.com/user/languages/python/). I think it might be a virtual environment problem? Not sure. Anyway, there are [workarounds](https://docs.travis-ci.com/user/multi-os/#Python-example-(unsupported-languages)) that could allow us to run on osx anyway. We can also [allow failures](https://docs.travis-ci.com/user/multi-os/#Allowing-Failures-on-Jobs-Running-on-One-Operating-System) for osx.\r\n\r\n## Progress\r\n\r\n- [x] Write generic script for determining what files were modified\r\n- [x] Write generic script for determining if all dependencies are present\r\n- [x] Only run unit tests when core Spack framework is modified\r\n- [x] Add documentation testing\r\n- [x] Resolve current documentation warnings and errors\r\n- [ ] Add package-specific testing\r\n- [ ] Automatically update Spack's official documentation after a doc PR is merged\r\n- [ ] Update documentation in Developer Guide to reflect new CI tests\r\n\r\n@tgamblin @alalazo @citibeth @becker33 @eschnett @davydden ","created_at":"2016-08-22T16:32:45Z","updated_at":"2016-08-31T03:43:45Z","closed_at":"2016-08-31T03:43:45Z","merged_at":"2016-08-31T03:43:45Z","merge_commit_sha":"fc748eb3d06db5fd6d72c7b1f9247fe9c05b900e","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/LLNL/spack/pulls/1576/commits","review_comments_url":"https://api.github.com/repos/LLNL/spack/pulls/1576/comments","review_comment_url":"https://api.github.com/repos/LLNL/spack/pulls/comments{/number}","comments_url":"https://api.github.com/repos/LLNL/spack/issues/1576/comments","statuses_url":"https://api.github.com/repos/LLNL/spack/statuses/fc748eb3d06db5fd6d72c7b1f9247fe9c05b900e","head":{"label":"adamjstewart:features/travis","ref":"features/travis","sha":"fc748eb3d06db5fd6d72c7b1f9247fe9c05b900e","user":{"login":"adamjstewart","id":12021217,"avatar_url":"https://avatars.githubusercontent.com/u/12021217?v=3","gravatar_id":"","url":"https://api.github.com/users/adamjstewart","html_url":"https://github.com/adamjstewart","followers_url":"https://api.github.com/users/adamjstewart/followers","following_url":"https://api.github.com/users/adamjstewart/following{/other_user}","gists_url":"https://api.github.com/users/adamjstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjstewart/subscriptions","organizations_url":"https://api.github.com/users/adamjstewart/orgs","repos_url":"https://api.github.com/users/adamjstewart/repos","events_url":"https://api.github.com/users/adamjstewart/events{/privacy}","received_events_url":"https://api.github.com/users/adamjstewart/received_events","type":"User","site_admin":false},"repo":{"id":49444175,"name":"spack","full_name":"adamjstewart/spack","owner":{"login":"adamjstewart","id":12021217,"avatar_url":"https://avatars.githubusercontent.com/u/12021217?v=3","gravatar_id":"","url":"https://api.github.com/users/adamjstewart","html_url":"https://github.com/adamjstewart","followers_url":"https://api.github.com/users/adamjstewart/followers","following_url":"https://api.github.com/users/adamjstewart/following{/other_user}","gists_url":"https://api.github.com/users/adamjstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjstewart/subscriptions","organizations_url":"https://api.github.com/users/adamjstewart/orgs","repos_url":"https://api.github.com/users/adamjstewart/repos","events_url":"https://api.github.com/users/adamjstewart/events{/privacy}","received_events_url":"https://api.github.com/users/adamjstewart/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/adamjstewart/spack","description":"A flexible package manager designed to support multiple versions, configurations, platforms, and compilers.","fork":true,"url":"https://api.github.com/repos/adamjstewart/spack","forks_url":"https://api.github.com/repos/adamjstewart/spack/forks","keys_url":"https://api.github.com/repos/adamjstewart/spack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/adamjstewart/spack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/adamjstewart/spack/teams","hooks_url":"https://api.github.com/repos/adamjstewart/spack/hooks","issue_events_url":"https://api.github.com/repos/adamjstewart/spack/issues/events{/number}","events_url":"https://api.github.com/repos/adamjstewart/spack/events","assignees_url":"https://api.github.com/repos/adamjstewart/spack/assignees{/user}","branches_url":"https://api.github.com/repos/adamjstewart/spack/branches{/branch}","tags_url":"https://api.github.com/repos/adamjstewart/spack/tags","blobs_url":"https://api.github.com/repos/adamjstewart/spack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/adamjstewart/spack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/adamjstewart/spack/git/refs{/sha}","trees_url":"https://api.github.com/repos/adamjstewart/spack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/adamjstewart/spack/statuses/{sha}","languages_url":"https://api.github.com/repos/adamjstewart/spack/languages","stargazers_url":"https://api.github.com/repos/adamjstewart/spack/stargazers","contributors_url":"https://api.github.com/repos/adamjstewart/spack/contributors","subscribers_url":"https://api.github.com/repos/adamjstewart/spack/subscribers","subscription_url":"https://api.github.com/repos/adamjstewart/spack/subscription","commits_url":"https://api.github.com/repos/adamjstewart/spack/commits{/sha}","git_commits_url":"https://api.github.com/repos/adamjstewart/spack/git/commits{/sha}","comments_url":"https://api.github.com/repos/adamjstewart/spack/comments{/number}","issue_comment_url":"https://api.github.com/repos/adamjstewart/spack/issues/comments{/number}","contents_url":"https://api.github.com/repos/adamjstewart/spack/contents/{+path}","compare_url":"https://api.github.com/repos/adamjstewart/spack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/adamjstewart/spack/merges","archive_url":"https://api.github.com/repos/adamjstewart/spack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/adamjstewart/spack/downloads","issues_url":"https://api.github.com/repos/adamjstewart/spack/issues{/number}","pulls_url":"https://api.github.com/repos/adamjstewart/spack/pulls{/number}","milestones_url":"https://api.github.com/repos/adamjstewart/spack/milestones{/number}","notifications_url":"https://api.github.com/repos/adamjstewart/spack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/adamjstewart/spack/labels{/name}","releases_url":"https://api.github.com/repos/adamjstewart/spack/releases{/id}","deployments_url":"https://api.github.com/repos/adamjstewart/spack/deployments","created_at":"2016-01-11T18:04:18Z","updated_at":"2016-01-11T18:04:19Z","pushed_at":"2016-08-30T21:38:51Z","git_url":"git://github.com/adamjstewart/spack.git","ssh_url":"git@github.com:adamjstewart/spack.git","clone_url":"https://github.com/adamjstewart/spack.git","svn_url":"https://github.com/adamjstewart/spack","homepage":"http://llnl.github.io/spack","size":9746,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"LLNL:develop","ref":"develop","sha":"7e4c6afd9160beedf7956bc5492c93eb30f93a3e","user":{"login":"LLNL","id":5921419,"avatar_url":"https://avatars.githubusercontent.com/u/5921419?v=3","gravatar_id":"","url":"https://api.github.com/users/LLNL","html_url":"https://github.com/LLNL","followers_url":"https://api.github.com/users/LLNL/followers","following_url":"https://api.github.com/users/LLNL/following{/other_user}","gists_url":"https://api.github.com/users/LLNL/gists{/gist_id}","starred_url":"https://api.github.com/users/LLNL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLNL/subscriptions","organizations_url":"https://api.github.com/users/LLNL/orgs","repos_url":"https://api.github.com/users/LLNL/repos","events_url":"https://api.github.com/users/LLNL/events{/privacy}","received_events_url":"https://api.github.com/users/LLNL/received_events","type":"Organization","site_admin":false},"repo":{"id":15730865,"name":"spack","full_name":"LLNL/spack","owner":{"login":"LLNL","id":5921419,"avatar_url":"https://avatars.githubusercontent.com/u/5921419?v=3","gravatar_id":"","url":"https://api.github.com/users/LLNL","html_url":"https://github.com/LLNL","followers_url":"https://api.github.com/users/LLNL/followers","following_url":"https://api.github.com/users/LLNL/following{/other_user}","gists_url":"https://api.github.com/users/LLNL/gists{/gist_id}","starred_url":"https://api.github.com/users/LLNL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLNL/subscriptions","organizations_url":"https://api.github.com/users/LLNL/orgs","repos_url":"https://api.github.com/users/LLNL/repos","events_url":"https://api.github.com/users/LLNL/events{/privacy}","received_events_url":"https://api.github.com/users/LLNL/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/LLNL/spack","description":"A flexible package manager designed to support multiple versions, configurations, platforms, and compilers.","fork":false,"url":"https://api.github.com/repos/LLNL/spack","forks_url":"https://api.github.com/repos/LLNL/spack/forks","keys_url":"https://api.github.com/repos/LLNL/spack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/LLNL/spack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/LLNL/spack/teams","hooks_url":"https://api.github.com/repos/LLNL/spack/hooks","issue_events_url":"https://api.github.com/repos/LLNL/spack/issues/events{/number}","events_url":"https://api.github.com/repos/LLNL/spack/events","assignees_url":"https://api.github.com/repos/LLNL/spack/assignees{/user}","branches_url":"https://api.github.com/repos/LLNL/spack/branches{/branch}","tags_url":"https://api.github.com/repos/LLNL/spack/tags","blobs_url":"https://api.github.com/repos/LLNL/spack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/LLNL/spack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/LLNL/spack/git/refs{/sha}","trees_url":"https://api.github.com/repos/LLNL/spack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/LLNL/spack/statuses/{sha}","languages_url":"https://api.github.com/repos/LLNL/spack/languages","stargazers_url":"https://api.github.com/repos/LLNL/spack/stargazers","contributors_url":"https://api.github.com/repos/LLNL/spack/contributors","subscribers_url":"https://api.github.com/repos/LLNL/spack/subscribers","subscription_url":"https://api.github.com/repos/LLNL/spack/subscription","commits_url":"https://api.github.com/repos/LLNL/spack/commits{/sha}","git_commits_url":"https://api.github.com/repos/LLNL/spack/git/commits{/sha}","comments_url":"https://api.github.com/repos/LLNL/spack/comments{/number}","issue_comment_url":"https://api.github.com/repos/LLNL/spack/issues/comments{/number}","contents_url":"https://api.github.com/repos/LLNL/spack/contents/{+path}","compare_url":"https://api.github.com/repos/LLNL/spack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/LLNL/spack/merges","archive_url":"https://api.github.com/repos/LLNL/spack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/LLNL/spack/downloads","issues_url":"https://api.github.com/repos/LLNL/spack/issues{/number}","pulls_url":"https://api.github.com/repos/LLNL/spack/pulls{/number}","milestones_url":"https://api.github.com/repos/LLNL/spack/milestones{/number}","notifications_url":"https://api.github.com/repos/LLNL/spack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/LLNL/spack/labels{/name}","releases_url":"https://api.github.com/repos/LLNL/spack/releases{/id}","deployments_url":"https://api.github.com/repos/LLNL/spack/deployments","created_at":"2014-01-08T09:22:12Z","updated_at":"2016-08-30T23:08:01Z","pushed_at":"2016-08-31T03:43:38Z","git_url":"git://github.com/LLNL/spack.git","ssh_url":"git@github.com:LLNL/spack.git","clone_url":"https://github.com/LLNL/spack.git","svn_url":"https://github.com/LLNL/spack","homepage":"http://software.llnl.gov/spack","size":18946,"stargazers_count":149,"watchers_count":149,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":133,"mirror_url":null,"open_issues_count":334,"forks":133,"open_issues":334,"watchers":149,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/LLNL/spack/pulls/1576"},"html":{"href":"https://github.com/LLNL/spack/pull/1576"},"issue":{"href":"https://api.github.com/repos/LLNL/spack/issues/1576"},"comments":{"href":"https://api.github.com/repos/LLNL/spack/issues/1576/comments"},"review_comments":{"href":"https://api.github.com/repos/LLNL/spack/pulls/1576/comments"},"review_comment":{"href":"https://api.github.com/repos/LLNL/spack/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/LLNL/spack/pulls/1576/commits"},"statuses":{"href":"https://api.github.com/repos/LLNL/spack/statuses/fc748eb3d06db5fd6d72c7b1f9247fe9c05b900e"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"tgamblin","id":299842,"avatar_url":"https://avatars.githubusercontent.com/u/299842?v=3","gravatar_id":"","url":"https://api.github.com/users/tgamblin","html_url":"https://github.com/tgamblin","followers_url":"https://api.github.com/users/tgamblin/followers","following_url":"https://api.github.com/users/tgamblin/following{/other_user}","gists_url":"https://api.github.com/users/tgamblin/gists{/gist_id}","starred_url":"https://api.github.com/users/tgamblin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgamblin/subscriptions","organizations_url":"https://api.github.com/users/tgamblin/orgs","repos_url":"https://api.github.com/users/tgamblin/repos","events_url":"https://api.github.com/users/tgamblin/events{/privacy}","received_events_url":"https://api.github.com/users/tgamblin/received_events","type":"User","site_admin":false},"comments":49,"review_comments":10,"commits":27,"additions":2067,"deletions":1783,"changed_files":79}}
{"actor":{"display_login":"tgamblin"}}