{"action":"created","issue":{"url":"https://api.github.com/repos/bolt-design-system/bolt/issues/1008","repository_url":"https://api.github.com/repos/bolt-design-system/bolt","labels_url":"https://api.github.com/repos/bolt-design-system/bolt/issues/1008/labels{/name}","comments_url":"https://api.github.com/repos/bolt-design-system/bolt/issues/1008/comments","events_url":"https://api.github.com/repos/bolt-design-system/bolt/issues/1008/events","html_url":"https://github.com/bolt-design-system/bolt/pull/1008","id":394493818,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQxMjI4MTc3","number":1008,"title":"Refactor Build and CI Pipeline + Unify Pattern Lab and Docs Site","user":{"login":"sghoweri","id":1617209,"node_id":"MDQ6VXNlcjE2MTcyMDk=","avatar_url":"https://avatars2.githubusercontent.com/u/1617209?v=4","gravatar_id":"","url":"https://api.github.com/users/sghoweri","html_url":"https://github.com/sghoweri","followers_url":"https://api.github.com/users/sghoweri/followers","following_url":"https://api.github.com/users/sghoweri/following{/other_user}","gists_url":"https://api.github.com/users/sghoweri/gists{/gist_id}","starred_url":"https://api.github.com/users/sghoweri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sghoweri/subscriptions","organizations_url":"https://api.github.com/users/sghoweri/orgs","repos_url":"https://api.github.com/users/sghoweri/repos","events_url":"https://api.github.com/users/sghoweri/events{/privacy}","received_events_url":"https://api.github.com/users/sghoweri/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-12-27T21:43:23Z","updated_at":"2018-12-28T15:36:10Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/bolt-design-system/bolt/pulls/1008","html_url":"https://github.com/bolt-design-system/bolt/pull/1008","diff_url":"https://github.com/bolt-design-system/bolt/pull/1008.diff","patch_url":"https://github.com/bolt-design-system/bolt/pull/1008.patch"},"body":"## Jira\r\nTBD - need to grab several related Jira ticket links for this.\r\n\r\n## Summary\r\nSpeeds up Travis CI build process by around **300%** (from ~20s down to around 7s), in part by combining the two separate docs and Pattern Lab sites into one single Bolt-powered `docs-site`.\r\n\r\n## Why?\r\n1. We're going to have [25+ minute Travis builds](https://travis-ci.org/bolt-design-system/bolt/builds/471199042?utm_source=github_status&utm_medium=notification) if we don't make some big architectural changes before #990 is merged in.\r\n2. Virtually all other major improvements to Bolt (exploring Bedrock integration, Gatsby.js work, in-browser Component Explorer, server-side rendering, more comprehensive ) require a more unified dev + build + deploy setup (ie. one single source of truth).\r\n3. The previous `apps/pattern-lab` and `apps/bolt-site` setup, while very forward thinking a year ago when we first put it in place, doesn't quite solve some of the ongoing challenges of effectively maintaining our design system's docs, demos, examples, and tests since we still have two separate codebases (Pattern Lab and the Docs Site) pulling from the same unified component codebase (👍) but failed to figure out how to share internal resources and data between the two (👎). \r\n4. The separate `apps/bolt-site` vs `apps/pattern-lab` build processes mean that Pattern Lab gets used almost exclusively (despite it's shortcomings in some areas) resulting in the docs site receiving very little usage internally + very infrequent improvements.\r\n\r\n## How?\r\n1. Build and set things up as we go along (ex. composer installs don't happen when composer-related dependencies aren't needed).\r\n2. Cache as much as we can locally (via [incache](https://www.npmjs.com/package/incache), via Travis's internal cache, and via Webpack's Hard Source Plugin).\r\n3. Run parallel jobs on Travis (linting, testing, and building / deploying) are now 3 separate parallel jobs on Travis that run independently.\r\n4. Put the heaviest build tasks (image optimization) on a diet when we don't need the full build to run (ie. on feature branches only generate and optimize a couple different image sizes vs all of them).\r\n5. Don't run language-specific Webpack builds (Japanese) every single time on normal feature branches (effectively doubling the amount of time it takes Webpack to run).\r\n6. Don't run two separate Bolt build processes for two -- separate but almost identical -- websites. Instead, unify the two so only one build process needs to run.\r\n7. Simplify the npm scripts being run to make it easier to orchestrate all this. Individual tasks / dependencies try to manage their own dependencies (NPM postinstall-triggered `composer install`) so the main package.json file in Bolt is simplified and Travis tasks aren't held together by bubble gum and duct tape.\r\n\r\n## Details\r\n* 200 to 300% faster Travis CI builds\r\n    * Went from around 17 to 20+ minute builds on Travis to around 7 minutes for an entire Bolt build \r\n    * Went from Travis only running a single task at a time to running 3 separate kinds of checks: linting, smoke tests and the full build + deploy\r\n* Re-added Browsersync for handling local the local dev server; combined with Webpack Hot Middleware + Webpack Dev Middleware for much more consistent HTML, CSS and JS reloads\r\n* Updated async font loading approach by adding `font-display: swap` to our global @font-face rules\r\n* Unified image assets directory\r\n* Updated the Webpack hard source plugin to \r\n* Updated Travis CI cache to retain copy of node_modules, vendor, and cache folders to speed up build\r\n* One unified `.boltrc` config to manage the static docs site + Pattern Lab\r\n* Auto-install (certain) PHP dependencies based on usage\r\n    * Only applies to internally used dependencies — shouldn’t have an unexpected side effect on Drupal-specific dependencies\r\n* Partial image resize / compression to speed things up on Travis (especially on feature branches)\r\n* Unified Twig include for the HTML head and footer (exact same now used for PL and the docs site)\r\n- Added a new simple file cache for handling the bolt version data (pulled from Github) instead of always having to fetch the bolt version data every single time the build runs\r\n- Re-enables most of the previously disabled utility class demos in Pattern Lab (width utility class docs are still disabled due to \r\n- Addresses a persistent issue with the docs site not running the same Twig extensions as the Pattern Lab environment, resulting in things like the homepage `<bolt-link>` being rendered incorrectly.\r\n- Adjusts some of the attribute logic on the button component (internal vs external attributes) based on some additional jest tests put together testing out class name placement / duplication in some instances\r\n- Fixes a long-standing issue with the web fonts not working as expected (broken font path) when doing local development\r\n- Moved the Sassdoc page out of Pattern Lab and over to the docs section (wired up to pull in dynamic data)\r\n- Standardized and fixed the CSS selector prefix used for docs site-specific components and styles: `c-bds-` so component-specific styles vs internal design system-specific styles can be more easily differentiated\r\n- Updated Node to latest stable version (v10.x)\r\n- Updated build tool dependencies\r\n\r\n\r\n## Known Issues (Next PR)\r\n1. ~~Enabling full on schema validation (CLI mode) been been blowing some stuff up in PHP (can’t compile) — had to disable temporarily so work is needed to troubleshoot and fix.~~ \r\n>  Now working in PL, still had to temporarily disable schema validation on the docs site Twig renderer (separate 500 error)\r\n2. Width utility class demo still has a divide by zero issue when compiling (current issue on master — why most of the utility class demos are currently disabled)\r\n3. ~~Switch composer `setup` script back to `@composer install` instead of  `@composer update`~~\r\n4. ~~Test Re-adding Pattern Lab postinstall scripts to composer.json — curious if any Twig extensions being loaded are related~~\r\n5. ~~Temporarily disabled the Jest image test — missing ratio component in generated snapshot on sever (despite it working locally on the client)~~ fixed!\r\n6. ~~Images currently aren’t generating the image srcset markup expected despite the images themselves being generated.~~ fixed!\r\n7. Travis task that pushes to separate read only repos is also currently disabled — needs to be re-enabled before these updates can be fully pushed to production (shouldn't block this initial set of updates from getting merged however)\r\n8. No deploy-related testing tasks are enabled at the moment (ie. Nightwatch.js) given that these have been broken for many months now. I’m thinking we need to have something run AFTER the site deploys to now.sh so the deploy isn’t held up by any additional rounds of tests that are being run.\r\n9. Occasional `Error: Input buffer has corrupt header` error when attempting to run the full build locally.\r\n![image](https://user-images.githubusercontent.com/1617209/50515891-639a0380-0a75-11e9-9a60-f69a3b47b90e.png)\r\n\r\n\r\n## Questions\r\n* Can we remove a ton of these images? Are all of these actually needed + used currently? If not, separate clean up task to reduce the number of surplus images we have in the repo?\r\n* How / when do we do a full image resize task? A simple check to conditionally do the full image resize task on Git branches without a `feature-` in the name is currently in place however a more refined solution would be better. Ex. there’s no global config in place (yet) to do the full round of image optimizations vs just doing a subset.\r\n\r\n\r\n\r\n## What’s Next (Future PR)\r\n* Rename the `.boltrc` env config option from `pwa` to `hybrid` or something? My thought is that we should have a separate config option to allow us to enable PWAs via Workbox — separate from specifying if the build is being run for Drupal, Pattern Lab, a Static Site generator, etc.\r\n* Still additional work to be done with cleaning up, refactoring, (and when appropriate, removing) some of the helper and demo components that are used to support the docs site. The frontend code for all this was a huge mess (cleaned up a bunch though).\r\n* Need to update deploy script to reuse Github comments vs always add a new one every time\r\n* Disable in-line critical CSS and JS in some environments to speed things up?\r\n"},"comment":{"url":"https://api.github.com/repos/bolt-design-system/bolt/issues/comments/450378164","html_url":"https://github.com/bolt-design-system/bolt/pull/1008#issuecomment-450378164","issue_url":"https://api.github.com/repos/bolt-design-system/bolt/issues/1008","id":450378164,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDM3ODE2NA==","user":{"login":"bolt-bot","id":36777128,"node_id":"MDQ6VXNlcjM2Nzc3MTI4","avatar_url":"https://avatars0.githubusercontent.com/u/36777128?v=4","gravatar_id":"","url":"https://api.github.com/users/bolt-bot","html_url":"https://github.com/bolt-bot","followers_url":"https://api.github.com/users/bolt-bot/followers","following_url":"https://api.github.com/users/bolt-bot/following{/other_user}","gists_url":"https://api.github.com/users/bolt-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/bolt-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bolt-bot/subscriptions","organizations_url":"https://api.github.com/users/bolt-bot/orgs","repos_url":"https://api.github.com/users/bolt-bot/repos","events_url":"https://api.github.com/users/bolt-bot/events{/privacy}","received_events_url":"https://api.github.com/users/bolt-bot/received_events","type":"User","site_admin":false},"created_at":"2018-12-28T15:36:10Z","updated_at":"2018-12-28T15:36:10Z","author_association":"COLLABORATOR","body":":zap: PR built on Travis and deployed a now preview here:\n\n- Branch link: https://feature-refactor-build-and-ci-pipeline.boltdesignsystem.com\n- Permalink: https://boltdesignsystem-zxrvjnvswz.now.sh\n\n<details>\n\n- Commit built: cefa190a46802f00ba3dd73b986897c338eb5031\n- [Travis build](https://travis-ci.org/bolt-design-system/bolt/builds/473039219)\n\n</details>"}}
{"actor":{"display_login":"bolt-bot"}}