{"action":"created","issue":{"url":"https://api.github.com/repos/t2ym/thin-hook/issues/266","repository_url":"https://api.github.com/repos/t2ym/thin-hook","labels_url":"https://api.github.com/repos/t2ym/thin-hook/issues/266/labels{/name}","comments_url":"https://api.github.com/repos/t2ym/thin-hook/issues/266/comments","events_url":"https://api.github.com/repos/t2ym/thin-hook/issues/266/events","html_url":"https://github.com/t2ym/thin-hook/issues/266","id":324639287,"number":266,"title":"Block access via automation like puppeteer","user":{"login":"t2ym","id":16043714,"avatar_url":"https://avatars0.githubusercontent.com/u/16043714?v=4","gravatar_id":"","url":"https://api.github.com/users/t2ym","html_url":"https://github.com/t2ym","followers_url":"https://api.github.com/users/t2ym/followers","following_url":"https://api.github.com/users/t2ym/following{/other_user}","gists_url":"https://api.github.com/users/t2ym/gists{/gist_id}","starred_url":"https://api.github.com/users/t2ym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t2ym/subscriptions","organizations_url":"https://api.github.com/users/t2ym/orgs","repos_url":"https://api.github.com/users/t2ym/repos","events_url":"https://api.github.com/users/t2ym/events{/privacy}","received_events_url":"https://api.github.com/users/t2ym/received_events","type":"User","site_admin":false},"labels":[{"id":812185646,"url":"https://api.github.com/repos/t2ym/thin-hook/labels/compatibility","name":"compatibility","color":"f4f066","default":false},{"id":524923563,"url":"https://api.github.com/repos/t2ym/thin-hook/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":793864017,"url":"https://api.github.com/repos/t2ym/thin-hook/labels/vulnerability","name":"vulnerability","color":"f92cf6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-19T15:51:06Z","updated_at":"2018-05-26T15:04:48Z","closed_at":null,"author_association":"OWNER","body":"Block access via automation like puppeteer\r\n\r\n## Current Status `thin-hook@0.1.13-stack.15` in [stack](https://github.com/t2ym/thin-hook/tree/stack) branch\r\n- NPM-published with `stack` tag \r\n  - `npm i thin-hook@0.1.13-stack.15` or `npm i thin-hook@stack`\r\n\r\n## Proof of Concept Implementation\r\n- [x] Hook native global object access\r\n  - [x] Load all mandatory no-hook scripts before the first `DomContentLoaded` event\r\n    - Insert this exact comment `<!-- end of mandatory no-hook scripts -->` to `original-index.html`\r\n    - Effects\r\n      - `caches` is inaccessible at the first `domcontentloaded` event\r\n      - Theoretically speaking, the main document is single-threaded and the script loading should not be interrupted by other evaluators like script evaluation requests from puppeteer.  Since the script evaluation is impossible at `Page.frameStartedLoading` event, there should be **HOPEFULLY** no chances of inspecting the main document until `domcontentloaded`.  It is not verified whether any events between `Page.frameStartedLoading` and `domcontentloaded` are fired or not.\r\n```js\r\n  // puppeteer client\r\n  page.on('domcontentloaded', async function onDomContentLoaded() {\r\n    console.log('domcontentloaded');\r\n    let result = await page.evaluate(function getCaches() {\r\n      return caches.constructor.name;\r\n    });\r\n    console.log('caches = ', result);\r\n  });\r\n```\r\n```\r\n[cacheBundleGeneration   ] goto https://localhost/components/thin-hook/demo/\r\n[cacheBundleGeneration   ] domcontentloaded\r\n[cacheBundleGeneration   ] (node:4362) UnhandledPromiseRejectionWarning: \r\n  Unhandled promise rejection (rejection id: 1): \r\n  Error: Evaluation failed: TypeError: Cannot read property 'constructor' of undefined\r\n[cacheBundleGeneration   ]     at getCaches (<anonymous>:2:21)\r\n```\r\n  - [ ] All getter/setter functions are wrapped\r\n    - [x] Native `window` object properties\r\n    - [x] `Object.prototype.constructor` (`=== Object`)\r\n    - [x] `EventTarget.prototype` object properties such as `addEventListener`\r\n    - [x] `Object.prototype` object properties such as `__lookupGetter__`\r\n    - [x] `__proto__`\r\n    - [x] Non-native `window` object properties\r\n    - [ ] Worker\r\n      - [ ] Dedicated Worker\r\n      - [ ] Shared Worker\r\n  - [x] Values are also wrapped via getter/setter functions\r\n- [ ] Block access\r\n  - [ ] return `undefined` for any global object access\r\n    - [x] return `undefined` for own native properties of `window`\r\n    - [x] return `undefined` for own non-native properties of `window`\r\n    - [x] return `undefined` for globals from no-hook APIs\r\n  - [ ] Handle exceptional objects (`hook`, `$hook$`, etc.) properly\r\n    - [x] Block access via `__hook__` by using Symbols for contexts\r\n      - About 25% slower\r\n  - [x] `about:blank` transition\r\n  - [ ] Clear `caches`\r\n- [ ] Handle whitelists\r\n  - [ ] Reduce overheads\r\n    - [x] Cache RegExp matching whitelist to full matching whitelist\r\n  - [x] Full matching whitelist\r\n  - [x] RegExp matching whitelist\r\n  - [x] Global object access not permitted by the whitelists is blocked\r\n- [ ] Handle async callbacks\r\n  - [ ] Handle async callbacks in no-hook scripts properly\r\n    - Currently, async callbacks in no-hook scripts are mistreated as unexpected global object access via automation\r\n      - Workaround: Add no-hook scripts to the whitelist\r\n- [ ] Allow `npm run cache-bundle` to fetch the bundle exclusively and securely\r\n  - [x] `cache-automation.js` to automate navigation for cache collection\r\n- [x] Eliminate global object access without contextStack in hooked scripts\r\n  - [x] Read access is always required to call a function\r\n```js\r\n// Original \r\nFunction('return 1');\r\n```\r\n```js\r\n// Hooked: old\r\n__hook__(Function, null, ['return 1'], 'context')\r\n```\r\n```js\r\n// Hooked: new (omitted)\r\n__hook__($hook$.global(Function)..., null, ['return 1'], 'context')\r\n```\r\n- [ ] Update tests\r\n  - [x] Add extra `$hook$.global(...)` in the expected hooked results\r\n  - [x] Symbols as contexts\r\n\r\n- [ ] Add puppeteer tests to `cacheBundleGeneration.js`\r\n  - [x] Read `caches`\r\n  - [x] Read `this.__proto__.__proto__.__proto__.__proto__.constructor` (`=== Object`)\r\n  - [x] Read non-native global property `Polymer`\r\n  - [x] Read `__lookupGetter__` originally from `Object.prototype`\r\n  - [x] Read `addEventListener` originally from `EventTarget.prototype`\r\n  - [x] Read `this.__proto__.__proto__.__proto__.__proto__.__lookupGetter__`\r\n  - [x] Read `Math.__lookupGetter__`\r\n  - [x] Read `Math.abs.__lookupGetter__`\r\n  - [x] Call `__hook__('.', this, ['navigator'], 'context')`\r\n  - [x] Call `__hook__('.', this, ['navigator'], Symbol.for('context'))`\r\n- [ ] Reduce Overheads\r\n  - [ ] Access global properties only once in `hook.min.js`\r\n    - [ ] `thin-hook/lib/*.js` scripts\r\n    - [ ] `t2ym/espree`\r\n    - [ ] `t2ym/escodegen`\r\n    - [ ] `acorn` (Forking required)\r\n    - [ ] Other components\r\n  - [ ] Optimize hooking performance for extra `$hook$.global()`\r\n- [ ] Issues\r\n  - [x] `event` is mistreated as a global object in `setAttribute('onXX', 'event.target')`\r\n    - Root Cause: `{ event: true }` is missing in the initial scope of hooking\r\n  - [x] Add `enableDebugging` flag in `hook-callback.js`\r\n  - [x] `Object.$__proto__$` is unexpectedly defined by `__hook__()`\r\n    - Root Cause: The statistics object `globalObjectAccess.constructor` unexpectedly points to `Object` while it is expected as a normal property of the `globalObjectAccess` object\r\n  - [x] `Object['/components/polymer/lib/mixins/property-accessors.html,script@741,props']` is unexpectedly defined by `__hook__()`\r\n    - Root Cause: same as above\r\n  - [x] Add comment \"Moved to hook-callback.js\" to `hook-native-api.js`\r\n  - [x] Unpredictable hook prefix\r\n    - Example: `hook-prefix=_uNpREdiC4aB1e_`\r\n  - [ ] `compact=false` is not supported by context symbols\r\n    - Root Cause: The first `]);` must be looked up instead of `']);`\r\n    - Note: Issue #257 must be avoided in fixing the issue\r\n\r\n## Notes\r\n- `npm run cache-bundle` fails as expected for now\r\n```\r\n[cacheBundleGeneration   ] (node:28710) UnhandledPromiseRejectionWarning: \r\n  Unhandled promise rejection (rejection id: 1): \r\n  Error: Evaluation failed: TypeError: Promise is not a constructor\r\n[cacheBundleGeneration   ]     at waitForBundleSetFetched (<anonymous>:2:12)\r\n```\r\n- `demo/cache-bundle.json` is generated via `?cache-bundle=save` option"},"comment":{"url":"https://api.github.com/repos/t2ym/thin-hook/issues/comments/392267178","html_url":"https://github.com/t2ym/thin-hook/issues/266#issuecomment-392267178","issue_url":"https://api.github.com/repos/t2ym/thin-hook/issues/266","id":392267178,"user":{"login":"t2ym","id":16043714,"avatar_url":"https://avatars0.githubusercontent.com/u/16043714?v=4","gravatar_id":"","url":"https://api.github.com/users/t2ym","html_url":"https://github.com/t2ym","followers_url":"https://api.github.com/users/t2ym/followers","following_url":"https://api.github.com/users/t2ym/following{/other_user}","gists_url":"https://api.github.com/users/t2ym/gists{/gist_id}","starred_url":"https://api.github.com/users/t2ym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t2ym/subscriptions","organizations_url":"https://api.github.com/users/t2ym/orgs","repos_url":"https://api.github.com/users/t2ym/repos","events_url":"https://api.github.com/users/t2ym/events{/privacy}","received_events_url":"https://api.github.com/users/t2ym/received_events","type":"User","site_admin":false},"created_at":"2018-05-26T15:04:48Z","updated_at":"2018-05-26T15:04:48Z","author_association":"OWNER","body":"## `cache-automation.js` to automate cache collection for `cache-bundle.json`\r\n- Put a special one-time `cache-bundle.json` containing `cache-automation.js` contents and `serverSecret`, which is a random hex value as below\r\n- If the one-time `serverSecret` is lost, the application cannot accept any automation scripts from a special `cache-bundle.json` since the matching `sha256(serverSecret + code) === authorization` is checked against `authorization` from `<script src=\"cache-bundle.js?no-hook=true&authorization={VALUE}\">`, which is embedded as a fixed value for the build.\r\n```javascript\r\nconst serverSecret = crypto.randomFillSync(Buffer.alloc(32)).toString('hex');\r\nconst cacheBundlePath = path.join('demo', 'cache-bundle.json');\r\nconst cacheAutomationScriptPath = path.join('demo', 'cache-automation.js');\r\nconst cacheAutomationScript = fs.readFileSync(cacheAutomationScriptPath, 'UTF-8');\r\nlet version = 'version_1';\r\nlet authorization; // sha256(serverSecret + cacheAutomationScript)\r\nlet hash = hook.utils.createHash('sha256');\r\nhash.update(serverSecret + cacheAutomationScript);\r\nauthorization = hash.digest('hex');\r\n\r\ngulp.task('get-version', (done) => {\r\n  return gulp.src(['demo/original-index.html'], { base: 'demo' })\r\n    .pipe(through.obj((file, enc, callback) => {\r\n      let html = String(file.contents);\r\n      let versionIndex = html.indexOf('/hook.min.js?version=') + '/hook.min.js?version='.length;\r\n      let versionIndexEnd = html.indexOf('&', versionIndex);\r\n      version = 'version_' + html.substring(versionIndex, versionIndexEnd);\r\n      callback(null, file);\r\n    }))\r\n    .pipe(through.obj((file, enc, callback) => {\r\n      done();\r\n    }));\r\n});\r\n\r\n// One-time special cache-bundle.json generation\r\ngulp.task('cache-bundle-automation-json', (done) => {\r\n  fs.writeFileSync(cacheBundlePath, JSON.stringify({\r\n    \"version\": version,\r\n    \"https://thin-hook.localhost.localdomain/automation.json\": JSON.stringify({\r\n      \"state\": \"init\", // update state in the script to perform operations including reloading\r\n      \"serverSecret\": serverSecret,\r\n      \"script\": cacheAutomationScript\r\n    },null,0)\r\n  },null,2))\r\n  done();\r\n});\r\n```\r\n- Embed `sha256(serverSecret + cacheAutomationScriptCode)` as `<script context-generator src=\"cache-bundle.js?no-hook=true&authorization={HERE}\"></script>`\r\n- In `cacheBundleGeneration.js`, wait for the global value `__{serverSecret}__` to obtain the raw cache-bundle JSON\r\n```javascript\r\n  let rawCacheBundleJSON;\r\n  while (!rawCacheBundleJSON) {\r\n    try {\r\n      rawCacheBundleJSON = await page.evaluate(new Function(`return async function cacheBundle() {\r\n        try {\r\n          return __${serverSecret}__;\r\n        }\r\n        catch (e) {\r\n          return [][0]; // undefined;\r\n        }\r\n      }`)());\r\n    }\r\n    catch (e) {\r\n      // try again\r\n      console.log(e.message);\r\n    }\r\n    await new Promise(resolve => setTimeout(resolve, 5000));\r\n  }\r\n  console.log('cacheBundle raw length = ', rawCacheBundleJSON.length, ' bytes');\r\n```\r\n- Example `cache-automation.js` can be customized for the target application\r\n```javascript\r\nasync function automationFunction() {\r\n  /*\r\n  @license https://github.com/t2ym/thin-hook/blob/master/LICENSE.md\r\n  Copyright (c) 2018, Tetsuya Mori <t2y3141592@gmail.com>. All rights reserved.\r\n  */\r\n  const CACHE_STATUS_PSEUDO_URL = 'https://thin-hook.localhost.localdomain/cache-status.json';\r\n  const AUTOMATION_PSEUDO_URL = 'https://thin-hook.localhost.localdomain/automation.json';\r\n  const PSEUDO_URL_PREFIX = 'https://thin-hook.localhost.localdomain/';\r\n  let automationStatus = JSON.parse(await (await (await caches.open(version)).match(AUTOMATION_PSEUDO_URL)).text());\r\n  if (automationStatus.state === 'init') {\r\n    let cache = await caches.open(version);\r\n    let keys = await cache.keys();\r\n    for (let key of keys) {\r\n      if (key.url.startsWith(PSEUDO_URL_PREFIX)) {\r\n        continue;\r\n      }\r\n      await cache.delete(key);\r\n    }\r\n    automationStatus.state = 'cleaned';\r\n    await (await caches.open(version)).put(new Request(AUTOMATION_PSEUDO_URL), new Response(JSON.stringify(automationStatus), { headers: { 'Content-Type': 'application/json' }}));\r\n    location.reload();\r\n  }\r\n\r\n  const timeoutForBundleSetFetched = 60000; // 60sec\r\n  // wait for bundle-set-fetched event\r\n  await new Promise((resolve, reject) => {\r\n    const start = Date.now();\r\n    let intervalId = setInterval(async () => {\r\n      const now = Date.now();\r\n      if (now - start > timeoutForBundleSetFetched) {\r\n        reject('timeout for bundle-set-fetched');\r\n      }\r\n      try {\r\n        let model = document.querySelector('live-localizer').shadowRoot\r\n          .getElementById('main').shadowRoot\r\n          .getElementById('dialog')\r\n          .querySelector('live-localizer-panel').shadowRoot\r\n          .getElementById('model');\r\n        if (model) {\r\n          clearInterval(intervalId);\r\n          // Note: bundle-set-fetched is the load completion event for live-localizer widget and irrelevant to cache-bundle.json\r\n          model.addEventListener('bundle-set-fetched', (event) => {\r\n            resolve(event.type);\r\n          });\r\n        }\r\n        else {\r\n          // try again\r\n        }\r\n      }\r\n      catch (e) {\r\n        // try again\r\n      }\r\n    }, 1000);\r\n  });\r\n\r\n  await new Promise(async (resolve, reject) => {\r\n    try {\r\n      let menuItems = document.querySelector('my-app').shadowRoot\r\n        .children[3] // app-drawer-layout\r\n        .querySelector('app-drawer')\r\n        .querySelector('iron-selector')\r\n        .querySelectorAll('a');\r\n      let result = [];\r\n      for (let i = menuItems.length - 1; i >= 0; i--) {\r\n        menuItems[i].click();\r\n        result.push(menuItems[i].href);\r\n        await new Promise(_resolve => {\r\n          setTimeout(_resolve, 20000); // Note: It is better to wait for specific events or conditions than just for a fixed period\r\n        });\r\n      }\r\n      resolve(result);\r\n    }\r\n    catch (e) {\r\n      reject(e.message);\r\n    }\r\n  });\r\n\r\n  let result = await (async function cacheBundle() {\r\n    const target = 'cache-bundle.json';\r\n    const cacheBundleURL = new URL(target, hook.parameters.baseURI);\r\n    const PSEUDO_URL_PREFIX = 'https://thin-hook.localhost.localdomain/';\r\n    const DEFAULT_VERSION = '1'\r\n    const version = 'version_' + (new URL(document.querySelector('script').src, location.href).searchParams.get('version') || DEFAULT_VERSION);\r\n    let cache = await caches.open(version);\r\n    let requests = await cache.keys();\r\n    let cacheBundle = { version: version };\r\n    let baseURI = hook.parameters.baseURI;\r\n    let origin = new URL(baseURI).origin;\r\n\r\n    for (let request of requests) {\r\n      if (request.url.startsWith(cacheBundleURL.href) || request.url.startsWith(PSEUDO_URL_PREFIX)) {\r\n        continue;\r\n      }\r\n      let response = await cache.match(request);\r\n      let text = await response.text();\r\n      let url = new URL(request.url, baseURI);\r\n      let key = origin === url.origin ? url.pathname : request.url;\r\n      cacheBundle[key] = text;\r\n    }\r\n    return JSON.stringify(cacheBundle, null, 0);\r\n  })();\r\n\r\n  automationStatus.state = 'done';\r\n  await (await caches.open(version)).put(new Request(AUTOMATION_PSEUDO_URL), new Response(JSON.stringify(automationStatus), { headers: { 'Content-Type': 'application/json' }}));\r\n\r\n  return result;\r\n}\r\n```"}}
{"actor":{"display_login":"t2ym"}}