{"action":"created","issue":{"url":"https://api.github.com/repos/web-platform-tests/wpt/issues/11388","repository_url":"https://api.github.com/repos/web-platform-tests/wpt","labels_url":"https://api.github.com/repos/web-platform-tests/wpt/issues/11388/labels{/name}","comments_url":"https://api.github.com/repos/web-platform-tests/wpt/issues/11388/comments","events_url":"https://api.github.com/repos/web-platform-tests/wpt/issues/11388/events","html_url":"https://github.com/web-platform-tests/wpt/issues/11388","id":329969677,"node_id":"MDU6SXNzdWUzMjk5Njk2Nzc=","number":11388,"title":"Improve caching for manifest and lint","user":{"login":"gsnedders","id":176218,"node_id":"MDQ6VXNlcjE3NjIxOA==","avatar_url":"https://avatars2.githubusercontent.com/u/176218?v=4","gravatar_id":"","url":"https://api.github.com/users/gsnedders","html_url":"https://github.com/gsnedders","followers_url":"https://api.github.com/users/gsnedders/followers","following_url":"https://api.github.com/users/gsnedders/following{/other_user}","gists_url":"https://api.github.com/users/gsnedders/gists{/gist_id}","starred_url":"https://api.github.com/users/gsnedders/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gsnedders/subscriptions","organizations_url":"https://api.github.com/users/gsnedders/orgs","repos_url":"https://api.github.com/users/gsnedders/repos","events_url":"https://api.github.com/users/gsnedders/events{/privacy}","received_events_url":"https://api.github.com/users/gsnedders/received_events","type":"User","site_admin":false},"labels":[{"id":45230790,"node_id":"MDU6TGFiZWw0NTIzMDc5MA==","url":"https://api.github.com/repos/web-platform-tests/wpt/labels/infra","name":"infra","color":"fbca04","default":false},{"id":693964558,"node_id":"MDU6TGFiZWw2OTM5NjQ1NTg=","url":"https://api.github.com/repos/web-platform-tests/wpt/labels/priority:backlog","name":"priority:backlog","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-06-06T17:50:25Z","updated_at":"2018-06-08T17:07:29Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Split out from https://github.com/web-platform-tests/wpt/issues/11356\r\n\r\nOne thing I've wanted to do for a long time is to massively improve caching for manifest and lint.\r\n\r\nThe status quo for manifest is we store a sha1 hash of the contents of each file and then compare it with a cached copy; this is expensive given it means we have to read every file to check if the cache is valid (this is especially notable on Windows due to https://github.com/web-platform-tests/wpt/issues/11386).\r\n\r\nFor lint we do all kinds of crazy magic to try and lint only a subset of files, based on git history.\r\n\r\nWe should instead:\r\n\r\n * use git blob IDs (in combination with the path where needed)\r\n * use file's mtime\r\n * maybe use hg node ids (or just the last revision, given I don't think hg has any public API or file format stability guarantees) for the sake of mozilla-central?\r\n\r\nWe need some combination of the above, given different things make sense in different cases (most obviously with untracked files but also the mozilla-central case).\r\n\r\nWe should absolutely be able to make manifest take almost no time at all when we have a cache (because we just need to verify that the tree still has the same blobs at the same paths) and make it possible for lint to always lint the entire tree mostly returning cached results (similarly in almost no time), though the simplest implementation would still to run all the `all_files_lints`.\r\n\r\nObviously any mtime-based caching would be filesystem specific, and hence not sharable. I'd like to suggest we add a local cache file, and make it optional to add a built-in cache to the generated manifest for the shared case. The local cache file could also easily store cached values for multiple revisions of a file, which would be useful."},"comment":{"url":"https://api.github.com/repos/web-platform-tests/wpt/issues/comments/395825264","html_url":"https://github.com/web-platform-tests/wpt/issues/11388#issuecomment-395825264","issue_url":"https://api.github.com/repos/web-platform-tests/wpt/issues/11388","id":395825264,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTgyNTI2NA==","user":{"login":"jgraham","id":294864,"node_id":"MDQ6VXNlcjI5NDg2NA==","avatar_url":"https://avatars1.githubusercontent.com/u/294864?v=4","gravatar_id":"","url":"https://api.github.com/users/jgraham","html_url":"https://github.com/jgraham","followers_url":"https://api.github.com/users/jgraham/followers","following_url":"https://api.github.com/users/jgraham/following{/other_user}","gists_url":"https://api.github.com/users/jgraham/gists{/gist_id}","starred_url":"https://api.github.com/users/jgraham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgraham/subscriptions","organizations_url":"https://api.github.com/users/jgraham/orgs","repos_url":"https://api.github.com/users/jgraham/repos","events_url":"https://api.github.com/users/jgraham/events{/privacy}","received_events_url":"https://api.github.com/users/jgraham/received_events","type":"User","site_admin":false},"created_at":"2018-06-08T17:07:29Z","updated_at":"2018-06-08T17:07:29Z","author_association":"CONTRIBUTOR","body":"> Is it acceptable in the Gecko case that the manifest that is downloaded must be generated within a Mercurial tree?\r\n\r\nYes. If we can put both kinds of data in the cache without having git installed that will work well (I previosuly didn't realise that git blob ids didn't have any environment-specific metadata).\r\n\r\nAre you imagining the cache being in the manifest itself, or in a seperate file? Given the mtime discussion below, I assume a separate file.\r\n\r\n> That's not the motivation: we shouldn't need to ask the VCS what's changed since the current HEAD for lint (we already don't for manifest) because we should be able to return the majority of lint results from cache so we can just always lint the whole tree.\r\n\r\nI guess that makes sense (using pure metadata as keys always worries me, but I guess there's a long history of using mtimes here). How do you imagine it working the first time you run this when you have (I assume) a VCS hash (assuming a tracked file), but no mtime (I'm assuming that we don't think they're portable). Do we then ask the VCS if the file changed, or do you imagine everyone having to do a full lint once?\r\n\r\n> The motivation here is that it deals with large numbers of local files (e.g., having the CSS built test suites in the working tree), because for performance we absolutely need them cached\r\n\r\nOK, but that seems like a really niche use case. Why are you doing that? :) But if the same work also helps with other common cases, it does seem worthwhile.\r\n\r\n> For lint I think there's three different cases:\r\n\r\nYes, that seems reasonable if it's easy to implement. "}}
{"actor":{"display_login":"jgraham"}}