{"action":"created","comment":{"url":"https://api.github.com/repos/talis/tripod-php/pulls/comments/66633123","id":66633123,"diff_hunk":"@@ -396,58 +406,16 @@ public function generateView($viewId,$resource=null,$context=null,$queueName=nul\n             $t = new \\Tripod\\Timer();\n             $t->start();\n \n-            $from = $this->getFromCollectionForViewSpec($viewSpec);\n+            $from = $viewSpec['from'];\n             $collection = $this->config->getCollectionForView($this->storeName, $viewId);\n \n+            $this->ensureIndices($viewSpec, $collection);","path":"src/mongo/delegates/Views.class.php","position":104,"original_position":96,"commit_id":"ef1bcb4c7b7d5e7f9aa9fe54bd42a22ff31ad1d0","original_commit_id":"28e522d4d38f4ee996bf484b43495539452430e9","user":{"login":"junglebarry","id":242248,"avatar_url":"https://avatars.githubusercontent.com/u/242248?v=3","gravatar_id":"","url":"https://api.github.com/users/junglebarry","html_url":"https://github.com/junglebarry","followers_url":"https://api.github.com/users/junglebarry/followers","following_url":"https://api.github.com/users/junglebarry/following{/other_user}","gists_url":"https://api.github.com/users/junglebarry/gists{/gist_id}","starred_url":"https://api.github.com/users/junglebarry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junglebarry/subscriptions","organizations_url":"https://api.github.com/users/junglebarry/orgs","repos_url":"https://api.github.com/users/junglebarry/repos","events_url":"https://api.github.com/users/junglebarry/events{/privacy}","received_events_url":"https://api.github.com/users/junglebarry/received_events","type":"User","site_admin":false},"body":"Ok.  I can see that there's not an \"on deployment, run this\" type of job where we could naturally hook that in, so I accept this approach is sensible.  Thanks for clarifying!\r\n\r\nWould you object to me centralising the shared indices to reduce code, as I have done with `Config::ensureIndexForSpecRevision`?","created_at":"2016-06-10T15:29:25Z","updated_at":"2016-06-10T15:29:25Z","html_url":"https://github.com/talis/tripod-php/pull/102#discussion_r66633123","pull_request_url":"https://api.github.com/repos/talis/tripod-php/pulls/102","_links":{"self":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/comments/66633123"},"html":{"href":"https://github.com/talis/tripod-php/pull/102#discussion_r66633123"},"pull_request":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/102"}}},"pull_request":{"url":"https://api.github.com/repos/talis/tripod-php/pulls/102","id":72007347,"html_url":"https://github.com/talis/tripod-php/pull/102","diff_url":"https://github.com/talis/tripod-php/pull/102.diff","patch_url":"https://github.com/talis/tripod-php/pull/102.patch","issue_url":"https://api.github.com/repos/talis/tripod-php/issues/102","number":102,"state":"open","locked":false,"title":"20% Outdated spec regeneration","user":{"login":"junglebarry","id":242248,"avatar_url":"https://avatars.githubusercontent.com/u/242248?v=3","gravatar_id":"","url":"https://api.github.com/users/junglebarry","html_url":"https://github.com/junglebarry","followers_url":"https://api.github.com/users/junglebarry/followers","following_url":"https://api.github.com/users/junglebarry/following{/other_user}","gists_url":"https://api.github.com/users/junglebarry/gists{/gist_id}","starred_url":"https://api.github.com/users/junglebarry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junglebarry/subscriptions","organizations_url":"https://api.github.com/users/junglebarry/orgs","repos_url":"https://api.github.com/users/junglebarry/repos","events_url":"https://api.github.com/users/junglebarry/events{/privacy}","received_events_url":"https://api.github.com/users/junglebarry/received_events","type":"User","site_admin":false},"body":"This PR is still work-in-progress, but at a point where feedback is sought.\r\n\r\n# Aims\r\n\r\n* Adds a `revision` to composite specifications, a monotonically-increasing integer\r\n* When a composite document is being saved, include the specification revision in the document\r\n* Provide a query to find composite documents that are generated with older (outdated) specifications\r\n* Regenerates outdated composite documents\r\n\r\nThe current code deals only with views, but should generalise to other composites.  Some of the mechanisms are already generic enough to support this, but where they are not, I've attempted to provide more abstract mechanisms.  There are some that need reworking.\r\n\r\nThe bulk of the code is in a new job, called `DiscoverOutdatedComposites`.  This is designed to run on a CRON schedule, and at each `perform` step, will iterate across all composites, find a list of documents that need regeneration, and trigger their regeneration.  The current code does this all inside `perform` for a single job, but the ultimate intention is to split out the regeneration from the discovery.\r\n\r\nNo effort has been made to include logging or profiling at this point.  No effort has been made to trigger jobs.  Testing is unit-test only.\r\n\r\n## Representing Revisions\r\n\r\nThe code currently uses independent revision numbers for every composite spec.  This is simple, and makes comparisons easy, so documents need regeneration when:\r\n\r\n* For a single document, its `revision` is less than the revision of its specification;\r\n* For a query, we can just use mongodb's `$lt` operator to extract all matches.\r\n\r\nIt's also easy for distribution, since outdated replicas will naturally stop contributing (eventually).\r\n\r\nDespite this simplicity, this revisioning scheme has issues:\r\n\r\n* It's a lot of manual work to manage spec updates;\r\n* It is hard to automate, because we need to retain a history of changes, making build-time revision update difficult (we need to store history somewhere external - a file of `<specification, revision, hash>` triples, perhaps?)\r\n\r\n### Alternative: single revision number, plus per-specification hashes\r\n\r\nThe original suggestion from Chris was to use a hash, since it picks up semantic changes only, and is easy to compute.  Without some notion of order, however, it's possible to get into \"tennis matches\", where two replicas disagree on the correct hash, and attempt to undo each other's work.  By adding in a single global revision (again monotonically-increasing), we can provide the order needed to resolve this problem.\r\n\r\nUsing a hash makes comparisons more complex, so documents need regeneration when:\r\n\r\n* For a single document, its `revision` is less than the global revision **and** its hash is different to the hash of its specification;\r\n* For a query, we need `{ _revision: { $lt: <GLOBAL REVISION> }, _hash: { $ne: <SPEC. HASH> }}`\r\n\r\nNote that the revision is checked first in both cases, so takes precedence over the hash.\r\n\r\nThe hash-based approach has a couple of nice properties:\r\n\r\n* We can trivially automate incrementing the global revision (since we don't need to compare with previous versions - the hash will do that)\r\n* The hash only captures semantic changes\r\n\r\n\r\n## Issues with the current code\r\n\r\n### Composite classes are tied to podNames\r\n\r\nIn order to construct a `Views` instance, I need a collection associated with a `podName`, which is broadly unused.  Within each composite, the collection/podName are rarely used, but they are enforced by the hierarchy (`Views <-- CompositeBase <-- DriverBase`).  It cannot be simply removed.\r\n\r\nI'm not sure that Views/Tables/Search is conceptually tied to a pod.  Could do with some discussion here to figure out why it is tied thus at present.\r\n\r\n### Regeneration is not uniformly available\r\n\r\nWe really need to create a generic mechanism for regenerating a document from its spec and root CBD, defined at the level of `IComposite`.  I've added another interface (`ICompositeRegen`) for now, so it need not be immediately applied to all composite types.\r\n\r\n\r\n## TODO\r\n\r\n* [ ] Remove comments that are present only for discussion in the PR;\r\n* [ ] Settle the revision/hash debate\r\n* [ ] Split the discovery and regeneration jobs on to separate queues\r\n* [x] Abstract the regeneration code into (a subset of) `IComposite` implementors.\r\n* [ ] Fetch all types of composites during the discovery step\r\n* [ ] Add logging back in - especially for counts of outdated documents\r\n* [ ] Add profiling\r\n","created_at":"2016-05-31T12:33:23Z","updated_at":"2016-06-10T15:29:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bc30cd27058f9bf5e7feff649318e59fac8440f9","assignee":{"login":"rsinger","id":72548,"avatar_url":"https://avatars.githubusercontent.com/u/72548?v=3","gravatar_id":"","url":"https://api.github.com/users/rsinger","html_url":"https://github.com/rsinger","followers_url":"https://api.github.com/users/rsinger/followers","following_url":"https://api.github.com/users/rsinger/following{/other_user}","gists_url":"https://api.github.com/users/rsinger/gists{/gist_id}","starred_url":"https://api.github.com/users/rsinger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsinger/subscriptions","organizations_url":"https://api.github.com/users/rsinger/orgs","repos_url":"https://api.github.com/users/rsinger/repos","events_url":"https://api.github.com/users/rsinger/events{/privacy}","received_events_url":"https://api.github.com/users/rsinger/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/talis/tripod-php/pulls/102/commits","review_comments_url":"https://api.github.com/repos/talis/tripod-php/pulls/102/comments","review_comment_url":"https://api.github.com/repos/talis/tripod-php/pulls/comments{/number}","comments_url":"https://api.github.com/repos/talis/tripod-php/issues/102/comments","statuses_url":"https://api.github.com/repos/talis/tripod-php/statuses/ef1bcb4c7b7d5e7f9aa9fe54bd42a22ff31ad1d0","head":{"label":"talis:tripod-eventual-view-regeneration","ref":"tripod-eventual-view-regeneration","sha":"ef1bcb4c7b7d5e7f9aa9fe54bd42a22ff31ad1d0","user":{"login":"talis","id":181416,"avatar_url":"https://avatars.githubusercontent.com/u/181416?v=3","gravatar_id":"","url":"https://api.github.com/users/talis","html_url":"https://github.com/talis","followers_url":"https://api.github.com/users/talis/followers","following_url":"https://api.github.com/users/talis/following{/other_user}","gists_url":"https://api.github.com/users/talis/gists{/gist_id}","starred_url":"https://api.github.com/users/talis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talis/subscriptions","organizations_url":"https://api.github.com/users/talis/orgs","repos_url":"https://api.github.com/users/talis/repos","events_url":"https://api.github.com/users/talis/events{/privacy}","received_events_url":"https://api.github.com/users/talis/received_events","type":"Organization","site_admin":false},"repo":{"id":17243040,"name":"tripod-php","full_name":"talis/tripod-php","owner":{"login":"talis","id":181416,"avatar_url":"https://avatars.githubusercontent.com/u/181416?v=3","gravatar_id":"","url":"https://api.github.com/users/talis","html_url":"https://github.com/talis","followers_url":"https://api.github.com/users/talis/followers","following_url":"https://api.github.com/users/talis/following{/other_user}","gists_url":"https://api.github.com/users/talis/gists{/gist_id}","starred_url":"https://api.github.com/users/talis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talis/subscriptions","organizations_url":"https://api.github.com/users/talis/orgs","repos_url":"https://api.github.com/users/talis/repos","events_url":"https://api.github.com/users/talis/events{/privacy}","received_events_url":"https://api.github.com/users/talis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/talis/tripod-php","description":"Object Graph Mapper for managing RDF data in Mongo","fork":false,"url":"https://api.github.com/repos/talis/tripod-php","forks_url":"https://api.github.com/repos/talis/tripod-php/forks","keys_url":"https://api.github.com/repos/talis/tripod-php/keys{/key_id}","collaborators_url":"https://api.github.com/repos/talis/tripod-php/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/talis/tripod-php/teams","hooks_url":"https://api.github.com/repos/talis/tripod-php/hooks","issue_events_url":"https://api.github.com/repos/talis/tripod-php/issues/events{/number}","events_url":"https://api.github.com/repos/talis/tripod-php/events","assignees_url":"https://api.github.com/repos/talis/tripod-php/assignees{/user}","branches_url":"https://api.github.com/repos/talis/tripod-php/branches{/branch}","tags_url":"https://api.github.com/repos/talis/tripod-php/tags","blobs_url":"https://api.github.com/repos/talis/tripod-php/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/talis/tripod-php/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/talis/tripod-php/git/refs{/sha}","trees_url":"https://api.github.com/repos/talis/tripod-php/git/trees{/sha}","statuses_url":"https://api.github.com/repos/talis/tripod-php/statuses/{sha}","languages_url":"https://api.github.com/repos/talis/tripod-php/languages","stargazers_url":"https://api.github.com/repos/talis/tripod-php/stargazers","contributors_url":"https://api.github.com/repos/talis/tripod-php/contributors","subscribers_url":"https://api.github.com/repos/talis/tripod-php/subscribers","subscription_url":"https://api.github.com/repos/talis/tripod-php/subscription","commits_url":"https://api.github.com/repos/talis/tripod-php/commits{/sha}","git_commits_url":"https://api.github.com/repos/talis/tripod-php/git/commits{/sha}","comments_url":"https://api.github.com/repos/talis/tripod-php/comments{/number}","issue_comment_url":"https://api.github.com/repos/talis/tripod-php/issues/comments{/number}","contents_url":"https://api.github.com/repos/talis/tripod-php/contents/{+path}","compare_url":"https://api.github.com/repos/talis/tripod-php/compare/{base}...{head}","merges_url":"https://api.github.com/repos/talis/tripod-php/merges","archive_url":"https://api.github.com/repos/talis/tripod-php/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/talis/tripod-php/downloads","issues_url":"https://api.github.com/repos/talis/tripod-php/issues{/number}","pulls_url":"https://api.github.com/repos/talis/tripod-php/pulls{/number}","milestones_url":"https://api.github.com/repos/talis/tripod-php/milestones{/number}","notifications_url":"https://api.github.com/repos/talis/tripod-php/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/talis/tripod-php/labels{/name}","releases_url":"https://api.github.com/repos/talis/tripod-php/releases{/id}","deployments_url":"https://api.github.com/repos/talis/tripod-php/deployments","created_at":"2014-02-27T09:09:51Z","updated_at":"2016-05-13T10:57:02Z","pushed_at":"2016-06-10T13:25:22Z","git_url":"git://github.com/talis/tripod-php.git","ssh_url":"git@github.com:talis/tripod-php.git","clone_url":"https://github.com/talis/tripod-php.git","svn_url":"https://github.com/talis/tripod-php","homepage":"","size":1779,"stargazers_count":23,"watchers_count":23,"language":"PHP","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":26,"forks":2,"open_issues":26,"watchers":23,"default_branch":"master"}},"base":{"label":"talis:master","ref":"master","sha":"d441fdf5184298afb9e741711e43d4694edfdbea","user":{"login":"talis","id":181416,"avatar_url":"https://avatars.githubusercontent.com/u/181416?v=3","gravatar_id":"","url":"https://api.github.com/users/talis","html_url":"https://github.com/talis","followers_url":"https://api.github.com/users/talis/followers","following_url":"https://api.github.com/users/talis/following{/other_user}","gists_url":"https://api.github.com/users/talis/gists{/gist_id}","starred_url":"https://api.github.com/users/talis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talis/subscriptions","organizations_url":"https://api.github.com/users/talis/orgs","repos_url":"https://api.github.com/users/talis/repos","events_url":"https://api.github.com/users/talis/events{/privacy}","received_events_url":"https://api.github.com/users/talis/received_events","type":"Organization","site_admin":false},"repo":{"id":17243040,"name":"tripod-php","full_name":"talis/tripod-php","owner":{"login":"talis","id":181416,"avatar_url":"https://avatars.githubusercontent.com/u/181416?v=3","gravatar_id":"","url":"https://api.github.com/users/talis","html_url":"https://github.com/talis","followers_url":"https://api.github.com/users/talis/followers","following_url":"https://api.github.com/users/talis/following{/other_user}","gists_url":"https://api.github.com/users/talis/gists{/gist_id}","starred_url":"https://api.github.com/users/talis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talis/subscriptions","organizations_url":"https://api.github.com/users/talis/orgs","repos_url":"https://api.github.com/users/talis/repos","events_url":"https://api.github.com/users/talis/events{/privacy}","received_events_url":"https://api.github.com/users/talis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/talis/tripod-php","description":"Object Graph Mapper for managing RDF data in Mongo","fork":false,"url":"https://api.github.com/repos/talis/tripod-php","forks_url":"https://api.github.com/repos/talis/tripod-php/forks","keys_url":"https://api.github.com/repos/talis/tripod-php/keys{/key_id}","collaborators_url":"https://api.github.com/repos/talis/tripod-php/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/talis/tripod-php/teams","hooks_url":"https://api.github.com/repos/talis/tripod-php/hooks","issue_events_url":"https://api.github.com/repos/talis/tripod-php/issues/events{/number}","events_url":"https://api.github.com/repos/talis/tripod-php/events","assignees_url":"https://api.github.com/repos/talis/tripod-php/assignees{/user}","branches_url":"https://api.github.com/repos/talis/tripod-php/branches{/branch}","tags_url":"https://api.github.com/repos/talis/tripod-php/tags","blobs_url":"https://api.github.com/repos/talis/tripod-php/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/talis/tripod-php/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/talis/tripod-php/git/refs{/sha}","trees_url":"https://api.github.com/repos/talis/tripod-php/git/trees{/sha}","statuses_url":"https://api.github.com/repos/talis/tripod-php/statuses/{sha}","languages_url":"https://api.github.com/repos/talis/tripod-php/languages","stargazers_url":"https://api.github.com/repos/talis/tripod-php/stargazers","contributors_url":"https://api.github.com/repos/talis/tripod-php/contributors","subscribers_url":"https://api.github.com/repos/talis/tripod-php/subscribers","subscription_url":"https://api.github.com/repos/talis/tripod-php/subscription","commits_url":"https://api.github.com/repos/talis/tripod-php/commits{/sha}","git_commits_url":"https://api.github.com/repos/talis/tripod-php/git/commits{/sha}","comments_url":"https://api.github.com/repos/talis/tripod-php/comments{/number}","issue_comment_url":"https://api.github.com/repos/talis/tripod-php/issues/comments{/number}","contents_url":"https://api.github.com/repos/talis/tripod-php/contents/{+path}","compare_url":"https://api.github.com/repos/talis/tripod-php/compare/{base}...{head}","merges_url":"https://api.github.com/repos/talis/tripod-php/merges","archive_url":"https://api.github.com/repos/talis/tripod-php/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/talis/tripod-php/downloads","issues_url":"https://api.github.com/repos/talis/tripod-php/issues{/number}","pulls_url":"https://api.github.com/repos/talis/tripod-php/pulls{/number}","milestones_url":"https://api.github.com/repos/talis/tripod-php/milestones{/number}","notifications_url":"https://api.github.com/repos/talis/tripod-php/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/talis/tripod-php/labels{/name}","releases_url":"https://api.github.com/repos/talis/tripod-php/releases{/id}","deployments_url":"https://api.github.com/repos/talis/tripod-php/deployments","created_at":"2014-02-27T09:09:51Z","updated_at":"2016-05-13T10:57:02Z","pushed_at":"2016-06-10T13:25:22Z","git_url":"git://github.com/talis/tripod-php.git","ssh_url":"git@github.com:talis/tripod-php.git","clone_url":"https://github.com/talis/tripod-php.git","svn_url":"https://github.com/talis/tripod-php","homepage":"","size":1779,"stargazers_count":23,"watchers_count":23,"language":"PHP","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":26,"forks":2,"open_issues":26,"watchers":23,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/102"},"html":{"href":"https://github.com/talis/tripod-php/pull/102"},"issue":{"href":"https://api.github.com/repos/talis/tripod-php/issues/102"},"comments":{"href":"https://api.github.com/repos/talis/tripod-php/issues/102/comments"},"review_comments":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/102/comments"},"review_comment":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/talis/tripod-php/pulls/102/commits"},"statuses":{"href":"https://api.github.com/repos/talis/tripod-php/statuses/ef1bcb4c7b7d5e7f9aa9fe54bd42a22ff31ad1d0"}}}}
{"actor":{"display_login":"junglebarry"}}