{"action":"created","issue":{"url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/155","repository_url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate","labels_url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/155/labels{/name}","comments_url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/155/comments","events_url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/155/events","html_url":"https://github.com/johnmyleswhite/ProjectTemplate/pull/155","id":178209499,"number":155,"title":"Faster cache function","user":{"login":"connectedblue","id":16526825,"avatar_url":"https://avatars.githubusercontent.com/u/16526825?v=3","gravatar_id":"","url":"https://api.github.com/users/connectedblue","html_url":"https://github.com/connectedblue","followers_url":"https://api.github.com/users/connectedblue/followers","following_url":"https://api.github.com/users/connectedblue/following{/other_user}","gists_url":"https://api.github.com/users/connectedblue/gists{/gist_id}","starred_url":"https://api.github.com/users/connectedblue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/connectedblue/subscriptions","organizations_url":"https://api.github.com/users/connectedblue/orgs","repos_url":"https://api.github.com/users/connectedblue/repos","events_url":"https://api.github.com/users/connectedblue/events{/privacy}","received_events_url":"https://api.github.com/users/connectedblue/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-09-20T23:46:29Z","updated_at":"2016-09-23T21:48:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/pulls/155","html_url":"https://github.com/johnmyleswhite/ProjectTemplate/pull/155","diff_url":"https://github.com/johnmyleswhite/ProjectTemplate/pull/155.diff","patch_url":"https://github.com/johnmyleswhite/ProjectTemplate/pull/155.patch"},"body":"This PR proposes an update to the `cache()` function and introduces a new function `clear.cache()`.  Together these changes give the user a faster experience when processing large datasets when `load.project()` is called.\r\n\r\nIt is a large update, and I have tested the scenarios on a variety of my own projects.  I have also made it backward compatible, so existing scripts will still work (but they will work faster).\r\n\r\nThe main change is the rules which are applied when the cache function is called, so that a cached object is only written out if it has changed.  There are two main ways to use it:\r\n\r\n```\r\ncache(\"foo\")\r\n```\r\n\r\nThis will create a cached object of the `foo` object in the global environment the first time it is encountered in a `munge` script.  If the object has not changed in subsequent invocations of `load.project()`, it will not be cached again.  Messages are output to tell the user what is going on.\r\n\r\nThe second way is as follows:\r\n\r\n```\r\ncache(\"foo\", CODE=\"\r\n     x<-loaded_data$name\r\n     x<-as.character(x)\r\n     x[[1]]\r\n\")\r\n```\r\n\r\nHere the `cache` function is \"wrapped around\" some code in the munge script directly.   In this case, the   `CODE` is executed in the global environment (so a variable x is created) and also the variable `foo` is assigned the result of the execution, also in the global environment (in this case the first entry of the vector `x`).  `foo` is then written to the cache.\r\n\r\nThe cool thing about subsequent invocations of `load.project()` is that if the code has changed since `foo` was cached, it will be evaluated and re-cached again.  If the code hasn't changed, it will not be re-evaluated which saves a lot of munge time.\r\n\r\nThere is a final variant of this as follows:\r\n\r\n```\r\ncache(\"foo\", depends=c(\"x\", \"y\"), CODE=\"\r\n     x<-loaded_data$name\r\n     x<-as.character(x)\r\n     x[[1]]\r\n\")\r\n```\r\n\r\nIn this case, if either the code, or the variables `x` or `y` have changed, then the code is re-executed.  This is important if the code chunk references variables where you want to re-evaluate if those other variables have changed.\r\n\r\nI tried this on a project where my munge processing was taking several minutes the first time.  In subsequent runs, the `load.project()` ran almost instanteously when there were no changes to the cached code or datasets.\r\n\r\nNote that the loading of data from the cache has not been touched in this PR.\r\n\r\nFinally, a `clear.cache` function is provided to allow the user fine grain control over the cache - they can remove objects from the cache if they want to force a re-run of all data from scratch.  this function takes the form:\r\n\r\n```\r\nclear.cache(\"y\", \"z\", \"a\")\r\n```\r\nor\r\n```\r\nclear.cache()\r\n```\r\nto clear everything.\r\n\r\nSorry for the long description.  This is a big update, but I hope is a useful addition to speed up the user experience during large analysis."},"comment":{"url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/comments/249310660","html_url":"https://github.com/johnmyleswhite/ProjectTemplate/pull/155#issuecomment-249310660","issue_url":"https://api.github.com/repos/johnmyleswhite/ProjectTemplate/issues/155","id":249310660,"user":{"login":"connectedblue","id":16526825,"avatar_url":"https://avatars.githubusercontent.com/u/16526825?v=3","gravatar_id":"","url":"https://api.github.com/users/connectedblue","html_url":"https://github.com/connectedblue","followers_url":"https://api.github.com/users/connectedblue/followers","following_url":"https://api.github.com/users/connectedblue/following{/other_user}","gists_url":"https://api.github.com/users/connectedblue/gists{/gist_id}","starred_url":"https://api.github.com/users/connectedblue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/connectedblue/subscriptions","organizations_url":"https://api.github.com/users/connectedblue/orgs","repos_url":"https://api.github.com/users/connectedblue/repos","events_url":"https://api.github.com/users/connectedblue/events{/privacy}","received_events_url":"https://api.github.com/users/connectedblue/received_events","type":"User","site_admin":false},"created_at":"2016-09-23T21:48:33Z","updated_at":"2016-09-23T21:48:33Z","body":"Pushed a fix for the issue @wlandau found - `cache()` will stop with an error if a variable does not exist in the global environment."}}
{"actor":{"display_login":"connectedblue"}}