{"action":"created","issue":{"url":"https://api.github.com/repos/hpc4cmb/toast/issues/231","repository_url":"https://api.github.com/repos/hpc4cmb/toast","labels_url":"https://api.github.com/repos/hpc4cmb/toast/issues/231/labels{/name}","comments_url":"https://api.github.com/repos/hpc4cmb/toast/issues/231/comments","events_url":"https://api.github.com/repos/hpc4cmb/toast/issues/231/events","html_url":"https://github.com/hpc4cmb/toast/issues/231","id":312538138,"number":231,"title":"Buffered time domain operationns","user":{"login":"tskisner","id":84221,"avatar_url":"https://avatars2.githubusercontent.com/u/84221?v=4","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},"labels":[{"id":197705300,"url":"https://api.github.com/repos/hpc4cmb/toast/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":{"login":"tskisner","id":84221,"avatar_url":"https://avatars2.githubusercontent.com/u/84221?v=4","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},"assignees":[{"login":"tskisner","id":84221,"avatar_url":"https://avatars2.githubusercontent.com/u/84221?v=4","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-04-09T13:48:11Z","updated_at":"2018-04-09T16:16:01Z","closed_at":null,"author_association":"COLLABORATOR","body":"There are many places in the code where we do vector operations using temporaries on very large vectors.  We would like to do these in a buffered way to reduce the size of the temporary arrays.  This will hopefully address #199.\r\n\r\nAlthough this is fairly straightforward, one challenge is that we now use the TOD.local_* methods everywhere.  These were implemented to reduce boilerplate code in the case of wanting to use cached data.  Recall for example, instead of (pseudocode):\r\n```\r\nif name in cache:\r\n    data = tod.cache.ref(name)\r\nelse:\r\n    data = tod.read(name)\r\n    tod.cache.put(name, data)\r\n```\r\nWe currently do:\r\n```\r\ndata = tod.local_data(name)\r\n```\r\nHowever, when doing buffered operations we don't want to cache the whole vector if we might throw it away at the end.  Also, some data objects (time stamps, boresight pointing, common flags, etc) might be cached internally by the TOD derived class in the case of reading data from disk.  In those cases, calling the local_* methods would make an extra copy in the user-visible cache.\r\n\r\nHere is an idea:\r\n\r\n1. For common data (timestamps, flags, boresight, velocity, position) go back to the original design where the TOD class is responsible for caching those internally (so that the read_* methods just transparently return data from the cache).  This would remove the need for the local_times, local_common_flags, local_velocity, and local_position methods.\r\n\r\n2. For detector-specific quantities, none of the TOD classes will ever cache those internally.  So in those cases we want to make it \"easy\" for the user to transparently use the cached data.  Currently we have the local_signal, local_flags, and local_pointing methods which return the full vector of data for the observation and always put the full data into the cache.\r\n\r\nI think some requirements for 2 above are:\r\n\r\n- We want to call a method that returns data from either cache or from the corresponding _get* method.\r\n\r\n- We want to support a sample range in those methods.\r\n\r\n- If data is not already in the cache, we should *optionally* read the entire vector, copy that into cache, and return the subset of samples.\r\n\r\nI think there are several possible ways of meeting these requirements.  For example:\r\n\r\n1.  Add a sample range and boolean \"cache\" / \"keep\" / etc flag to the detector local_* methods.\r\n\r\n2.  Add a boolean \"cache\" / \"keep\" / etc flag to the read_* methods (which already have a sample range).\r\n\r\nBoth the read_* and local_* methods are in the TOD base class and do almost identical things.  We should consider whether we can simplify the API by just removing one of them.  For example, if the \"local_\" name is more clear to the user, then we should just remove the read_* methods and modify the local_ methods to meet the requirements we need.\r\n\r\n@keskitalo and @zonca, any thoughts?\r\n    \r\n"},"comment":{"url":"https://api.github.com/repos/hpc4cmb/toast/issues/comments/379808257","html_url":"https://github.com/hpc4cmb/toast/issues/231#issuecomment-379808257","issue_url":"https://api.github.com/repos/hpc4cmb/toast/issues/231","id":379808257,"user":{"login":"keskitalo","id":596250,"avatar_url":"https://avatars1.githubusercontent.com/u/596250?v=4","gravatar_id":"","url":"https://api.github.com/users/keskitalo","html_url":"https://github.com/keskitalo","followers_url":"https://api.github.com/users/keskitalo/followers","following_url":"https://api.github.com/users/keskitalo/following{/other_user}","gists_url":"https://api.github.com/users/keskitalo/gists{/gist_id}","starred_url":"https://api.github.com/users/keskitalo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keskitalo/subscriptions","organizations_url":"https://api.github.com/users/keskitalo/orgs","repos_url":"https://api.github.com/users/keskitalo/repos","events_url":"https://api.github.com/users/keskitalo/events{/privacy}","received_events_url":"https://api.github.com/users/keskitalo/received_events","type":"User","site_admin":false},"created_at":"2018-04-09T16:16:01Z","updated_at":"2018-04-09T16:16:01Z","author_association":"COLLABORATOR","body":"Or we could add the ranges to the local_* methods. In this case the logic should be:\r\n* if the object is cached, return the requested range\r\n* if the object is not cached, read and return only the specified range, do not cache anything."}}
{"actor":{"display_login":"keskitalo"}}