{"action":"created","issue":{"url":"https://api.github.com/repos/PecanProject/pecan/issues/1009","repository_url":"https://api.github.com/repos/PecanProject/pecan","labels_url":"https://api.github.com/repos/PecanProject/pecan/issues/1009/labels{/name}","comments_url":"https://api.github.com/repos/PecanProject/pecan/issues/1009/comments","events_url":"https://api.github.com/repos/PecanProject/pecan/issues/1009/events","html_url":"https://github.com/PecanProject/pecan/issues/1009","id":176181505,"number":1009,"title":"met.process refactor: met.module & user workflows","user":{"login":"mdietze","id":3081466,"avatar_url":"https://avatars.githubusercontent.com/u/3081466?v=3","gravatar_id":"","url":"https://api.github.com/users/mdietze","html_url":"https://github.com/mdietze","followers_url":"https://api.github.com/users/mdietze/followers","following_url":"https://api.github.com/users/mdietze/following{/other_user}","gists_url":"https://api.github.com/users/mdietze/gists{/gist_id}","starred_url":"https://api.github.com/users/mdietze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdietze/subscriptions","organizations_url":"https://api.github.com/users/mdietze/orgs","repos_url":"https://api.github.com/users/mdietze/repos","events_url":"https://api.github.com/users/mdietze/events{/privacy}","received_events_url":"https://api.github.com/users/mdietze/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/PecanProject/pecan/labels/Ideas","name":"Ideas","color":"006b75"}],"state":"open","locked":false,"assignee":{"login":"robkooper","id":1580696,"avatar_url":"https://avatars.githubusercontent.com/u/1580696?v=3","gravatar_id":"","url":"https://api.github.com/users/robkooper","html_url":"https://github.com/robkooper","followers_url":"https://api.github.com/users/robkooper/followers","following_url":"https://api.github.com/users/robkooper/following{/other_user}","gists_url":"https://api.github.com/users/robkooper/gists{/gist_id}","starred_url":"https://api.github.com/users/robkooper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robkooper/subscriptions","organizations_url":"https://api.github.com/users/robkooper/orgs","repos_url":"https://api.github.com/users/robkooper/repos","events_url":"https://api.github.com/users/robkooper/events{/privacy}","received_events_url":"https://api.github.com/users/robkooper/received_events","type":"User","site_admin":false},"assignees":[{"login":"robkooper","id":1580696,"avatar_url":"https://avatars.githubusercontent.com/u/1580696?v=3","gravatar_id":"","url":"https://api.github.com/users/robkooper","html_url":"https://github.com/robkooper","followers_url":"https://api.github.com/users/robkooper/followers","following_url":"https://api.github.com/users/robkooper/following{/other_user}","gists_url":"https://api.github.com/users/robkooper/gists{/gist_id}","starred_url":"https://api.github.com/users/robkooper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robkooper/subscriptions","organizations_url":"https://api.github.com/users/robkooper/orgs","repos_url":"https://api.github.com/users/robkooper/repos","events_url":"https://api.github.com/users/robkooper/events{/privacy}","received_events_url":"https://api.github.com/users/robkooper/received_events","type":"User","site_admin":false},{"login":"mdietze","id":3081466,"avatar_url":"https://avatars.githubusercontent.com/u/3081466?v=3","gravatar_id":"","url":"https://api.github.com/users/mdietze","html_url":"https://github.com/mdietze","followers_url":"https://api.github.com/users/mdietze/followers","following_url":"https://api.github.com/users/mdietze/following{/other_user}","gists_url":"https://api.github.com/users/mdietze/gists{/gist_id}","starred_url":"https://api.github.com/users/mdietze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdietze/subscriptions","organizations_url":"https://api.github.com/users/mdietze/orgs","repos_url":"https://api.github.com/users/mdietze/repos","events_url":"https://api.github.com/users/mdietze/events{/privacy}","received_events_url":"https://api.github.com/users/mdietze/received_events","type":"User","site_admin":false},{"login":"ankurdesai","id":3084183,"avatar_url":"https://avatars.githubusercontent.com/u/3084183?v=3","gravatar_id":"","url":"https://api.github.com/users/ankurdesai","html_url":"https://github.com/ankurdesai","followers_url":"https://api.github.com/users/ankurdesai/followers","following_url":"https://api.github.com/users/ankurdesai/following{/other_user}","gists_url":"https://api.github.com/users/ankurdesai/gists{/gist_id}","starred_url":"https://api.github.com/users/ankurdesai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankurdesai/subscriptions","organizations_url":"https://api.github.com/users/ankurdesai/orgs","repos_url":"https://api.github.com/users/ankurdesai/repos","events_url":"https://api.github.com/users/ankurdesai/events{/privacy}","received_events_url":"https://api.github.com/users/ankurdesai/received_events","type":"User","site_admin":false},{"login":"bcow","id":5503595,"avatar_url":"https://avatars.githubusercontent.com/u/5503595?v=3","gravatar_id":"","url":"https://api.github.com/users/bcow","html_url":"https://github.com/bcow","followers_url":"https://api.github.com/users/bcow/followers","following_url":"https://api.github.com/users/bcow/following{/other_user}","gists_url":"https://api.github.com/users/bcow/gists{/gist_id}","starred_url":"https://api.github.com/users/bcow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcow/subscriptions","organizations_url":"https://api.github.com/users/bcow/orgs","repos_url":"https://api.github.com/users/bcow/repos","events_url":"https://api.github.com/users/bcow/events{/privacy}","received_events_url":"https://api.github.com/users/bcow/received_events","type":"User","site_admin":false},{"login":"rykelly","id":6820366,"avatar_url":"https://avatars.githubusercontent.com/u/6820366?v=3","gravatar_id":"","url":"https://api.github.com/users/rykelly","html_url":"https://github.com/rykelly","followers_url":"https://api.github.com/users/rykelly/followers","following_url":"https://api.github.com/users/rykelly/following{/other_user}","gists_url":"https://api.github.com/users/rykelly/gists{/gist_id}","starred_url":"https://api.github.com/users/rykelly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rykelly/subscriptions","organizations_url":"https://api.github.com/users/rykelly/orgs","repos_url":"https://api.github.com/users/rykelly/repos","events_url":"https://api.github.com/users/rykelly/events{/privacy}","received_events_url":"https://api.github.com/users/rykelly/received_events","type":"User","site_admin":false},{"login":"jsimkins2","id":13629742,"avatar_url":"https://avatars.githubusercontent.com/u/13629742?v=3","gravatar_id":"","url":"https://api.github.com/users/jsimkins2","html_url":"https://github.com/jsimkins2","followers_url":"https://api.github.com/users/jsimkins2/followers","following_url":"https://api.github.com/users/jsimkins2/following{/other_user}","gists_url":"https://api.github.com/users/jsimkins2/gists{/gist_id}","starred_url":"https://api.github.com/users/jsimkins2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsimkins2/subscriptions","organizations_url":"https://api.github.com/users/jsimkins2/orgs","repos_url":"https://api.github.com/users/jsimkins2/repos","events_url":"https://api.github.com/users/jsimkins2/events{/privacy}","received_events_url":"https://api.github.com/users/jsimkins2/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/PecanProject/pecan/milestones/21","html_url":"https://github.com/PecanProject/pecan/milestones/1.4.9%20code%20hardening","labels_url":"https://api.github.com/repos/PecanProject/pecan/milestones/21/labels","id":1920602,"number":21,"title":"1.4.9 code hardening","description":"","creator":{"login":"mdietze","id":3081466,"avatar_url":"https://avatars.githubusercontent.com/u/3081466?v=3","gravatar_id":"","url":"https://api.github.com/users/mdietze","html_url":"https://github.com/mdietze","followers_url":"https://api.github.com/users/mdietze/followers","following_url":"https://api.github.com/users/mdietze/following{/other_user}","gists_url":"https://api.github.com/users/mdietze/gists{/gist_id}","starred_url":"https://api.github.com/users/mdietze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdietze/subscriptions","organizations_url":"https://api.github.com/users/mdietze/orgs","repos_url":"https://api.github.com/users/mdietze/repos","events_url":"https://api.github.com/users/mdietze/events{/privacy}","received_events_url":"https://api.github.com/users/mdietze/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":0,"state":"open","created_at":"2016-08-03T12:18:20Z","updated_at":"2016-09-10T14:09:37Z","due_on":"2016-10-07T00:00:00Z","closed_at":null},"comments":6,"created_at":"2016-09-10T14:09:36Z","updated_at":"2016-09-13T01:10:05Z","closed_at":null,"body":"met.process has gotten too complicated, and as we discover things the original design couldn't handle, it's becoming a mess of hacks that is hard to maintain and extend. This is in contrast to the original idea, which was that it would be easy for contributors to add new modules or swap out modules (e.g. choose between alternative gap filling or down scaling routines).  What I propose below is intended to reduce code redundancy, reduce special cases, and increase flexibility & clarity.\r\n\r\n**First idea**: A lot of what happens at each step in met process is the same. If the preprocessing says your supposed to run a step, you: (1) query the database and file system to perform checks [do files exist? do I need to extend date ranges? is this a force update? etc.] , (2) get variables organized, (3) pass the request off to download/convert.inputs/BrownDog, (4) do a bit of clean up, (5) pass the required ID's off to the next step.\r\n\r\nI propose that these steps be refactored into a single function, met.module, that would take the module name, site info, and an arbitrary number (0 to n) of blocks of stuff returned by previous met.module calls.  Furthermore, lets store what's returned in a general list, rather than having a bunch of stage-specific id's floating around (raw.id, cf.id, ready.id, model.id, etc.)\r\n\r\nSo, for example, CRUNCEP might look like:\r\n```\r\nmet.step[[1]] = met.module(\"download.CRUNCEP\", site.info)\r\nmet.step[[2]] = met.module(paste0(\"met2model\",model),site.info,met.step[[1]])\r\n```\r\nwhile Ameriflux might look like:\r\n```\r\nmet.step[[1]] = met.module(\"download.Ameriflux\", site.info)\r\nmet.step[[2]] = met.module(\"met2CF.Ameriflux\", site.info,met.step[[1]])\r\nmet.step[[3]] = met.module(\"metgapfill\",site.info,met.step[[2]])\r\nmet.step[[4]] = met.module(paste0(\"met2model\",model),site.info,met.step[[3]])\r\n```\r\n \r\n**Second idea**: The above approach would be easy to put in a loop that could be generalized to allow more flexible workflow specifications.  \r\n\r\nFor example, the above Ameriflux example might be stored as:\r\n```\r\n1 = download.Ameriflux\r\n2 = met2CF.Ameriflux(1)\r\n3 = metgapfill(2)\r\n4 = met2model.[MODEL](3)\r\n```\r\nor some XML equivalent and parsed at runtime as the loop iterates. A 'default' workflow for each met type could be stored in the met registry, but users could also specify workflows explicitly in the XML.  The approach would also allow for more flexible extensions, such as using one product to gapfill another:\r\n```\r\n1 = download.CRUNCEP\r\n2 = download.Ameriflux\r\n3 = met2CF.Ameriflux(2)\r\n4 = gapFillFusion(3,1)\r\n5 = met2model.[MODEL](4)\r\n```\r\nor using one to train the downscaling of another:\r\n```\r\n1 = download.CRUNCEP\r\n2 = download.Ameriflux\r\n3 = met2CF.Ameriflux(2)\r\n4 = downscale(1,3)\r\n5 = met2model.[MODEL](4)\r\n```\r\n\r\nThoughts and opinions?"},"comment":{"url":"https://api.github.com/repos/PecanProject/pecan/issues/comments/246543584","html_url":"https://github.com/PecanProject/pecan/issues/1009#issuecomment-246543584","issue_url":"https://api.github.com/repos/PecanProject/pecan/issues/1009","id":246543584,"user":{"login":"bcow","id":5503595,"avatar_url":"https://avatars.githubusercontent.com/u/5503595?v=3","gravatar_id":"","url":"https://api.github.com/users/bcow","html_url":"https://github.com/bcow","followers_url":"https://api.github.com/users/bcow/followers","following_url":"https://api.github.com/users/bcow/following{/other_user}","gists_url":"https://api.github.com/users/bcow/gists{/gist_id}","starred_url":"https://api.github.com/users/bcow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcow/subscriptions","organizations_url":"https://api.github.com/users/bcow/orgs","repos_url":"https://api.github.com/users/bcow/repos","events_url":"https://api.github.com/users/bcow/events{/privacy}","received_events_url":"https://api.github.com/users/bcow/received_events","type":"User","site_admin":false},"created_at":"2016-09-13T01:10:05Z","updated_at":"2016-09-13T01:10:05Z","body":"In terms of the `met.module` proposal - we should use this as an opportunity to remove any met specific tasks from `convert.input` - especially in the browndog section. "}}
{"actor":{"display_login":"bcow"}}