{"action":"created","issue":{"url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/62","repository_url":"https://api.github.com/repos/weecology/bbs-forecasting","labels_url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/62/labels{/name}","comments_url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/62/comments","events_url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/62/events","html_url":"https://github.com/weecology/bbs-forecasting/issues/62","id":170275693,"number":62,"title":"filter_ts() not actually filtering","user":{"login":"sdtaylor","id":12282062,"avatar_url":"https://avatars.githubusercontent.com/u/12282062?v=3","gravatar_id":"","url":"https://api.github.com/users/sdtaylor","html_url":"https://github.com/sdtaylor","followers_url":"https://api.github.com/users/sdtaylor/followers","following_url":"https://api.github.com/users/sdtaylor/following{/other_user}","gists_url":"https://api.github.com/users/sdtaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/sdtaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdtaylor/subscriptions","organizations_url":"https://api.github.com/users/sdtaylor/orgs","repos_url":"https://api.github.com/users/sdtaylor/repos","events_url":"https://api.github.com/users/sdtaylor/events{/privacy}","received_events_url":"https://api.github.com/users/sdtaylor/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/weecology/bbs-forecasting/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-09T21:29:52Z","updated_at":"2016-08-10T13:02:22Z","closed_at":null,"body":"If I call `get_richness_ts_env_data(start_yr, end_yr, min_num_yrs) %>% na.omit()`, there are sites within it that have less than `min_num_yrs'.  Looks like it has to do with the ordering of things.\r\n\r\nThis is from `forecast-bbs-core.R` line 186. `complete(site_id, year)` fills in all possible years for *all* sites, then `filter_ts()` a few lines down thinks that *all* sites then have plenty of years in their timeseries.  If I take out `complete(site_id, year)` it seems to work but I'm not sure what that might be breaking elsewhere. \r\n\r\n```\r\nget_richness_ts_env_data <- function(start_yr, end_yr, min_num_yrs){\r\n  bbs_data <- get_bbs_data(start_yr, end_yr, min_num_yrs)\r\n  richness_data <- bbs_data %>%\r\n    group_by(site_id, year, lat, long) %>%\r\n    dplyr::summarise(richness = n_distinct(species_id)) %>%\r\n    ungroup() %>%\r\n    complete(site_id, year)\r\n  richness_ts_env_data <- richness_data %>%\r\n    add_env_data() %>%\r\n    filter_ts(start_yr, end_yr, min_num_yrs)\r\n}\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/comments/238859422","html_url":"https://github.com/weecology/bbs-forecasting/issues/62#issuecomment-238859422","issue_url":"https://api.github.com/repos/weecology/bbs-forecasting/issues/62","id":238859422,"user":{"login":"ethanwhite","id":744427,"avatar_url":"https://avatars.githubusercontent.com/u/744427?v=3","gravatar_id":"","url":"https://api.github.com/users/ethanwhite","html_url":"https://github.com/ethanwhite","followers_url":"https://api.github.com/users/ethanwhite/followers","following_url":"https://api.github.com/users/ethanwhite/following{/other_user}","gists_url":"https://api.github.com/users/ethanwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/ethanwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethanwhite/subscriptions","organizations_url":"https://api.github.com/users/ethanwhite/orgs","repos_url":"https://api.github.com/users/ethanwhite/repos","events_url":"https://api.github.com/users/ethanwhite/events{/privacy}","received_events_url":"https://api.github.com/users/ethanwhite/received_events","type":"User","site_admin":false},"created_at":"2016-08-10T13:02:22Z","updated_at":"2016-08-10T13:02:22Z","body":"Good catch! Looks like the ordering logic isn't quite right, and is a little tricky. The complete adds in null richnesses for missing years (since @davharris pointed out that the time-series methods we were using could handle missing data). It turns out that getting the ordering perfect is tricky because we have some sites/years that are missing environmental data and a separate (but not necessarily distinct) set of sites/years that are missing richness. My initial thought was that we could just move it to be the last thing done to the data.\r\n\r\n```\r\nget_richness_ts_env_data <- function(start_yr, end_yr, min_num_yrs){\r\n  bbs_data <- get_bbs_data(start_yr, end_yr, min_num_yrs)\r\n  richness_data <- bbs_data %>%\r\n    group_by(site_id, year, lat, long) %>%\r\n    dplyr::summarise(richness = n_distinct(species_id)) %>%\r\n    ungroup()\r\n  richness_ts_env_data <- richness_data %>%\r\n    add_env_data() %>%\r\n    filter_ts(start_yr, end_yr, min_num_yrs) %>%\r\n    complete(site_id, year)\r\n}\r\n```\r\n\r\nThis appears to technically work without breaking anything but it isn't ideal because we end up with rows in the resulting that have null values for lat, long, and the environmental  data when really all that's missing is richness. Removing it entirely shouldn't work because the time-series methods will then just get shorter time-series when there are richness values missing with no information on the fact that there's data missing.\r\n\r\nI'll play around with the logic when I find a few minutes and try to get this working optimally."}}
{"actor":{"display_login":"ethanwhite"}}