{"action":"created","issue":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/790","repository_url":"https://api.github.com/repos/StrikeNP/trac_test","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/790/labels{/name}","comments_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/790/comments","events_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/790/events","html_url":"https://github.com/StrikeNP/trac_test/issues/790","id":330469419,"node_id":"MDU6SXNzdWUzMzA0Njk0MTk=","number":790,"title":"SILHS: Errors in statistical moments involving thl using new sampling method (Trac #790)","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"labels":[{"id":958019852,"node_id":"MDU6TGFiZWw5NTgwMTk4NTI=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/Migrated%20from%20Trac","name":"Migrated from Trac","color":"FFFFFF","default":false},{"id":958042830,"node_id":"MDU6TGFiZWw5NTgwNDI4MzA=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/clubb_src","name":"clubb_src","color":"FFFFFF","default":false},{"id":958019855,"node_id":"MDU6TGFiZWw5NTgwMTk4NTU=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/defect","name":"defect","color":"FFFFFF","default":false},{"id":958114258,"node_id":"MDU6TGFiZWw5NTgxMTQyNTg=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/raut@uwm.edu","name":"raut@uwm.edu","color":"FFFFFF","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/8","html_url":"https://github.com/StrikeNP/trac_test/milestone/8","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/8/labels","id":3408876,"node_id":"MDk6TWlsZXN0b25lMzQwODg3Ng==","number":8,"title":"Improve SILHS","description":null,"creator":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":64,"state":"open","created_at":"2018-06-07T19:59:06Z","updated_at":"2018-06-10T12:48:56Z","due_on":null,"closed_at":null},"comments":21,"created_at":"2018-06-07T23:24:52Z","updated_at":"2018-06-10T14:17:26Z","closed_at":null,"author_association":"OWNER","body":"'''Introduction'''\n\nWe have noted that the new sampling method (#736, #740, #752, etc.) produces very noisy estimates of `thlm` (ticket:752#comment:85) and \"grossly\" overestimated (biased) estimates of `thlp2` (ticket:752#comment:86) in CLUBB standalone. We also noted that in simulations of (S)CAM-CLUBB-SILHS (with 4 sample points), `thlp2_mc`, whose compututation depends on `lh_thlm`, has a negative spike near the surface with the new sampling method but not with the old one (https://github.com/larson-group/cam/issues/61#comment:70). Naturally, we believe these problems are related.\n\nThe root of the problem seems to be that our sample point weights do not add up to one. This fact produces an extra source of variance, which in the case of thl seems to significantly outweigh the natural variance of the field. We see this even with 64 sample points (attachment:thlm_timeseries_compare.https://github.com/larson-group/svg/issues/752)!\n\n'''Technical spec'''\n\nWe have some options:\n\n* '''\"Normalize\" the sample point weights such that they add to N (number of samples).''' This was suggested by Brian (ticket:752#comment:79) and me (ticket:736#comment:39) and experimented with by Kate in global simulations (https://github.com/larson-group/cpt/issues/201#comment:41). I have been hesitant to do this because in general it would result in sampling bias (see ticket:752#comment:80 et seq.) But perhaps that bias wouldn't be nearly as bad as the strange results we see with the new sampling method.\n\n* '''Modify the variance and covariance estimators to produce an unbiased result.''' I don't know whether this is possible. If we somehow manage to do this, it wouldn't fix the poor sample estimate of `thlm`, but maybe `thlp2` and `thlp2_mc` would be better predicted.\n\n* '''Modify the sampling algorithm such that the weights add to N and the result is still unbiased.''' One idea: according to Lemieux's seminal book (__Monte Carlo and Quasi-Monte Carlo Sampling__, 2009), the method in bullet point 1 (which I think is equivalent to what she calls \"poststratification\") is unbiased as long as every category is sampled at least once. The current \"default\" code (2Cat-!CldPcp) uses only two categories, so it would be straightforward to ensure that each category is sampled at least once with our default of 4 sample points. We couldn't ensure each category is sampled in the next generation sampling method, 8Cat, unless we used 8 or more sample points.\n\n* '''Give up.''' Why torture ourselves?\n\nAttachments:\n\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/HesterbergDiss.pdf\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/thlp2_forcing_sampling_compare.svg\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter49.diff\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter49.zip\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/normalized_convergence_test.maff\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_newsamp.diff\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_newsamp.zip\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_newsamp_3sc.zip\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_newsamp_2sc.zip\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_oldsamp_2sc.zip\nhttp://carson.math.uwm.edu/trac/clubb/attachment/ticket/790/tuning_iter48_newsamp_pf_profiles.zip\n\nMigrated from http://carson.math.uwm.edu/trac/clubb/ticket/790\n```json\n{\n    \"status\": \"closed\", \n    \"changetime\": \"2016-07-22T21:45:44\", \n    \"description\": \"'''Introduction'''\\n\\nWe have noted that the new sampling method (#736, #740, #752, etc.) produces very noisy estimates of `thlm` (ticket:752#comment:85) and \\\"grossly\\\" overestimated (biased) estimates of `thlp2` (ticket:752#comment:86) in CLUBB standalone. We also noted that in simulations of (S)CAM-CLUBB-SILHS (with 4 sample points), `thlp2_mc`, whose compututation depends on `lh_thlm`, has a negative spike near the surface with the new sampling method but not with the old one (cam:ticket:61#comment:70). Naturally, we believe these problems are related.\\n\\nThe root of the problem seems to be that our sample point weights do not add up to one. This fact produces an extra source of variance, which in the case of thl seems to significantly outweigh the natural variance of the field. We see this even with 64 sample points (attachment:thlm_timeseries_compare.svg:ticket:752)!\\n\\n'''Technical spec'''\\n\\nWe have some options:\\n\\n* '''\\\"Normalize\\\" the sample point weights such that they add to N (number of samples).''' This was suggested by Brian (ticket:752#comment:79) and me (ticket:736#comment:39) and experimented with by Kate in global simulations (cpt:ticket:201#comment:41). I have been hesitant to do this because in general it would result in sampling bias (see ticket:752#comment:80 et seq.) But perhaps that bias wouldn't be nearly as bad as the strange results we see with the new sampling method.\\n\\n* '''Modify the variance and covariance estimators to produce an unbiased result.''' I don't know whether this is possible. If we somehow manage to do this, it wouldn't fix the poor sample estimate of `thlm`, but maybe `thlp2` and `thlp2_mc` would be better predicted.\\n\\n* '''Modify the sampling algorithm such that the weights add to N and the result is still unbiased.''' One idea: according to Lemieux's seminal book (__Monte Carlo and Quasi-Monte Carlo Sampling__, 2009), the method in bullet point 1 (which I think is equivalent to what she calls \\\"poststratification\\\") is unbiased as long as every category is sampled at least once. The current \\\"default\\\" code (2Cat-!CldPcp) uses only two categories, so it would be straightforward to ensure that each category is sampled at least once with our default of 4 sample points. We couldn't ensure each category is sampled in the next generation sampling method, 8Cat, unless we used 8 or more sample points.\\n\\n* '''Give up.''' Why torture ourselves?\", \n    \"reporter\": \"raut@uwm.edu\", \n    \"cc\": \"vlarson@uwm.edu\", \n    \"resolution\": \"fixed\", \n    \"_ts\": \"1469223944118357\", \n    \"component\": \"clubb_src\", \n    \"summary\": \"SILHS: Errors in statistical moments involving thl using new sampling method\", \n    \"priority\": \"major\", \n    \"keywords\": \"SILHS covariance\", \n    \"time\": \"2016-06-17T00:51:39\", \n    \"milestone\": \"Improve SILHS\", \n    \"owner\": \"raut@uwm.edu\", \n    \"type\": \"defect\"\n}\n```\n"},"comment":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/comments/396052718","html_url":"https://github.com/StrikeNP/trac_test/issues/790#issuecomment-396052718","issue_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/790","id":396052718,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjA1MjcxOA==","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"created_at":"2018-06-10T14:17:26Z","updated_at":"2018-06-10T14:17:26Z","author_association":"OWNER","body":"Trac update at 2016/07/07 at 18:31\n\n\n* raut@uwm.edu changed attachment from \"\" to \"tuning_iter48_newsamp.zip\"\n* raut@uwm.edu commented:\n\nThis can be compared to cam:attachment:tuning_iter48.https://github.com/larson-group/zip/issues/64.\n\n"}}
{"actor":{"display_login":"StrikeNP"}}