{"action":"created","issue":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/297","repository_url":"https://api.github.com/repos/StrikeNP/trac_test","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/297/comments","events_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/297/events","html_url":"https://github.com/StrikeNP/trac_test/issues/297","id":334605253,"node_id":"MDU6SXNzdWUzMzQ2MDUyNTM=","number":297,"title":"Latin hypercube sampling: Add vertical coherence (Trac #297)","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"labels":[{"id":971577014,"node_id":"MDU6TGFiZWw5NzE1NzcwMTQ=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/Migrated%20from%20Trac","name":"Migrated from Trac","color":"FFFFFF","default":false},{"id":971577274,"node_id":"MDU6TGFiZWw5NzE1NzcyNzQ=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/clubb_src","name":"clubb_src","color":"FFFFFF","default":false},{"id":971577273,"node_id":"MDU6TGFiZWw5NzE1NzcyNzM=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/dschanen@uwm.edu","name":"dschanen@uwm.edu","color":"FFFFFF","default":false},{"id":971576851,"node_id":"MDU6TGFiZWw5NzE1NzY4NTE=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/enhancement","name":"enhancement","color":"a2eeef","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/8","html_url":"https://github.com/StrikeNP/trac_test/milestone/8","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/8/labels","id":3442840,"node_id":"MDk6TWlsZXN0b25lMzQ0Mjg0MA==","number":8,"title":"Improve SILHS","description":null,"creator":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"open_issues":66,"closed_issues":15,"state":"open","created_at":"2018-06-21T18:03:51Z","updated_at":"2018-06-22T10:43:30Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-06-21T18:22:16Z","updated_at":"2018-06-22T13:40:32Z","closed_at":null,"author_association":"OWNER","body":"''Dave Schanen's additions are in italics'' \n\n**Introduction:** \n\nIn nature, cloud fields are correlated in the vertical.  That is, instead of clouds at each altitude being randomly overlapped, there is positive correlation in the vertical.\n\nRight now, the LH sample of s_mellor is chosen independently at each altitude.  This corresponds to random overlap.  Instead, we'll choose a random sample value of s_mellor at one altitude, and then choose s_mellor at other altitudes conditional on the value of s_mellor at the first altitude.  The other variates (t_mellor, w, etc.) will be chosen as is currently done.  \n\n**Requirements:**  \n\nCalculate the degree of correlation between s_mellor at different altitudes.  Then generate sample points of s_mellor that satisfy the desired correlation.\n\n**Technical spec:**\n\n1.  Keep the old random-overlap code intact.  Enclose all new code in a logical flag, l_lh_vert_overlap.  We might need to put this flag in the generate_lh_sample_points module and use it in multiple places unless the code is restructured.  Having flags in multiple places is undesirable, so it is worth exploring whether the code can be restructured easily.\n\n2.  For now, we can use maximal overlap between levels, but later, we'll create a function that computes the correlation between any two altitudes z1 and z2.  Ultimately, we'll want to use a function like\n\n```text\nvert_corr(k) = exp( \n                    - vert_corr_coef \n                    * abs( gr%zm(k) - gr%zm(km1) )\n                    / ( vertical_avg( km1, kp1, ... Lscale) )\n                   )\n```\nwhere vert_corr_coef ~= 0.1 is an empirical constant.   Also, let's insert a debug-level-1 assertion check to ensure that vert_corr lies within [0 , 1].  \n\n3.  Choose a starting altitude, i.e. a grid level for the first sample point.  Call it k_lh_start.  At first, we can manually prescribe it to be somewhere within the middle of the cloud layer.  That is, for now, let's hardwire in the grid level for RICO.  \nLater, we could set k_lh_start to be the grid level where Lscale or rc is a maximum using the Fortran intrinsic MAXLOC to see where rc is highest, but for a case like MPACE A, k_lh_start might vary a lot with time.  \n\nStill later, we can be more sophisticated and either choose the starting altitude randomly, or else cycle with time through different starting altitudes within the cloud layer, or choose multiple starting points per grid column, or else something different.\n\n4.  At k_lh_start, choose a value of X_u_one_lev(..., d_variables+1).  Later, use this same value of the d_variables+1 element at all altitudes.  The d+1 element will vary in the n_micro_calls dimension but not with altitude when l_lh_vert_overlap is true.  (This can be relaxed in future versions if we don't want maximum overlap.)\n \n5.  At k_lh_start, choose the value of s_mellor as we do in the current version of truncate_gaus_mixt.  \n\n6.  Then, do an upwards loop from k_lh_start to nnzp.  Choose the value of s_mellor at k_lh_start+1 based on the value of X_u_one_lev at k_lh_start, and so on up the loop.\n\nCurrently, truncate_gaus_mixt chooses the value of s_mellor given a uniformly distributed random number (in X_u_one_lev(...,1)) at k_lh_start+1.  The new code will choose s_mellor at k_lh_start+1 using the value of X_u_one_lev at the level below, k_lh_start.\n\nCurrently, generate_lh_sample only knows about s_mellor(k).  We will feed in a new variable X_nl_one_lev_prior.  This could be the variates at k-1 or k+1 depending on which way we want to loop.\n\nThen the other variates at k_lh_start+1 (t,w,etc.) are chosen as usual.\n\nLater, if we don't want maximum overlap, we can choose s_mellor using gaus_condt.  The input arguments will include the means for s_mellor(k) and s_mellor(k+1), and the 2x2 covariance matrix between s_mellor(k) and s_mellor(k+1).  The covariance matrix can be built from s_vert_corr above and the variances of s_mellor at k and k+1.  \n\n7.  Second, do a downwards loop from k_lh_start to 1.  Choose the value of s_mellor as in the upward loop.\n\n8.  In the future, we may want to weight the columns of sample points unequally, but for now we'll weight them equally.\n\n**Tests:**  \n\n1.  Test whether the new LH sample converges to the analytic solution for KK microphysics for the RF02 and RICO cases.   \n\n2.  When we choose one LH sample point per grid box and timestep, plot the profiles and see whether they appear qualitatively more correlated in the vertical than the randomly overlapped samples (i.e. the samples when l_lh_vert_overlap = .false.).\n\nWhen we're ready to test the case in which there is more than one LH sample point per grid box and timestep, we should output the full 3D netCDF file with all the sample points as well.  \n\n**Time estimate:** \n\nRoughly how many hours do you think this will take?\nDave estimates: \"Once I have a better understanding of how truncate_gaus_mixt, gaus_condt, et cetera actually work I think the modifications themselves won't be more than a few hours, but testing will probably take some time.\"  (However, for the simple maximal overlap case, we'll only need to modify X_u_one_lev, not truncate_gaus_mixt or gaus_condt.)\n\nAttachments:\n\n[plot_explicit_ta_configs.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_explicit_ta_configs.maff)\n[plot_new_pdf_config_1_plot_2.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_new_pdf_config_1_plot_2.maff)\n[plot_combo_pdf_run_3.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_combo_pdf_run_3.maff)\n[plot_input_fields_rtp3_thlp3_1.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_input_fields_rtp3_thlp3_1.maff)\n[plot_new_pdf_20180522_test_1.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_new_pdf_20180522_test_1.maff)\n[plot_attempts_8_10.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_attempts_8_10.maff)\n[plot_attempt_8_only.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_attempt_8_only.maff)\n[plot_beta_1p3.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_beta_1p3.maff)\n[plot_beta_1p3_all.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_beta_1p3_all.maff)\n\nMigrated from http://carson.math.uwm.edu/trac/clubb/ticket/297\n```json\n{\n    \"status\": \"closed\", \n    \"changetime\": \"2010-04-21T21:36:26\", \n    \"description\": \"''Dave Schanen's additions are in italics'' \\n\\n'''Introduction:''' \\n\\nIn nature, cloud fields are correlated in the vertical.  That is, instead of clouds at each altitude being randomly overlapped, there is positive correlation in the vertical.\\n\\nRight now, the LH sample of s_mellor is chosen independently at each altitude.  This corresponds to random overlap.  Instead, we'll choose a random sample value of s_mellor at one altitude, and then choose s_mellor at other altitudes conditional on the value of s_mellor at the first altitude.  The other variates (t_mellor, w, etc.) will be chosen as is currently done.  \\n\\n'''Requirements:'''  \\n\\nCalculate the degree of correlation between s_mellor at different altitudes.  Then generate sample points of s_mellor that satisfy the desired correlation.\\n\\n'''Technical spec:'''\\n\\n1.  Keep the old random-overlap code intact.  Enclose all new code in a logical flag, l_lh_vert_overlap.  We might need to put this flag in the generate_lh_sample_points module and use it in multiple places unless the code is restructured.  Having flags in multiple places is undesirable, so it is worth exploring whether the code can be restructured easily.\\n\\n2.  For now, we can use maximal overlap between levels, but later, we'll create a function that computes the correlation between any two altitudes z1 and z2.  Ultimately, we'll want to use a function like\\n{{{\\nvert_corr(k) = exp( \\n                    - vert_corr_coef \\n                    * abs( gr%zm(k) - gr%zm(km1) )\\n                    / ( vertical_avg( km1, kp1, ... Lscale) )\\n                   )\\n}}}\\nwhere vert_corr_coef ~= 0.1 is an empirical constant.   Also, let's insert a debug-level-1 assertion check to ensure that vert_corr lies within [0 , 1].  \\n\\n3.  Choose a starting altitude, i.e. a grid level for the first sample point.  Call it k_lh_start.  At first, we can manually prescribe it to be somewhere within the middle of the cloud layer.  That is, for now, let's hardwire in the grid level for RICO.  \\nLater, we could set k_lh_start to be the grid level where Lscale or rc is a maximum using the Fortran intrinsic MAXLOC to see where rc is highest, but for a case like MPACE A, k_lh_start might vary a lot with time.  \\n\\nStill later, we can be more sophisticated and either choose the starting altitude randomly, or else cycle with time through different starting altitudes within the cloud layer, or choose multiple starting points per grid column, or else something different.\\n\\n4.  At k_lh_start, choose a value of X_u_one_lev(..., d_variables+1).  Later, use this same value of the d_variables+1 element at all altitudes.  The d+1 element will vary in the n_micro_calls dimension but not with altitude when l_lh_vert_overlap is true.  (This can be relaxed in future versions if we don't want maximum overlap.)\\n \\n5.  At k_lh_start, choose the value of s_mellor as we do in the current version of truncate_gaus_mixt.  \\n\\n6.  Then, do an upwards loop from k_lh_start to nnzp.  Choose the value of s_mellor at k_lh_start+1 based on the value of X_u_one_lev at k_lh_start, and so on up the loop.\\n\\nCurrently, truncate_gaus_mixt chooses the value of s_mellor given a uniformly distributed random number (in X_u_one_lev(...,1)) at k_lh_start+1.  The new code will choose s_mellor at k_lh_start+1 using the value of X_u_one_lev at the level below, k_lh_start.\\n\\nCurrently, generate_lh_sample only knows about s_mellor(k).  We will feed in a new variable X_nl_one_lev_prior.  This could be the variates at k-1 or k+1 depending on which way we want to loop.\\n\\nThen the other variates at k_lh_start+1 (t,w,etc.) are chosen as usual.\\n\\nLater, if we don't want maximum overlap, we can choose s_mellor using gaus_condt.  The input arguments will include the means for s_mellor(k) and s_mellor(k+1), and the 2x2 covariance matrix between s_mellor(k) and s_mellor(k+1).  The covariance matrix can be built from s_vert_corr above and the variances of s_mellor at k and k+1.  \\n\\n7.  Second, do a downwards loop from k_lh_start to 1.  Choose the value of s_mellor as in the upward loop.\\n\\n8.  In the future, we may want to weight the columns of sample points unequally, but for now we'll weight them equally.\\n\\n'''Tests:'''  \\n\\n1.  Test whether the new LH sample converges to the analytic solution for KK microphysics for the RF02 and RICO cases.   \\n\\n2.  When we choose one LH sample point per grid box and timestep, plot the profiles and see whether they appear qualitatively more correlated in the vertical than the randomly overlapped samples (i.e. the samples when l_lh_vert_overlap = .false.).\\n\\nWhen we're ready to test the case in which there is more than one LH sample point per grid box and timestep, we should output the full 3D netCDF file with all the sample points as well.  \\n\\n'''Time estimate:''' \\n\\nRoughly how many hours do you think this will take?\\nDave estimates: \\\"Once I have a better understanding of how truncate_gaus_mixt, gaus_condt, et cetera actually work I think the modifications themselves won't be more than a few hours, but testing will probably take some time.\\\"  (However, for the simple maximal overlap case, we'll only need to modify X_u_one_lev, not truncate_gaus_mixt or gaus_condt.)\", \n    \"reporter\": \"vlarson@uwm.edu\", \n    \"cc\": \"vlarson@uwm.edu\", \n    \"resolution\": \"Verified by V. Larson\", \n    \"_ts\": \"1271885786000000\", \n    \"component\": \"clubb_src\", \n    \"summary\": \"Latin hypercube sampling: Add vertical coherence\", \n    \"priority\": \"major\", \n    \"keywords\": \"\", \n    \"time\": \"2010-03-06T13:38:57\", \n    \"milestone\": \"Improve SILHS\", \n    \"owner\": \"dschanen@uwm.edu\", \n    \"type\": \"enhancement\"\n}\n```\n"},"comment":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/comments/399446101","html_url":"https://github.com/StrikeNP/trac_test/issues/297#issuecomment-399446101","issue_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/297","id":399446101,"node_id":"MDEyOklzc3VlQ29tbWVudDM5OTQ0NjEwMQ==","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"created_at":"2018-06-22T13:40:32Z","updated_at":"2018-06-22T13:40:32Z","author_association":"OWNER","body":"**Trac update on 2010/03/07 at 22:35**\n**vlarson@uwm.edu commented:**\n\nA simpler case than the generalized overlap described in the header is maximum overlap.  For this case, we skip step two (function s_vert_corr) and modify step 6 (upwards loop).  In the upwards loop, instead of computing s_mellor using the value of s_mellor at the level below, we keep the same value of X_u_one_lev(..., 1) at all grid levels, and compute s_mellor at all levels as in the current code.\n\nLet's start with the maximum overlap case because it is simpler.\n\n"}}
{"actor":{"display_login":"StrikeNP"}}