{"action":"created","issue":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/687","repository_url":"https://api.github.com/repos/StrikeNP/trac_test","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/687/labels{/name}","comments_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/687/comments","events_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/687/events","html_url":"https://github.com/StrikeNP/trac_test/issues/687","id":334662773,"node_id":"MDU6SXNzdWUzMzQ2NjI3NzM=","number":687,"title":"CLUBB frustrations! -- switching the hydrometeor indices produces different results. (Trac #687)","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"labels":[{"id":971577014,"node_id":"MDU6TGFiZWw5NzE1NzcwMTQ=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/Migrated%20from%20Trac","name":"Migrated from Trac","color":"FFFFFF","default":false},{"id":971577274,"node_id":"MDU6TGFiZWw5NzE1NzcyNzQ=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/clubb_src","name":"clubb_src","color":"FFFFFF","default":false},{"id":971577020,"node_id":"MDU6TGFiZWw5NzE1NzcwMjA=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/defect","name":"defect","color":"FFFFFF","default":false},{"id":971649570,"node_id":"MDU6TGFiZWw5NzE2NDk1NzA=","url":"https://api.github.com/repos/StrikeNP/trac_test/labels/raut@uwm.edu","name":"raut@uwm.edu","color":"FFFFFF","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/13","html_url":"https://github.com/StrikeNP/trac_test/milestone/13","labels_url":"https://api.github.com/repos/StrikeNP/trac_test/milestones/13/labels","id":3442872,"node_id":"MDk6TWlsZXN0b25lMzQ0Mjg3Mg==","number":13,"title":"4. Fix bugs","description":null,"creator":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"open_issues":27,"closed_issues":26,"state":"open","created_at":"2018-06-21T18:21:13Z","updated_at":"2018-06-23T18:46:34Z","due_on":null,"closed_at":null},"comments":59,"created_at":"2018-06-21T21:22:23Z","updated_at":"2018-06-23T20:13:50Z","closed_at":null,"author_association":"OWNER","body":"I was trying to refactor some code on Thursday.  The original problem was that the hydrometeor indices were not a proper subset of the PDF indices when some options were enabled with Morrison microphysics.  Here is an excerpt from the SVN log for r6892:\n\n```text\nI recently ran across a problem in CLUBB. When looping over all the hydrometeors in\nhydromet_dim and mapping the hydromet index to the PDF index, I ran across an array\nout of bounds error. In a nutshell, it turned out that the hydrometeor set of indices\n(iirrainm, iiNrm, etc.) was not always entirely subset of the PDF set of indices\n(iiPDF_rrain, iiPDF_Nr, etc.). To make a point, each index in the hydrometeor set of\nindices should also correspond to an index in the PDF set of indices. In other words,\nthe hydrometeor set of indices should be a proper subset of the PDF set of indices.\nWhen there are indices found in the hydrometeor set that are not found in the PDF set,\nthis causes problems.\n```\n\nWhile doing some refactoring to handle the problem, **I MADE A HORRIFIC NEW DISCOVERY**!  I changed the order of the ''hydrometeor indices'' from the current Morrison arrangement of:\n\n```text\n       iirrainm    = 1\n       iirsnowm    = 2\n       iiricem     = 3\n       iirgraupelm = 4\n\n       iiNrm       = 5\n       iiNsnowm    = 6\n       iiNim       = 7\n       iiNgraupelm = 8\n\n       hydromet_dim = 8\n```\nto:\n\n```text\n       iirrainm = 1\n       iiNrm    = 2\n\n       if ( l_ice_micro ) then\n\n          iiricem  = 3\n          iiNim    = 4\n          iirsnowm = 5\n          iiNsnowm = 6\n\n          doicemicro = .true.\n\n          if ( l_graupel ) then\n\n             iirgraupelm = 7\n             iiNgraupelm = 8\n\n             hydromet_dim = 8\n\n             dograupel = .true.\n\n          else ! l_graupel disabled\n\n             iirgraupelm = -1\n             iiNgraupelm = -1\n\n             hydromet_dim = 6\n\n             dograupel = .false.\n\n          endif ! l_graupel\n\n       else ! l_ice_micro disabled\n\n          iirsnowm    = -1\n          iiricem     = -1\n          iiNsnowm    = -1\n          iiNim       = -1\n          iirgraupelm = -1\n          iiNgraupelm = -1\n\n          hydromet_dim = 2\n\n          doicemicro = .false.\n          dograupel = .false.\n\n       endif ! l_ice_micro\n```\nThis was the only change I made.  Even for cases that still had 8 hydrometeors (```l_ice_micro``` and ```l_graupel```), **the results changed**.  In fact, all stats output file (zt, zm, and sfc) results changed for ARM 97, LBA, TWP-ICE, MPACE-B LH, and MC3E.  Additionally, only the zt file changed for MPACE A and CLEX-9 NOV 02.  However, all other cases that use Morrison microphysics (the CGILS cases, CLEX-9 OCT 14, Jun. 25, Nov. 11, etc.) all had bin-diff identical results.\n\nAfter being freaked out by this, I decided to perform the same test for KK microphysics.  I simply switched around the order of the ```iirrainm``` and ```iiNrm``` indices (the default is ```irrainm = 1``` and ```iiNrm = 2```; I switched them to ```iirrainm = 2``` and ```iiNrm = 1```).  I ran the RICO cases.  Again, all three statistical output files changed (zt, zm, and sfc).\n\nThe code should produce the same results no matter what order the hydrometeor array indices have.  When hydrometeor index orders are switched the code produces differing results from all SILHS-Morrison cases and some local Morrison cases (MC3E, MPACE A, CLEX 9 NOV 02).  It also produces differing results for upscaled KK cases (RICO was the only one I checked).  This tells me that the error is not found in any one of the microphysics schemes.  Rather, it must be found in a piece of code that is ''common to all microphysics schemes! ''  However, it must be a piece of code that is not always activated (which would explain why some cases that use Morrison microphysics don't change at all).\n\nI was looking at this a long time on Thursday and couldn't figure it out.  Any suggestions of where we might start to look?\n\nAttachments:\n\n[plot_explicit_ta_configs.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_explicit_ta_configs.maff)\n[plot_new_pdf_config_1_plot_2.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_new_pdf_config_1_plot_2.maff)\n[plot_combo_pdf_run_3.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_combo_pdf_run_3.maff)\n[plot_input_fields_rtp3_thlp3_1.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_input_fields_rtp3_thlp3_1.maff)\n[plot_new_pdf_20180522_test_1.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_new_pdf_20180522_test_1.maff)\n[plot_attempts_8_10.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_attempts_8_10.maff)\n[plot_attempt_8_only.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_attempt_8_only.maff)\n[plot_beta_1p3.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_beta_1p3.maff)\n[plot_beta_1p3_all.maff](https://github.com/larson-group/trac_attachment_archive/blob/master/trac_test/822/plot_beta_1p3_all.maff)\n\nMigrated from http://carson.math.uwm.edu/trac/clubb/ticket/687\n```json\n{\n    \"status\": \"closed\", \n    \"changetime\": \"2014-06-07T04:10:51\", \n    \"description\": \"I was trying to refactor some code on Thursday.  The original problem was that the hydrometeor indices were not a proper subset of the PDF indices when some options were enabled with Morrison microphysics.  Here is an excerpt from the SVN log for r6892:\\n{{{\\nI recently ran across a problem in CLUBB. When looping over all the hydrometeors in\\nhydromet_dim and mapping the hydromet index to the PDF index, I ran across an array\\nout of bounds error. In a nutshell, it turned out that the hydrometeor set of indices\\n(iirrainm, iiNrm, etc.) was not always entirely subset of the PDF set of indices\\n(iiPDF_rrain, iiPDF_Nr, etc.). To make a point, each index in the hydrometeor set of\\nindices should also correspond to an index in the PDF set of indices. In other words,\\nthe hydrometeor set of indices should be a proper subset of the PDF set of indices.\\nWhen there are indices found in the hydrometeor set that are not found in the PDF set,\\nthis causes problems.\\n}}}\\n\\nWhile doing some refactoring to handle the problem, '''I MADE A HORRIFIC NEW DISCOVERY'''!  I changed the order of the ''hydrometeor indices'' from the current Morrison arrangement of:\\n{{{\\n       iirrainm    = 1\\n       iirsnowm    = 2\\n       iiricem     = 3\\n       iirgraupelm = 4\\n\\n       iiNrm       = 5\\n       iiNsnowm    = 6\\n       iiNim       = 7\\n       iiNgraupelm = 8\\n\\n       hydromet_dim = 8\\n}}}\\nto:\\n{{{\\n       iirrainm = 1\\n       iiNrm    = 2\\n\\n       if ( l_ice_micro ) then\\n\\n          iiricem  = 3\\n          iiNim    = 4\\n          iirsnowm = 5\\n          iiNsnowm = 6\\n\\n          doicemicro = .true.\\n\\n          if ( l_graupel ) then\\n\\n             iirgraupelm = 7\\n             iiNgraupelm = 8\\n\\n             hydromet_dim = 8\\n\\n             dograupel = .true.\\n\\n          else ! l_graupel disabled\\n\\n             iirgraupelm = -1\\n             iiNgraupelm = -1\\n\\n             hydromet_dim = 6\\n\\n             dograupel = .false.\\n\\n          endif ! l_graupel\\n\\n       else ! l_ice_micro disabled\\n\\n          iirsnowm    = -1\\n          iiricem     = -1\\n          iiNsnowm    = -1\\n          iiNim       = -1\\n          iirgraupelm = -1\\n          iiNgraupelm = -1\\n\\n          hydromet_dim = 2\\n\\n          doicemicro = .false.\\n          dograupel = .false.\\n\\n       endif ! l_ice_micro\\n}}}\\nThis was the only change I made.  Even for cases that still had 8 hydrometeors ({{{l_ice_micro}}} and {{{l_graupel}}}), '''the results changed'''.  In fact, all stats output file (zt, zm, and sfc) results changed for ARM 97, LBA, TWP-ICE, MPACE-B LH, and MC3E.  Additionally, only the zt file changed for MPACE A and CLEX-9 NOV 02.  However, all other cases that use Morrison microphysics (the CGILS cases, CLEX-9 OCT 14, Jun. 25, Nov. 11, etc.) all had bin-diff identical results.\\n\\nAfter being freaked out by this, I decided to perform the same test for KK microphysics.  I simply switched around the order of the {{{iirrainm}}} and {{{iiNrm}}} indices (the default is {{{irrainm = 1}}} and {{{iiNrm = 2}}}; I switched them to {{{iirrainm = 2}}} and {{{iiNrm = 1}}}).  I ran the RICO cases.  Again, all three statistical output files changed (zt, zm, and sfc).\\n\\nThe code should produce the same results no matter what order the hydrometeor array indices have.  When hydrometeor index orders are switched the code produces differing results from all SILHS-Morrison cases and some local Morrison cases (MC3E, MPACE A, CLEX 9 NOV 02).  It also produces differing results for upscaled KK cases (RICO was the only one I checked).  This tells me that the error is not found in any one of the microphysics schemes.  Rather, it must be found in a piece of code that is ''common to all microphysics schemes! ''  However, it must be a piece of code that is not always activated (which would explain why some cases that use Morrison microphysics don't change at all).\\n\\nI was looking at this a long time on Thursday and couldn't figure it out.  Any suggestions of where we might start to look?\", \n    \"reporter\": \"bmg2@uwm.edu\", \n    \"cc\": \"vlarson@uwm.edu, janhft@uwm.edu, dschanen@uwm.edu\", \n    \"resolution\": \"fixed\", \n    \"_ts\": \"1402114251123321\", \n    \"component\": \"clubb_src\", \n    \"summary\": \"CLUBB frustrations! -- switching the hydrometeor indices produces different results.\", \n    \"priority\": \"critical\", \n    \"keywords\": \"hydrometeor indices\", \n    \"time\": \"2014-05-24T17:50:01\", \n    \"milestone\": \"4. Fix bugs\", \n    \"owner\": \"raut@uwm.edu\", \n    \"type\": \"defect\"\n}\n```\n"},"comment":{"url":"https://api.github.com/repos/StrikeNP/trac_test/issues/comments/399705759","html_url":"https://github.com/StrikeNP/trac_test/issues/687#issuecomment-399705759","issue_url":"https://api.github.com/repos/StrikeNP/trac_test/issues/687","id":399705759,"node_id":"MDEyOklzc3VlQ29tbWVudDM5OTcwNTc1OQ==","user":{"login":"StrikeNP","id":2577059,"node_id":"MDQ6VXNlcjI1NzcwNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2577059?v=4","gravatar_id":"","url":"https://api.github.com/users/StrikeNP","html_url":"https://github.com/StrikeNP","followers_url":"https://api.github.com/users/StrikeNP/followers","following_url":"https://api.github.com/users/StrikeNP/following{/other_user}","gists_url":"https://api.github.com/users/StrikeNP/gists{/gist_id}","starred_url":"https://api.github.com/users/StrikeNP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrikeNP/subscriptions","organizations_url":"https://api.github.com/users/StrikeNP/orgs","repos_url":"https://api.github.com/users/StrikeNP/repos","events_url":"https://api.github.com/users/StrikeNP/events{/privacy}","received_events_url":"https://api.github.com/users/StrikeNP/received_events","type":"User","site_admin":false},"created_at":"2018-06-23T20:13:50Z","updated_at":"2018-06-23T20:13:50Z","author_association":"OWNER","body":"**Trac update on 2014/05/29 at 16:48**\n\n* raut@uwm.edu changed _comment0 from:\n\n> Replying to [comment:55 raut@…]:\n> > Using print-line debugging, I once again found that the first difference between the two runs is in `precip_frac` at CLUBB timestep 721.\n> > \n> > Since MC3E outputs to stats every 12 timesteps, this is close to statistical timestep `721/12~60`, which is close to that reported in comment:53.\n> > \n> > Again, I am inclined to believe that the `precip_frac` code is the original source of the discrepancy. The fact that MC3E could be so unstable as for this to have a significant effect on the output is slightly alarming.\n> \n> I don't know how valuable it is to proceed to track the source of the discrepancy in MC3E, but two ideas popped up in my head:\n> \n> 1. Replace (only for the purpose of debugging) the code in the subroutine `precip_fraction` with something that will return exact answers regardless of the order of the indices, e.g. (`r_tot_hm_1 = hm1(k,iirrainm) + hm1(k,iirsnowm) + ...`). It is unclear how many places in the code we would have to do this before we either find a real bug or we achieve bit-for-bit identical results.\n> \n> 2. Run CLUBB in quad-precision. I imagine that doing so would help relieve some of the numerical instability in MC3E, and we could see if we get closer to identical output in the GrADS files.\n> \n> I think it would be nice to find out if MC3E really is just numerically unstable.\n\nto:\n\n> 1401382184583483\n\n\n* raut@uwm.edu changed _comment1 from:\n\n> Replying to [comment:55 raut@…]:\n> > Using print-line debugging, I once again found that the first difference between the two runs is in `precip_frac` at CLUBB timestep 721.\n> > \n> > Since MC3E outputs to stats every 12 timesteps, this is close to statistical timestep `721/12~60`, which is close to that reported in comment:53.\n> > \n> > Again, I am inclined to believe that the `precip_frac` code is the original source of the discrepancy. The fact that MC3E could be so unstable as for this to have a significant effect on the output is slightly alarming.\n> \n> I don't know how valuable it is to proceed to track the source of the discrepancy in MC3E, but two ideas popped up in my head:\n> \n> 1. Replace (only for the purpose of debugging) the code in the subroutine `precip_fraction` with something that will return exact answers regardless of the order of the indices, e.g. (`r_tot_hm_1 = hm1(k,iirrainm) + hm1(k,iirsnowm) + ...`). It is unclear how many places in the code we would have to do this before we either find a real bug or we achieve bit-for-bit identical results.\n> \n> 2. Run CLUBB in quad-precision. I imagine that doing so would help relieve some of the numerical instability in MC3E, and we could see if we get closer to identical output in the GrADS files.\n> \n> I think it would be nice to find out if MC3E really is just numerically unstable or there is a larger problem at play.\n\nto:\n\n> 1401382311783417\n\n\n* **raut@uwm.edu commented:**\n\nReplying to [comment:55 raut@…]:\n> Using print-line debugging, I once again found that the first difference between the two runs is in `precip_frac` at CLUBB timestep 721.\n> \n> Since MC3E outputs to stats every 12 timesteps, this is close to statistical timestep `721/12~60`, which is close to that reported in comment:53.\n> \n> Again, I am inclined to believe that the `precip_frac` code is the original source of the discrepancy. The fact that MC3E could be so unstable as for this to have a significant effect on the output is slightly alarming.\n\n\nI don't know how valuable it is to proceed to track the source of the discrepancy in MC3E, but two ideas popped up in my head:\n\n1. Replace (only for the purpose of debugging) the code in the subroutine `precip_fraction` with something that will return exact answers regardless of the order of the indices, e.g. (`r_tot_hm_1 = hm1(k,iirrainm) + hm1(k,iirsnowm) + ...`). It is unclear how many places in the code we would have to do this before we either find a real bug or we achieve bit-for-bit identical results.\n\n2. Run CLUBB in quad-precision. I imagine that doing so would help relieve some of the numerical instability in MC3E, and we could see if we get closer to identical output in the GrADS files.\n\nI think it would be nice to find out if MC3E really is just numerically unstable or if there is a larger problem at play.\n\n"}}
{"actor":{"display_login":"StrikeNP"}}