{"action":"created","issue":{"url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/853","repository_url":"https://api.github.com/repos/NOAA-GFDL/MOM6","labels_url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/853/labels{/name}","comments_url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/853/comments","events_url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/853/events","html_url":"https://github.com/NOAA-GFDL/MOM6/issues/853","id":360700567,"node_id":"MDU6SXNzdWUzNjA3MDA1Njc=","number":853,"title":"Unable to print out CPU_stats","user":{"login":"hlkong","id":27735148,"node_id":"MDQ6VXNlcjI3NzM1MTQ4","avatar_url":"https://avatars3.githubusercontent.com/u/27735148?v=4","gravatar_id":"","url":"https://api.github.com/users/hlkong","html_url":"https://github.com/hlkong","followers_url":"https://api.github.com/users/hlkong/followers","following_url":"https://api.github.com/users/hlkong/following{/other_user}","gists_url":"https://api.github.com/users/hlkong/gists{/gist_id}","starred_url":"https://api.github.com/users/hlkong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hlkong/subscriptions","organizations_url":"https://api.github.com/users/hlkong/orgs","repos_url":"https://api.github.com/users/hlkong/repos","events_url":"https://api.github.com/users/hlkong/events{/privacy}","received_events_url":"https://api.github.com/users/hlkong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-17T01:34:57Z","updated_at":"2018-09-21T12:01:32Z","closed_at":null,"author_association":"NONE","body":"I got this issue when running [my configuration](https://github.com/hlkong/MOM6/tree/shoebox_std) under 0.1 deg resolution: I would only be able to get the first few lines written into CPU_stats. For example,\r\n\r\n> Startup CPU time:     1928.374 sec summed across  168 PEs.\r\n>        Day, Step number,     CPU time, CPU time change\r\n>  256042.000,           0,     1928.374,     1928.374\r\n\r\nand I wouldn't get any more lines. I checked MOM_PARAMETER_DOC.all but everything seems fine: \r\n\r\n> WRITE_CPU_STEPS = 1000\r\n> MAXCPU = -1.0\r\n> CPU_TIME_FILE = \"CPU_stats\"\r\n\r\nThis has only occurred at high resolution runs. At 1 deg it never happened. \r\n\r\nDoes anyone happen to know what may be the issue?\r\n\r\nThanks!!\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/comments/423508043","html_url":"https://github.com/NOAA-GFDL/MOM6/issues/853#issuecomment-423508043","issue_url":"https://api.github.com/repos/NOAA-GFDL/MOM6/issues/853","id":423508043,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzUwODA0Mw==","user":{"login":"Hallberg-NOAA","id":6529173,"node_id":"MDQ6VXNlcjY1MjkxNzM=","avatar_url":"https://avatars1.githubusercontent.com/u/6529173?v=4","gravatar_id":"","url":"https://api.github.com/users/Hallberg-NOAA","html_url":"https://github.com/Hallberg-NOAA","followers_url":"https://api.github.com/users/Hallberg-NOAA/followers","following_url":"https://api.github.com/users/Hallberg-NOAA/following{/other_user}","gists_url":"https://api.github.com/users/Hallberg-NOAA/gists{/gist_id}","starred_url":"https://api.github.com/users/Hallberg-NOAA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hallberg-NOAA/subscriptions","organizations_url":"https://api.github.com/users/Hallberg-NOAA/orgs","repos_url":"https://api.github.com/users/Hallberg-NOAA/repos","events_url":"https://api.github.com/users/Hallberg-NOAA/events{/privacy}","received_events_url":"https://api.github.com/users/Hallberg-NOAA/received_events","type":"User","site_admin":false},"created_at":"2018-09-21T12:01:32Z","updated_at":"2018-09-21T12:01:32Z","author_association":"CONTRIBUTOR","body":"@hlkong, it sounds to me like your symptoms are most consistent with either a run that is either finishing (or crashing or timing out) without flushing all of its I/O buffers, or your  frequency with which you are writing the CPU stats is inconsistent with the length of your run.\r\n\r\n  Try making WRITE_CPU_STEPS much shorter (like 2) to see whether you start to see more entries written to the CPU_stats file. Also check whether the other text output files (like the ocean.stats files) are being written, and how long your simulation is actually going based on the output to std_out.\r\n\r\n  It is also possible that we may need to add an explicit call to flush the buffer to the CPU time I/O file at line 176 of MOM_write_cputime.F90 (although doing so could slow the model down), or that we should add a new MOM_write_cputime_end routine to flush and close the I/O unit number.\r\n\r\n  Please advise us, @hlkong, if the above steps give you a better idea of what might be going on."}}
{"actor":{"display_login":"Hallberg-NOAA"}}