{"action":"opened","issue":{"url":"https://api.github.com/repos/geoschem/gchp/issues/14","repository_url":"https://api.github.com/repos/geoschem/gchp","labels_url":"https://api.github.com/repos/geoschem/gchp/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/geoschem/gchp/issues/14/comments","events_url":"https://api.github.com/repos/geoschem/gchp/issues/14/events","html_url":"https://github.com/geoschem/gchp/issues/14","id":392796345,"node_id":"MDU6SXNzdWUzOTI3OTYzNDU=","number":14,"title":"GCHP c48 runs on AWS within Docker container die within 1 hour","user":{"login":"yantosca","id":5880296,"node_id":"MDQ6VXNlcjU4ODAyOTY=","avatar_url":"https://avatars0.githubusercontent.com/u/5880296?v=4","gravatar_id":"","url":"https://api.github.com/users/yantosca","html_url":"https://github.com/yantosca","followers_url":"https://api.github.com/users/yantosca/followers","following_url":"https://api.github.com/users/yantosca/following{/other_user}","gists_url":"https://api.github.com/users/yantosca/gists{/gist_id}","starred_url":"https://api.github.com/users/yantosca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yantosca/subscriptions","organizations_url":"https://api.github.com/users/yantosca/orgs","repos_url":"https://api.github.com/users/yantosca/repos","events_url":"https://api.github.com/users/yantosca/events{/privacy}","received_events_url":"https://api.github.com/users/yantosca/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-19T22:08:06Z","updated_at":"2018-12-19T22:08:06Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I ran a GCHP c48 run AWS cloud using \r\n\r\n    AMI         : container_geoschem_tutorial_2018121\r\n    Machine     : r4.4xlarge\r\n    Diagnostics : SpeciesConc_avg and SpeciesConc_inst \r\n\r\nand it died after an hour. \r\n\r\nIn runConfig.sh:\r\n\r\n    # Make sure your settings here match the resources you request on your\r\n    # cluster in your run script!!!\r\n    NUM_NODES=1\r\n    NUM_CORES_PER_NODE=12\r\n    NY=12\r\n    NX=1\r\n\r\n    # MAPL shared memory option (0: off, 1: on). Keep off unless you know what\r\n    # you are doing. Contact GCST for more information if you have memory\r\n    # problems you are unable to fix.\r\n    USE_SHMEM=0\r\n\r\n    #------------------------------------------------\r\n    #   Internal Cubed Sphere Resolution\r\n    #------------------------------------------------\r\n    CS_RES=48    # 24 ~ 4x5, 48 ~ 2x2.5, 90 ~ 1x1.25, 180 ~ 1/2 deg, 360 ~ 1/4 deg\r\n\r\n    ...\r\n\tStart_Time=\"20160701 000000\"\r\n    End_Time=\"20160701 010000\"\r\n    Duration=\"00000000 010000\"\r\n    ....\r\n\tcommon_freq=\"010000\"\r\n    common_dur=\"010000\"\r\n    common_mode=\"'time-averaged'\"\r\n\r\nThe Docker commands were:\r\n \r\n    docker pull docker pull geoschem/gchp_model\r\n    docker run --rm -it -v $HOME/ExtData:/ExtData -v $HOME/OutputDir:/OutputDir geoschem/gchp_model\r\n    mpirun -np 12 -oversubscribe --allow-run-as-root ./geos | tee gchp.log.c48\r\n\r\nTail end of log file:\r\n\r\n     AGCM Date: 2016/07/01  Time: 00:10:00\r\n    \r\n    Program received signal SIGSEGV: Segmentation fault - invalid memory reference.\r\n\r\n    Backtrace for this error:\r\n    --------------------------------------------------------------------------\r\n    Primary job  terminated normally, but 1 process returned\r\n    a non-zero exit code. Per user-direction, the job has been aborted.\r\n    --------------------------------------------------------------------------\r\n    --------------------------------------------------------------------------\r\n    mpirun noticed that process rank 0 with PID 0 on node 22262d174fea exited on signal 11 (Segmentation fault).\r\n    --------------------------------------------------------------------------\r\n\r\n\r\nI then commented out SpeciesConc_avg from the HISTORY.rc file and re-ran.\r\nNow, the only diagnostic active was SpeciesConc_inst.  This also died at 1 hour:\r\n\r\n    AGCM Date: 2016/07/01  Time: 01:00:00\r\n    \r\n     Writing:  11592 Slices (  1 Nodes,  1 PartitionRoot) to File:  OutputDir/GCHP.SpeciesConc_inst.20160701_0100z.nc4\r\n    free(): invalid next size (normal)\r\n    \r\n    Program received signal SIGABRT: Process abort signal.\r\n    Backtrace for this error:\r\n    #0  0x7efd1c1dd2da in ???\r\n    #1  0x7efd1c1dc503 in ???\r\n    ..etc..\r\n \r\n    Times for GIGCenv\r\n    TOTAL                   :       0.726\r\n    INITIALIZE              :       0.000\r\n    RUN                     :       0.723\r\n    ...etc...\r\n    HEMCO::Finalize... OK.\r\n    Chem::State_Diag Finalize... OK.\r\n    Chem::State_Chm Finalize... OK.\r\n    Chem::State_Met Finalize... OK.\r\n    Chem::Input_Opt Finalize... OK.\r\n     Using parallel NetCDF for file: gcchem_internal_checkpoint_c48.nc\r\n    --------------------------------------------------------------------------\r\n    Primary job  terminated normally, but 1 process returned\r\n    a non-zero exit code. Per user-direction, the job has been aborted.\r\n    --------------------------------------------------------------------------\r\n    --------------------------------------------------------------------------\r\n    mpirun noticed that process rank 6 with PID 0 on node 22262d174fea exited on signal 6 (Aborted).\r\n    --------------------------------------------------------------------------\r\n\r\nThis message:\r\n\r\n    free(): invalid next size (normal)\r\n\t\r\nmight be indicative of an out-of-bounds error, perhaps where we deallocate arrays (or fields of State_* objects)."}}
{"actor":{"display_login":"yantosca"}}