{"action":"created","issue":{"url":"https://api.github.com/repos/UIUC-PPL/charm/issues/1316","repository_url":"https://api.github.com/repos/UIUC-PPL/charm","labels_url":"https://api.github.com/repos/UIUC-PPL/charm/issues/1316/labels{/name}","comments_url":"https://api.github.com/repos/UIUC-PPL/charm/issues/1316/comments","events_url":"https://api.github.com/repos/UIUC-PPL/charm/issues/1316/events","html_url":"https://github.com/UIUC-PPL/charm/issues/1316","id":432917385,"node_id":"MDU6SXNzdWU0MzI5MTczODU=","number":1316,"title":"AMPI Isomalloc/Mempool abort with PUP sizing mismatch","user":{"login":"stwhite91","id":12042181,"node_id":"MDQ6VXNlcjEyMDQyMTgx","avatar_url":"https://avatars2.githubusercontent.com/u/12042181?v=4","gravatar_id":"","url":"https://api.github.com/users/stwhite91","html_url":"https://github.com/stwhite91","followers_url":"https://api.github.com/users/stwhite91/followers","following_url":"https://api.github.com/users/stwhite91/following{/other_user}","gists_url":"https://api.github.com/users/stwhite91/gists{/gist_id}","starred_url":"https://api.github.com/users/stwhite91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stwhite91/subscriptions","organizations_url":"https://api.github.com/users/stwhite91/orgs","repos_url":"https://api.github.com/users/stwhite91/repos","events_url":"https://api.github.com/users/stwhite91/events{/privacy}","received_events_url":"https://api.github.com/users/stwhite91/received_events","type":"User","site_admin":false},"labels":[{"id":1317804942,"node_id":"MDU6TGFiZWwxMzE3ODA0OTQy","url":"https://api.github.com/repos/UIUC-PPL/charm/labels/AMPI","name":"AMPI","color":"ededed","default":false},{"id":412798323,"node_id":"MDU6TGFiZWw0MTI3OTgzMjM=","url":"https://api.github.com/repos/UIUC-PPL/charm/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":{"login":"stwhite91","id":12042181,"node_id":"MDQ6VXNlcjEyMDQyMTgx","avatar_url":"https://avatars2.githubusercontent.com/u/12042181?v=4","gravatar_id":"","url":"https://api.github.com/users/stwhite91","html_url":"https://github.com/stwhite91","followers_url":"https://api.github.com/users/stwhite91/followers","following_url":"https://api.github.com/users/stwhite91/following{/other_user}","gists_url":"https://api.github.com/users/stwhite91/gists{/gist_id}","starred_url":"https://api.github.com/users/stwhite91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stwhite91/subscriptions","organizations_url":"https://api.github.com/users/stwhite91/orgs","repos_url":"https://api.github.com/users/stwhite91/repos","events_url":"https://api.github.com/users/stwhite91/events{/privacy}","received_events_url":"https://api.github.com/users/stwhite91/received_events","type":"User","site_admin":false},"assignees":[{"login":"stwhite91","id":12042181,"node_id":"MDQ6VXNlcjEyMDQyMTgx","avatar_url":"https://avatars2.githubusercontent.com/u/12042181?v=4","gravatar_id":"","url":"https://api.github.com/users/stwhite91","html_url":"https://github.com/stwhite91","followers_url":"https://api.github.com/users/stwhite91/followers","following_url":"https://api.github.com/users/stwhite91/following{/other_user}","gists_url":"https://api.github.com/users/stwhite91/gists{/gist_id}","starred_url":"https://api.github.com/users/stwhite91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stwhite91/subscriptions","organizations_url":"https://api.github.com/users/stwhite91/orgs","repos_url":"https://api.github.com/users/stwhite91/repos","events_url":"https://api.github.com/users/stwhite91/events{/privacy}","received_events_url":"https://api.github.com/users/stwhite91/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/UIUC-PPL/charm/milestones/7","html_url":"https://github.com/UIUC-PPL/charm/milestone/7","labels_url":"https://api.github.com/repos/UIUC-PPL/charm/milestones/7/labels","id":4227137,"node_id":"MDk6TWlsZXN0b25lNDIyNzEzNw==","number":7,"title":"6.8.0","description":null,"creator":{"login":"pplimport","id":40410281,"node_id":"MDQ6VXNlcjQwNDEwMjgx","avatar_url":"https://avatars2.githubusercontent.com/u/40410281?v=4","gravatar_id":"","url":"https://api.github.com/users/pplimport","html_url":"https://github.com/pplimport","followers_url":"https://api.github.com/users/pplimport/followers","following_url":"https://api.github.com/users/pplimport/following{/other_user}","gists_url":"https://api.github.com/users/pplimport/gists{/gist_id}","starred_url":"https://api.github.com/users/pplimport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pplimport/subscriptions","organizations_url":"https://api.github.com/users/pplimport/orgs","repos_url":"https://api.github.com/users/pplimport/repos","events_url":"https://api.github.com/users/pplimport/events{/privacy}","received_events_url":"https://api.github.com/users/pplimport/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":134,"state":"open","created_at":"2019-04-13T16:51:01Z","updated_at":"2019-04-14T00:34:39Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2016-11-28T23:55:43Z","updated_at":"2019-04-14T00:34:46Z","closed_at":"2016-12-20T05:03:38Z","author_association":"CONTRIBUTOR","body":"*Original issue: https://charm.cs.illinois.edu/redmine/issues/1316*\n\n---\nWhen built without production:\n```\n./charmun  +p3 ./jacobi.iso 2 2 2 +vp8 +balancer RotateLB +LBDebug 1 ++local\r\nCharmrun> scalable start enabled. \r\nCharmrun> started all node programs in 0.003 seconds.\r\nCharm++> Running in non-SMP mode: numPes 3\r\nConverse/Charm++ Commit ID: v6.7.0-454-g5d205de\r\nCharm++> scheduler running in netpoll mode.\r\nCharmLB> Verbose level 1, load balancing period: 0.5 seconds\r\nCharmLB> Load balancer assumes all CPUs are same.\r\nCharm++> Running on 1 unique compute nodes (8-way SMP).\r\nCharm++> cpu topology info is gathered in 0.000 seconds.\r\n[0] RotateLB created\r\niter 1 time: 0.127829 maxerr: 2020.200000\r\niter 2 time: 0.134596 maxerr: 1696.968000\r\niter 3 time: 0.097164 maxerr: 1477.170240\r\niter 4 time: 0.108787 maxerr: 1319.433024\r\niter 5 time: 0.094910 maxerr: 1200.918072\r\niter 6 time: 0.093392 maxerr: 1108.425519\r\niter 7 time: 0.093861 maxerr: 1033.970839\r\niter 8 time: 0.105934 maxerr: 972.509242\r\niter 9 time: 0.109089 maxerr: 920.721889\r\niter 10 time: 0.132259 maxerr: 876.344030\r\n\r\nCharmLB> RotateLB: PE [0] step 0 starting at 1.195512 Memory: 29.244904 MB\r\nCharmLB> RotateLB: PE [0] strategy starting at 1.195622\r\nCharmLB> RotateLB: PE [0] Memory: LBManager: 920 KB CentralLB: 1 KB\r\nCharmLB> RotateLB: PE [0] #Objects migrating: 8, LBMigrateMsg size: 0.00 MB\r\nCharmLB> RotateLB: PE [0] strategy finished at 1.195636 duration 0.000014 s\r\nERROR! Array element claimed it was 18911037 bytes to a sizing PUP::er, but copied 18910837 bytes into the packing PUP::er!\r\n------------- Processor 1 Exiting: Called CmiAbort ------------\r\nReason: Array element's pup routine has a direction mismatch.\r\n\r\nERROR! Array element claimed it was 18908653 bytes to a sizing PUP::er, but copied 18908453 bytes into the packing PUP::er!\r\nERROR! Array element claimed it was 18903037 bytes to a sizing PUP::er, but copied 18902837 bytes into the packing PUP::er!\r\n------------- Processor 0 Exiting: Called CmiAbort ------------\r\nReason: Array element's pup routine has a direction mismatch.\r\n\r\n------------- Processor 2 Exiting: Called CmiAbort ------------\r\nReason: Array element's pup routine has a direction mismatch.\r\n\r\n[1] Stack Traceback:\r\n  [1:0] CmiAbortHelper+0xb3  [0x725e3a]\r\n[0] Stack Traceback:\r\n  [0:0] CmiAbortHelper+0xb3  [0x725e3a]\r\n  [0:1] CmiAbort+0x2d  [0x725e75]\r\n  [0:2] _ZN8CkLocMgr8emigrateEP8CkLocReci+0x26e  [0x64d83c]\r\n  [0:3] _ZN8CkLocRec9migrateMeEi+0x28  [0x6493ae]\r\n  [0:4] _ZN8CkLocRec11recvMigrateEi+0x2c  [0x649988]\r\n  [0:5] _ZN8CkLocRec13staticMigrateE11LDObjHandlei+0x29  [0x649959]\r\n  [0:6] _ZN4LBOM7MigrateE11LDObjHandlei+0x4a  [0x6c96aa]\r\n  [0:7] _ZN4LBDB7MigrateE11LDObjHandlei+0x15a  [0x6c84be]\r\n  [0:8] LDMigrate+0x50  [0x6ae37a]\r\n[2] Stack Traceback:\r\n  [2:0] CmiAbortHelper+0xb3  [0x725e3a]\r\n  [2:1] CmiAbort+0x2d  [0x725e75]\r\n  [2:2] _ZN8CkLocMgr8emigrateEP8CkLocReci+0x26e  [0x64d83c]\r\n  [2:3] _ZN8CkLocRec9migrateMeEi+0x28  [0x6493ae]\r\n  [2:4] _ZN8CkLocRec11recvMigrateEi+0x2c  [0x649988]\r\n  [2:5] _ZN8CkLocRec13staticMigrateE11LDObjHandlei+0x29  [0x649959]\r\n  [2:6] _ZN4LBOM7MigrateE11LDObjHandlei+0x4a  [0x6c96aa]\r\n  [2:7] _ZN4LBDB7MigrateE11LDObjHandlei+0x15a  [0x6c84be]\r\n  [2:8] LDMigrate+0x50  [0x6ae37a]\r\n  [1:1] CmiAbort+0x2d  [0x725e75]\r\n  [1:2] _ZN8CkLocMgr8emigrateEP8CkLocReci+0x26e  [0x64d83c]\r\n  [1:3] _ZN8CkLocRec9migrateMeEi+0x28  [0x6493ae]\r\n  [1:4] _ZN8CkLocRec11recvMigrateEi+0x2c  [0x649988]\r\n  [1:5] _ZN8CkLocRec13staticMigrateE11LDObjHandlei+0x29  [0x649959]\r\n  [1:6] _ZN4LBOM7MigrateE11LDObjHandlei+0x4a  [0x6c96aa]\r\n  [1:7] _ZN4LBDB7MigrateE11LDObjHandlei+0x15a  [0x6c84be]\r\n  [1:8] LDMigrate+0x50  [0x6ae37a]\r\n  [1:9] _ZN10LBDatabase7MigrateE11LDObjHandlei+0x45  [0x6e28b1]\r\n  [1:10] _ZN9CentralLB23ProcessReceiveMigrationEv+0x206  [0x6d503a]\r\n  [0:9] _ZN10LBDatabase7MigrateE11LDObjHandlei+0x45  [0x6e28b1]\r\n  [0:10] _ZN9CentralLB23ProcessReceiveMigrationEv+0x206  [0x6d503a]\r\n  [0:11] _ZN17CkIndex_CentralLB47_call_redn_wrapper_ProcessReceiveMigration_voidEPvS0_+0x35  [0x6d89c7]\r\n  [0:12] CkDeliverMessageFree+0x4e  [0x62c0a1]\r\n  [0:13]   [0x62c1eb]\r\n  [0:14]   [0x62c306]\r\n  [0:15]   [0x62da4a]\r\n  [0:16]   [0x62daf5]\r\n  [0:17] _Z15_processHandlerPvP11CkCoreState+0x126  [0x62df7d]\r\n  [0:18] CmiHandleMessage+0x4d  [0x72c9c2]\r\n  [2:9] _ZN10LBDatabase7MigrateE11LDObjHandlei+0x45  [0x6e28b1]\r\n  [2:10] _ZN9CentralLB23ProcessReceiveMigrationEv+0x206  [0x6d503a]\r\n  [2:11] _ZN17CkIndex_CentralLB47_call_redn_wrapper_ProcessReceiveMigration_voidEPvS0_+0x35  [0x6d89c7]\r\n  [2:12] CkDeliverMessageFree+0x4e  [0x62c0a1]\r\n  [2:13]   [0x62c1eb]\r\n  [2:14]   [0x62c306]\r\n  [2:15]   [0x62da4a]\r\n  [2:16]   [0x62daf5]\r\n  [2:17] _Z15_processHandlerPvP11CkCoreState+0x126  [0x62df7d]\r\n  [2:18] CmiHandleMessage+0x4d  [0x72c9c2]\r\n  [1:11] _ZN17CkIndex_CentralLB47_call_redn_wrapper_ProcessReceiveMigration_voidEPvS0_+0x35  [0x6d89c7]\r\n  [1:12] CkDeliverMessageFree+0x4e  [0x62c0a1]\r\n  [1:13]   [0x62c1eb]\r\n  [1:14]   [0x62c306]\r\n  [1:15]   [0x62da4a]\r\n  [1:16]   [0x62daf5]\r\n  [1:17] _Z15_processHandlerPvP11CkCoreState+0x126  [0x62df7d]\r\n  [1:18] CmiHandleMessage+0x4d  [0x72c9c2]\r\n  [1:19] CsdScheduleForever+0xad  [0x72cc43]\r\n  [1:20] CsdScheduler+0x16  [0x72cb74]\r\n  [0:19] CsdScheduleForever+0xad  [0x72cc43]\r\n  [0:20] CsdScheduler+0x16  [0x72cb74]\r\n  [0:21]   [0x725c3c]\r\n  [0:22] ConverseInit+0x324  [0x725b46]\r\n  [0:23] main+0x3f  [0x61b01a]\r\n  [0:24] __libc_start_main+0xf5  [0x2aaaab511f45]\r\n  [0:25]   [0x58ccc9]\r\nFatal error on PE 0> Array element's pup routine has a direction mismatch.\r\n\n```\n"},"comment":{"url":"https://api.github.com/repos/UIUC-PPL/charm/issues/comments/482907324","html_url":"https://github.com/UIUC-PPL/charm/issues/1316#issuecomment-482907324","issue_url":"https://api.github.com/repos/UIUC-PPL/charm/issues/1316","id":482907324,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4MjkwNzMyNA==","user":{"login":"stwhite91","id":12042181,"node_id":"MDQ6VXNlcjEyMDQyMTgx","avatar_url":"https://avatars2.githubusercontent.com/u/12042181?v=4","gravatar_id":"","url":"https://api.github.com/users/stwhite91","html_url":"https://github.com/stwhite91","followers_url":"https://api.github.com/users/stwhite91/followers","following_url":"https://api.github.com/users/stwhite91/following{/other_user}","gists_url":"https://api.github.com/users/stwhite91/gists{/gist_id}","starred_url":"https://api.github.com/users/stwhite91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stwhite91/subscriptions","organizations_url":"https://api.github.com/users/stwhite91/orgs","repos_url":"https://api.github.com/users/stwhite91/repos","events_url":"https://api.github.com/users/stwhite91/events{/privacy}","received_events_url":"https://api.github.com/users/stwhite91/received_events","type":"User","site_admin":false},"created_at":"2019-04-14T00:34:46Z","updated_at":"2019-04-14T00:34:46Z","author_association":"CONTRIBUTOR","body":"*Original date: 2016-12-12 19:39:06*\n\n---\nThe issue is a memory leak, which is caught by the runtime resulting in an abort when --enable-error-checking is not used at build time.\r\n\n```\nBuild:\r\n$ ./build AMPI netlrts-linux-x86_64 -j16 --without-romio -g -O0\r\n\r\nRun: \r\n$ cd examples/ampi/Cjacobi3D\r\n$ ./charmrun +p2 ./jacobi.iso 1 1 1 +vp1 +balancer RotateLB +LBDebug 1 ++local\n```\n\r\nValgrind reports the an invalid read inside CmiIsomallocBlockListPup when using the mempool implementation:\n```\n==31215== Invalid read of size 1\r\n==31215==    at 0x4C2F950: memcpy``GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)\r\n==31215==    by 0x69D19B: PUP::toMem::bytes(void*, unsigned long, unsigned long, PUP::dataType) (pup_util.C:158)\r\n==31215==    by 0x693FD3: CmiIsomallocBlockListPup (isomalloc.c:2448)\r\n==31215==    by 0x4D4408: TCharm::pupThread(PUP::er&) (tcharm.C:324)\r\n==31215==    by 0x4D4897: TCharm::pup(PUP::er&) (tcharm.C:302)\r\n==31215==    by 0x5A3069: CkLocMgr::pupElementsFor(PUP::er&, CkLocRec*, CkElementCreation_t, bool) [clone .constprop.447] (cklocation.C:2996)\r\n==31215==    by 0x5A1D97: CkLocMgr::emigrate(CkLocRec*, int) (cklocation.C:3101)\r\n==31215==    by 0x5A1FDC: migrateMe (cklocation.C:1882)\r\n==31215==    by 0x5A1FDC: recvMigrate (cklocation.C:2018)\r\n==31215==    by 0x5A1FDC: CkLocRec::staticMigrate(LDObjHandle, int) (cklocation.C:2011)\r\n==31215==    by 0x626081: Migrate (LBOM.h:37)\r\n==31215==    by 0x626081: LBDB::Migrate(LDObjHandle, int) (LBDBManager.C:324)\r\n==31215==    by 0x6075DB: LDMigrate (lbdb.C:392)\r\n==31215==    by 0x6359DE: Migrate (LBDatabase.h:340)\r\n==31215==    by 0x6359DE: CentralLB::ProcessReceiveMigration() (CentralLB.C:1120)\r\n==31215==    by 0x635BAB: CkIndex_CentralLB::_call_redn_wrapper_ProcessReceiveMigration_void(void*, void*) (CentralLB.def.h:788)\r\n==31215==  Address 0x18000fc839 is not stack'd, malloc'd or (recently) free'd\n```\n\r\nWe can see the issue here, where the mismatched slots are noted:\n```\nIn ampi::pup, p.size       = 3528\r\nIn ampiParent::pup, p.size = 18801\r\n\r\njj = 0, ii =  0, curr slot size = 2097152\r\njj = 0, ii =  1, sizeof(slot_header) = 56\r\njj = 0, ii =  2, sizeof(slot_header) = 56\r\njj = 0, ii =  3, sizeof(slot_header) = 56\r\njj = 0, ii =  4, sizeof(slot_header) = 56\r\njj = 0, ii =  5, sizeof(slot_header) = 56\r\njj = 0, ii =  6, sizeof(slot_header) = 56\r\njj = 0, ii =  7, sizeof(slot_header) = 56\r\njj = 0, ii =  8, curr slot size      = 4096\r\njj = 0, ii =  9, curr slot size      = 256                <- MISMATCH HERE (SIZING)\r\njj = 0, ii = 10, sizeof(slot_header) = 56\r\njj = 0, ii = 11, sizeof(slot_header) = 56\r\njj = 0, ii = 12, sizeof(slot_header) = 56\r\njj = 0, ii = 13, sizeof(slot_header) = 56\r\njj = 0, ii = 14, curr slot size      = 128\r\njj = 0, ii = 15, curr slot size      = 128\r\njj = 0, ii = 16, curr slot size      = 128\r\njj = 0, ii = 17, curr slot size      = 256\r\njj = 0, ii = 18, sizeof(slot_header) = 56\r\njj = 0, ii = 19, curr slot size      = 128\r\njj = 0, ii = 20, curr slot size      = 128\r\njj = 0, ii = 21, curr slot size      = 256\r\njj = 0, ii = 22, curr slot size      = 128\r\njj = 0, ii = 23, sizeof(slot_header) = 56\r\n\r\njj = 1, ii = 0,  curr slot size      = 16777216\r\njj = 1, ii = 1,  sizeof(slot_header) = 56\r\njj = 1, ii = 2,  sizeof(slot_header) = 56\r\njj = 1, ii = 3,  sizeof(slot_header) = 56\r\njj = 1, ii = 4,  sizeof(slot_header) = 56\r\njj = 1, ii = 5,  sizeof(slot_header) = 56\r\njj = 1, ii = 6,  sizeof(slot_header) = 56\r\njj = 1, ii = 7,  sizeof(slot_header) = 56\r\njj = 1, ii = 8,  sizeof(slot_header) = 56\r\njj = 1, ii = 9,  sizeof(slot_header) = 56\r\njj = 1, ii = 10, sizeof(slot_header) = 56\r\njj = 1, ii = 11, sizeof(slot_header) = 56\r\njj = 1, ii = 12, sizeof(slot_header) = 56\r\njj = 1, ii = 13, sizeof(slot_header) = 56\r\n\r\nIn CmiIsomallocBlockListPup, p.size() = 18901281, p was sizing\r\n\r\nIn ampi::pup, p.size       = 3528\r\nIn ampiParent::pup, p.size = 18801\r\n\r\njj = 0, ii = 0,  curr slot size      = 2097152\r\njj = 0, ii = 1,  sizeof(slot_header) = 56\r\njj = 0, ii = 2,  sizeof(slot_header) = 56\r\njj = 0, ii = 3,  sizeof(slot_header) = 56\r\njj = 0, ii = 4,  sizeof(slot_header) = 56\r\njj = 0, ii = 5,  sizeof(slot_header) = 56\r\njj = 0, ii = 6,  sizeof(slot_header) = 56\r\njj = 0, ii = 7,  sizeof(slot_header) = 56\r\njj = 0, ii = 8,  curr slot size      = 4096\r\njj = 0, ii = 9,  sizeof(slot_header) = 56               <- ... AND HERE (PACKING)\r\njj = 0, ii = 10, sizeof(slot_header) = 56\r\njj = 0, ii = 11, curr slot size      = 128\r\njj = 0, ii = 12, curr slot size      = 128\r\njj = 0, ii = 13, curr slot size      = 128\r\njj = 0, ii = 14, curr slot size      = 256\r\njj = 0, ii = 15, sizeof(slot_header) = 56\r\njj = 0, ii = 16, curr slot size      = 128\r\njj = 0, ii = 17, curr slot size      = 128\r\njj = 0, ii = 18, curr slot size      = 256\r\njj = 0, ii = 19, curr slot size      = 128\r\njj = 0, ii = 20, sizeof(slot_header) = 56\r\n\r\njj = 1, ii = 0,  curr slot size      = 16777216\r\njj = 1, ii = 1,  sizeof(slot_header) = 56\r\njj = 1, ii = 2,  sizeof(slot_header) = 56\r\njj = 1, ii = 3,  sizeof(slot_header) = 56\r\njj = 1, ii = 4,  sizeof(slot_header) = 56\r\njj = 1, ii = 5,  sizeof(slot_header) = 56\r\njj = 1, ii = 6,  sizeof(slot_header) = 56\r\njj = 1, ii = 7,  sizeof(slot_header) = 56\r\njj = 1, ii = 8,  sizeof(slot_header) = 56\r\njj = 1, ii = 9,  sizeof(slot_header) = 56\r\njj = 1, ii = 10, sizeof(slot_header) = 56\r\njj = 1, ii = 11, sizeof(slot_header) = 56\r\njj = 1, ii = 12, sizeof(slot_header) = 56\r\njj = 1, ii = 13, sizeof(slot_header) = 56\r\n\r\nIn CmiIsomallocBlockListPup, p.size() = 18900889, p was packing\r\n\r\nERROR! Array element claimed it was 18902901 bytes to a sizing PUP::er, but copied 18902509 bytes into the packing PUP::er!\r\n\n```\n"}}
{"actor":{"display_login":"stwhite91"}}