{"push_id":2806328963,"size":1,"distinct_size":1,"ref":"refs/heads/_v5__powerpc_topology_Get_topology_for_shared_processors_at_boot_mbox_next","head":"e2d338983252369862b97ab5170a770a0f55ee46","before":"64532a6986b7049f3ab8c0b92c5fc1d91b9712f6","commits":[{"sha":"e2d338983252369862b97ab5170a770a0f55ee46","author":{"name":"Srikar Dronamraju","email":"1f69195f6d934d4caf5816234bde319316df237e@linux.vnet.ibm.com"},"message":"powerpc/topology: Get topology for shared processors at boot\n\nOn a shared lpar, Phyp will not update the cpu associativity at boot\ntime. Just after the boot system does recognize itself as a shared lpar and\ntrigger a request for correct cpu associativity. But by then the scheduler\nwould have already created/destroyed its sched domains.\n\nThis causes\n- Broken load balance across Nodes causing islands of cores.\n- Performance degradation esp if the system is lightly loaded\n- dmesg to wrongly report all cpus to be in Node 0.\n- Messages in dmesg saying borken topology.\n- With commit 051f3ca02e46 (\"sched/topology: Introduce NUMA identity\n  node sched domain\"), can cause rcu stalls at boot up.\n\n>From a scheduler maintainer's perspective, moving cpus from one node to\nanother or creating more numa levels after boot is not appropriate\nwithout some notification to the user space.\nhttps://lore.kernel.org/lkml/20150406214558.GA38501@linux.vnet.ibm.com/T/#u\n\nThe sched_domains_numa_masks table which is used to generate cpumasks is\nonly created at boot time just before creating sched domains and never\nupdated.  Hence, its better to get the topology correct before the sched\ndomains are created.\n\nFor example on 64 core Power 8 shared lpar, dmesg reports\n\n[    2.088360] Brought up 512 CPUs\n[    2.088368] Node 0 CPUs: 0-511\n[    2.088371] Node 1 CPUs:\n[    2.088373] Node 2 CPUs:\n[    2.088375] Node 3 CPUs:\n[    2.088376] Node 4 CPUs:\n[    2.088378] Node 5 CPUs:\n[    2.088380] Node 6 CPUs:\n[    2.088382] Node 7 CPUs:\n[    2.088386] Node 8 CPUs:\n[    2.088388] Node 9 CPUs:\n[    2.088390] Node 10 CPUs:\n[    2.088392] Node 11 CPUs:\n...\n[    3.916091] BUG: arch topology borken\n[    3.916103]      the DIE domain not a subset of the NUMA domain\n[    3.916105] BUG: arch topology borken\n[    3.916106]      the DIE domain not a subset of the NUMA domain\n...\n\nnumactl/lscpu output will still be correct with cores spreading across\nall nodes.\n\nSocket(s):             64\nNUMA node(s):          12\nModel:                 2.0 (pvr 004d 0200)\nModel name:            POWER8 (architected), altivec supported\nHypervisor vendor:     pHyp\nVirtualization type:   para\nL1d cache:             64K\nL1i cache:             32K\nNUMA node0 CPU(s): 0-7,32-39,64-71,96-103,176-183,272-279,368-375,464-471\nNUMA node1 CPU(s): 8-15,40-47,72-79,104-111,184-191,280-287,376-383,472-479\nNUMA node2 CPU(s): 16-23,48-55,80-87,112-119,192-199,288-295,384-391,480-487\nNUMA node3 CPU(s): 24-31,56-63,88-95,120-127,200-207,296-303,392-399,488-495\nNUMA node4 CPU(s):     208-215,304-311,400-407,496-503\nNUMA node5 CPU(s):     168-175,264-271,360-367,456-463\nNUMA node6 CPU(s):     128-135,224-231,320-327,416-423\nNUMA node7 CPU(s):     136-143,232-239,328-335,424-431\nNUMA node8 CPU(s):     216-223,312-319,408-415,504-511\nNUMA node9 CPU(s):     144-151,240-247,336-343,432-439\nNUMA node10 CPU(s):    152-159,248-255,344-351,440-447\nNUMA node11 CPU(s):    160-167,256-263,352-359,448-455\n\nCurrently on this lpar, the scheduler detects 2 levels of Numa and\ncreated numa sched domains for all cpus, but it finds a single DIE\ndomain consisting of all cpus. Hence it deletes all numa sched domains.\n\nTo address this, detect the shared processor and update topology soon after\ncpus are setup so that correct topology is updated just before scheduler\ncreates sched domain.\n\nWith the fix, dmesg reports\n\n[    0.491336] numa: Node 0 CPUs: 0-7 32-39 64-71 96-103 176-183 272-279 368-375 464-471\n[    0.491351] numa: Node 1 CPUs: 8-15 40-47 72-79 104-111 184-191 280-287 376-383 472-479\n[    0.491359] numa: Node 2 CPUs: 16-23 48-55 80-87 112-119 192-199 288-295 384-391 480-487\n[    0.491366] numa: Node 3 CPUs: 24-31 56-63 88-95 120-127 200-207 296-303 392-399 488-495\n[    0.491374] numa: Node 4 CPUs: 208-215 304-311 400-407 496-503\n[    0.491379] numa: Node 5 CPUs: 168-175 264-271 360-367 456-463\n[    0.491384] numa: Node 6 CPUs: 128-135 224-231 320-327 416-423\n[    0.491389] numa: Node 7 CPUs: 136-143 232-239 328-335 424-431\n[    0.491394] numa: Node 8 CPUs: 216-223 312-319 408-415 504-511\n[    0.491399] numa: Node 9 CPUs: 144-151 240-247 336-343 432-439\n[    0.491404] numa: Node 10 CPUs: 152-159 248-255 344-351 440-447\n[    0.491409] numa: Node 11 CPUs: 160-167 256-263 352-359 448-455\n\nand lscpu would also report\n\nSocket(s):             64\nNUMA node(s):          12\nModel:                 2.0 (pvr 004d 0200)\nModel name:            POWER8 (architected), altivec supported\nHypervisor vendor:     pHyp\nVirtualization type:   para\nL1d cache:             64K\nL1i cache:             32K\nNUMA node0 CPU(s): 0-7,32-39,64-71,96-103,176-183,272-279,368-375,464-471\nNUMA node1 CPU(s): 8-15,40-47,72-79,104-111,184-191,280-287,376-383,472-479\nNUMA node2 CPU(s): 16-23,48-55,80-87,112-119,192-199,288-295,384-391,480-487\nNUMA node3 CPU(s): 24-31,56-63,88-95,120-127,200-207,296-303,392-399,488-495\nNUMA node4 CPU(s):     208-215,304-311,400-407,496-503\nNUMA node5 CPU(s):     168-175,264-271,360-367,456-463\nNUMA node6 CPU(s):     128-135,224-231,320-327,416-423\nNUMA node7 CPU(s):     136-143,232-239,328-335,424-431\nNUMA node8 CPU(s):     216-223,312-319,408-415,504-511\nNUMA node9 CPU(s):     144-151,240-247,336-343,432-439\nNUMA node10 CPU(s):    152-159,248-255,344-351,440-447\nNUMA node11 CPU(s):    160-167,256-263,352-359,448-455\n\nPrevious attempt to solve this problem\nhttps://patchwork.ozlabs.org/patch/530090/\n\nReported-by: Manjunatha H R <manjuhr1@in.ibm.com>\nSigned-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>","distinct":true,"url":"https://api.github.com/repos/ruscur/linux/commits/e2d338983252369862b97ab5170a770a0f55ee46"}]}
{"actor":{"display_login":"ruscur"}}