{"push_id":1043958679,"size":5,"distinct_size":5,"ref":"refs/heads/thread-local-abi-v6-4.5","head":"5674b06cf90c7ca3320a40422387cd536efa2501","before":"3113bdfb97dfee618754a64f5eacf656471a3a8e","commits":[{"sha":"0e005cfac215b43554065a955a32fce7697cb05f","author":{"email":"007bef6a47fb57dff675bfa1b0b798b4bdfbfc25@efficios.com","name":"Mathieu Desnoyers"},"message":"Thread local ABI system call: cache CPU number of running thread (v6)\n\nExpose a new system call allowing threads to register one userspace\nmemory area where to store the CPU number on which the calling thread is\nrunning. Scheduler migration sets the TIF_NOTIFY_RESUME flag on the\ncurrent thread. Upon return to user-space, a notify-resume handler\nupdates the current CPU value within each registered user-space memory\narea. User-space can then read the current CPU number directly from\nmemory.\n\nThis thread-local ABI can be extended to add features in the future.\nOne future feature extension is the restartable critical sections\n(percpu atomics) work undergone by Paul Turner and Andrew Hunter,\nwhich lets the kernel handle restart of critical sections. [1] [2]\n\nThis cpu id cache is an improvement over current mechanisms available to\nread the current CPU number, which has the following benefits:\n\n- 44x speedup on ARM vs system call through glibc,\n- 20x speedup on x86 compared to calling glibc, which calls vdso\n  executing a \"lsl\" instruction,\n- 16x speedup on x86 compared to inlined \"lsl\" instruction,\n- Unlike vdso approaches, this cached value can be read from an inline\n  assembly, which makes it a useful building block for restartable\n  sequences.\n- The cpu id cache approach is portable (e.g. ARM), which is not the\n  case for the lsl-based x86 vdso.\n\nOn x86, yet another possible approach would be to use the gs segment\nselector to point to user-space per-cpu data. This approach performs\nsimilarly to the cpu id cache, but it has two disadvantages: it is\nnot portable, and it is incompatible with existing applications already\nusing the gs segment selector for other purposes.\n\nBenchmarking various approaches for reading the current CPU number:\n\nARMv7 Processor rev 10 (v7l)\nMachine model: Wandboard i.MX6 Quad Board\n- Baseline (empty loop):                                   10.1 ns\n- Read CPU from __thread_local_abi.cpu_id:                 10.1 ns\n- Read CPU from __thread_local_abi.cpu_id (lazy register): 12.4 ns\n- glibc 2.19-0ubuntu6.6 getcpu:                           445.6 ns\n- getcpu system call:                                     322.2 ns\n\nx86-64 Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz:\n- Baseline (empty loop):                                    0.8 ns\n- Read CPU from __thread_local_abi.cpu_id:                  0.8 ns\n- Read CPU from __thread_local_abi.cpu_id (lazy register):  1.6 ns\n- Read using gs segment selector:                           0.8 ns\n- \"lsl\" inline assembly:                                   13.0 ns\n- glibc 2.19-0ubuntu6 getcpu:                              16.5 ns\n- getcpu system call:                                      52.5 ns\n\n- Speed\n\nRunning 10 runs of hackbench -l 100000 seems to indicate that the sched\nswitch impact of this new configuration option is within the noise:\n\nConfiguration: 2 sockets * 8-core Intel(R) Xeon(R) CPU E5-2630 v3 @\n2.40GHz (directly on hardware, hyperthreading disabled in BIOS, energy\nsaving disabled in BIOS, turboboost disabled in BIOS, cpuidle.off=1\nkernel parameter), with a Linux v4.5 defconfig+localyesconfig,\nthread-local ABI series applied.\n\nSetup:\n\n* CONFIG_GETCPU_CACHE=n\n\navg.:      40.42 s\nstd.dev.:   0.29 s\n\n* CONFIG_GETCPU_CACHE=y\n\navg.:      40.60 s\nstd.dev.:   0.17 s\n\n- Size\n\nOn x86-64, between CONFIG_THREAD_LOCAL_ABI_CPU_ID=n/y, the text size\nincrease of vmlinux is 640 bytes, and the data size increase of vmlinux\nis 512 bytes.\n\n* CONFIG_THREAD_LOCAL_ABI_CPU_ID=n\n   text            data            bss             dec            hex        filename\n17018635        2762368        1564672        21345675        145b58b        vmlinux\n\n* CONFIG_THREAD_LOCAL_ABI_CPU_ID=y\n   text            data            bss             dec            hex        filename\n17019275        2762880         1564672       21346827        145ba0b        vmlinux\n\n[1] https://lwn.net/Articles/650333/\n[2] http://www.linuxplumbersconf.org/2013/ocw/system/presentations/1695/original/LPC%20-%20PerCpu%20Atomics.pdf\n\nLink: http://lkml.kernel.org/r/20151027235635.16059.11630.stgit@pjt-glaptop.roam.corp.google.com\nLink: http://lkml.kernel.org/r/20150624222609.6116.86035.stgit@kitami.mtv.corp.google.com\nSigned-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\nCC: Thomas Gleixner <tglx@linutronix.de>\nCC: Paul Turner <pjt@google.com>\nCC: Andrew Hunter <ahh@google.com>\nCC: Peter Zijlstra <peterz@infradead.org>\nCC: Andy Lutomirski <luto@amacapital.net>\nCC: Andi Kleen <andi@firstfloor.org>\nCC: Dave Watson <davejwatson@fb.com>\nCC: Chris Lameter <cl@linux.com>\nCC: Ingo Molnar <mingo@redhat.com>\nCC: \"H. Peter Anvin\" <hpa@zytor.com>\nCC: Ben Maurer <bmaurer@fb.com>\nCC: Steven Rostedt <rostedt@goodmis.org>\nCC: \"Paul E. McKenney\" <paulmck@linux.vnet.ibm.com>\nCC: Josh Triplett <josh@joshtriplett.org>\nCC: Linus Torvalds <torvalds@linux-foundation.org>\nCC: Andrew Morton <akpm@linux-foundation.org>\nCC: Russell King <linux@arm.linux.org.uk>\nCC: Catalin Marinas <catalin.marinas@arm.com>\nCC: Will Deacon <will.deacon@arm.com>\nCC: Michael Kerrisk <mtk.manpages@gmail.com>\nCC: Boqun Feng <boqun.feng@gmail.com>\nCC: linux-api@vger.kernel.org\n---\n\nChanges since v1:\n- Return -1, errno=EINVAL if cpu_cache pointer is not aligned on\n  sizeof(int32_t).\n- Update man page to describe the pointer alignement requirements and\n  update atomicity guarantees.\n- Add MAINTAINERS file GETCPU_CACHE entry.\n- Remove dynamic memory allocation: go back to having a single\n  getcpu_cache entry per thread. Update documentation accordingly.\n- Rebased on Linux 4.4.\n\nChanges since v2:\n- Introduce a \"cmd\" argument, along with an enum with GETCPU_CACHE_GET\n  and GETCPU_CACHE_SET. Introduce a uapi header linux/getcpu_cache.h\n  defining this enumeration.\n- Split resume notifier architecture implementation from the system call\n  wire up in the following arch-specific patches.\n- Man pages updates.\n- Handle 32-bit compat pointers.\n- Simplify handling of getcpu_cache GETCPU_CACHE_SET compiler barrier:\n  set the current cpu cache pointer before doing the cache update, and\n  set it back to NULL if the update fails. Setting it back to NULL on\n  error ensures that no resume notifier will trigger a SIGSEGV if a\n  migration happened concurrently.\n\nChanges since v3:\n- Fix __user annotations in compat code,\n- Update memory ordering comments.\n- Rebased on kernel v4.5-rc5.\n\nChanges since v4:\n- Inline getcpu_cache_fork, getcpu_cache_execve, and getcpu_cache_exit.\n- Add new line between if() and switch() to improve readability.\n- Added sched switch benchmarks (hackbench) and size overhead comparison\n  to change log.\n\nChanges since v5:\n- Rename \"getcpu_cache\" to \"thread_local_abi\", allowing to extend\n  this system call to cover future features such as restartable critical\n  sections. Generalizing this system call ensures that we can add\n  features similar to the cpu_id field within the same cache-line\n  without having to track one pointer per feature within the task\n  struct.\n- Add a tlabi_nr parameter to the system call, thus allowing to extend\n  the ABI beyond the initial 64-byte structure by registering structures\n  with tlabi_nr greater than 0. The initial ABI structure is associated\n  with tlabi_nr 0.\n- Rebased on kernel v4.5.\n\nMan page associated:\n\nTHREAD_LOCAL_ABI(2)     Linux Programmer's Manual     THREAD_LOCAL_ABI(2)\n\nNAME\n       thread_local_abi  -  Shared  memory  interface  between user-space\n       threads and the kernel\n\nSYNOPSIS\n       #include <linux/thread_local_abi.h>\n\n       int thread_local_abi(uint32_t tlabi_nr, void * tlabi, uint32_t feature_mask, int flags);\n\nDESCRIPTION\n       The thread_local_abi() accelerates some frequent user-space opera‐\n       tions  by  defining a shared data structure ABI between each user-\n       space thread and the kernel.\n\n       The tlabi_nr argument is the thread-local  ABI  structure  number.\n       Currently, only tlabi_nr 0 is supported.  tlabi_nr 0 expects tlabi\n       to hold a pointer to struct thread_local_abi features and  layout,\n       or NULL.\n\n       The layout of struct thread_local_abi is as follows:\n\n       Structure alignment\n              This  structure  needs to be aligned at least on the archi‐\n              tecture pointer size.\n\n       Structure size\n              This structure has a fixed size of 64 bytes.\n\n       Fields\n\n           features\n              Bitmask of the features enabled for this thread's  tlabi_nr\n              0.\n\n           cpu_id\n              Cache of the CPU number on which the calling thread is run‐\n              ning.\n\n       The tlabi argument is a pointer to the thread-local ABI  structure\n       to  be shared between kernel and user-space. If tlabi is NULL, the\n       currently registered address will be used.\n\n       The feature_mask is a bitmask  of  the  features  to  enable.  For\n       tlabi_nr 0, it is a OR'd mask of the following features:\n\n       TLABI_FEATURE_CPU_ID\n              Cache the CPU number on which the calling thread is running\n              into the cpu_id field of the struct thread_local_abi struc‐\n              ture.\n\n       The flags argument is currently unused and must be specified as 0.\n\n       Typically, a library or application will keep the thread-local ABI\n       in a thread-local storage variable, or other memory areas  belong‐\n       ing to each thread. It is recommended to perform volatile reads of\n       the thread-local cache to prevent the  compiler  from  doing  load\n       tearing.  An  alternative approach is to read the cpu number cache\n       from inline assembly in a single instruction.\n\n       Each thread is responsible for registering  its  thread-local  ABI\n       structure. Only one thread-local ABI structure address can be reg‐\n       istered per thread for each tlabi_nr number. Once set, the thread-\n       local  ABI  address  associated to a tlabi_nr number is idempotent\n       for a given thread.\n\n       The symbol __thread_local_abi is recommended  to  be  used  across\n       libraries  and applications wishing to register a the thread-local\n       ABI structure for tlabi_nr 0. The attribute \"weak\" is  recommended\n       when  declaring  this  variable  in  libraries.   Applications can\n       choose to define their own version of this symbol without the weak\n       attribute as a performance improvement.\n\n       In  a  typical  usage scenario, the thread registering the thread-\n       local ABI structure will be performing reads from that  structure.\n       It  is  however also allowed to read the that structure from other\n       threads. The thread-local ABI field updates performed by the  ker‐\n       nel  provide single-copy atomicity semantics, which guarantee that\n       other threads performing single-copy atomic reads of the cpu  num‐\n       ber cache will always observe a consistent value.\n\n       Memory  registered  as  thread-local ABI structure should never be\n       deallocated before the thread which registered it exits:  specifi‐\n       cally, it should not be freed, and the library containing the reg‐\n       istered thread-local storage should not  be  dlclose'd.  Violating\n       this  constraint may cause a SIGSEGV signal to be delivered to the\n       thread.\n\n       Unregistration of associated thread-local ABI structure is implic‐\n       itly performed when a thread or process exit.\n\nRETURN VALUE\n       A  return  value of 0 indicates success. On error, -1 is returned,\n       and errno is set appropriately.\n\nERRORS\n       EINVAL Either flags is non-zero, an unexpected tlabi_nr  has  been\n              specified,  or  tlabi  contains  an  address  which  is not\n              aligned on the architecture pointer size.\n\n       ENOSYS The thread_local_abi() system call is  not  implemented  by\n              this kernel, or the requested feature is not available.\n\n       EFAULT tlabi is an invalid address.\n\n       EBUSY  The  tlabi  argument contains a non-NULL address which dif‐\n              fers from the memory location already registered  for  this\n              thread for the given tlabi_nr number.\n\n       ENOENT The  tlabi argument is NULL, but no memory location is cur‐\n              rently registered for this thread for  the  given  tlabi_nr\n              number.\n\nVERSIONS\n       The thread_local_abi() system call was added in Linux 4.X (TODO).\n\nCONFORMING TO\n       thread_local_abi() is Linux-specific.\n\nEXAMPLE\n       The following code uses the thread_local_abi() system call to keep\n       a thread local storage variable up to date with  the  current  CPU\n       number,  with  a  fallback  on sched_getcpu(3) if the cache is not\n       available. For example simplicity, it is done in main(), but  mul‐\n       tithreaded  programs  would need to invoke thread_local_abi() from\n       each program thread.\n\n           #define _GNU_SOURCE\n           #include <stdlib.h>\n           #include <stdio.h>\n           #include <unistd.h>\n           #include <stdint.h>\n           #include <sched.h>\n           #include <stddef.h>\n           #include <sys/syscall.h>\n           #include <linux/thread_local_abi.h>\n\n           static inline int\n           thread_local_abi(uint32_t tlabi_nr,\n                   volatile struct thread_local_abi *tlabi,\n                   uint32_t feature_mask, int flags)\n           {\n               return syscall(__NR_thread_local_abi, tlabi_nr, tlabi,\n                       feature_mask, flags);\n           }\n\n           /*\n            * __thread_local_abi is recommended as symbol name for the\n            * thread-local ABI. Weak attribute is recommended when declaring\n            * this variable in libraries.\n            */\n           __thread __attribute__((weak))\n           volatile struct thread_local_abi __thread_local_abi;\n\n           static int\n           tlabi_cpu_id_register(void)\n           {\n               if (thread_local_abi(0, &__thread_local_abi,\n                       TLABI_FEATURE_CPU_ID, 0))\n                   return -1;\n               return 0;\n           }\n\n           static int32_t\n           read_cpu_id(void)\n           {\n               if (!(__thread_local_abi.features & TLABI_FEATURE_CPU_ID))\n                   return sched_getcpu();\n               return __thread_local_abi.cpu_id;\n           }\n\n           int\n           main(int argc, char **argv)\n           {\n               if (tlabi_cpu_id_register()) {\n                   fprintf(stderr,\n                       \"Unable to initialize thread-local ABI cpu_id feature.\\n\");\n                   fprintf(stderr, \"Using sched_getcpu() as fallback.\\n\");\n               }\n               printf(\"Current CPU number: %d\\n\", read_cpu_id());\n               printf(\"TLABI features: 0x%x\\n\", __thread_local_abi.features);\n\n               exit(EXIT_SUCCESS);\n           }\n\nSEE ALSO\n       sched_getcpu(3)\n\nLinux                           2016-01-27            THREAD_LOCAL_ABI(2)","distinct":true,"url":"https://api.github.com/repos/compudj/linux-percpu-dev/commits/0e005cfac215b43554065a955a32fce7697cb05f"},{"sha":"4612569f78ef93318ea1ea8d9d861940433b37f4","author":{"email":"007bef6a47fb57dff675bfa1b0b798b4bdfbfc25@efficios.com","name":"Mathieu Desnoyers"},"message":"Thread local ABI cpu_id: ARM resume notifier\n\nCall the tlabi_cpu_id_handle_notify_resume() function on return to\nuserspace if TIF_NOTIFY_RESUME thread flag is set.\n\nSigned-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\nCC: Russell King <linux@arm.linux.org.uk>\nCC: Catalin Marinas <catalin.marinas@arm.com>\nCC: Will Deacon <will.deacon@arm.com>\nCC: Thomas Gleixner <tglx@linutronix.de>\nCC: Paul Turner <pjt@google.com>\nCC: Andrew Hunter <ahh@google.com>\nCC: Peter Zijlstra <peterz@infradead.org>\nCC: Andy Lutomirski <luto@amacapital.net>\nCC: Andi Kleen <andi@firstfloor.org>\nCC: Dave Watson <davejwatson@fb.com>\nCC: Chris Lameter <cl@linux.com>\nCC: Ingo Molnar <mingo@redhat.com>\nCC: Ben Maurer <bmaurer@fb.com>\nCC: Steven Rostedt <rostedt@goodmis.org>\nCC: \"Paul E. McKenney\" <paulmck@linux.vnet.ibm.com>\nCC: Josh Triplett <josh@joshtriplett.org>\nCC: Linus Torvalds <torvalds@linux-foundation.org>\nCC: Andrew Morton <akpm@linux-foundation.org>\nCC: Boqun Feng <boqun.feng@gmail.com>\nCC: linux-api@vger.kernel.org","distinct":true,"url":"https://api.github.com/repos/compudj/linux-percpu-dev/commits/4612569f78ef93318ea1ea8d9d861940433b37f4"},{"sha":"3da84591b45104ce436104400d1ffd82164b7f96","author":{"email":"007bef6a47fb57dff675bfa1b0b798b4bdfbfc25@efficios.com","name":"Mathieu Desnoyers"},"message":"Thread local ABI: wire up ARM system call\n\nWire up the thread-local ABI system call on 32-bit ARM.\n\nThis provides an ABI improving the speed of a getcpu operation\non ARM by skipping the getcpu system call on the fast path.\n\nThe thread-local ABI can be extended to add features in the future.\n\nSigned-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\nCC: Russell King <linux@arm.linux.org.uk>\nCC: Catalin Marinas <catalin.marinas@arm.com>\nCC: Will Deacon <will.deacon@arm.com>\nCC: Thomas Gleixner <tglx@linutronix.de>\nCC: Paul Turner <pjt@google.com>\nCC: Andrew Hunter <ahh@google.com>\nCC: Peter Zijlstra <peterz@infradead.org>\nCC: Andy Lutomirski <luto@amacapital.net>\nCC: Andi Kleen <andi@firstfloor.org>\nCC: Dave Watson <davejwatson@fb.com>\nCC: Chris Lameter <cl@linux.com>\nCC: Ingo Molnar <mingo@redhat.com>\nCC: Ben Maurer <bmaurer@fb.com>\nCC: Steven Rostedt <rostedt@goodmis.org>\nCC: \"Paul E. McKenney\" <paulmck@linux.vnet.ibm.com>\nCC: Josh Triplett <josh@joshtriplett.org>\nCC: Linus Torvalds <torvalds@linux-foundation.org>\nCC: Andrew Morton <akpm@linux-foundation.org>\nCC: Boqun Feng <boqun.feng@gmail.com>\nCC: linux-api@vger.kernel.org","distinct":true,"url":"https://api.github.com/repos/compudj/linux-percpu-dev/commits/3da84591b45104ce436104400d1ffd82164b7f96"},{"sha":"c6c86a1d547e9339db0ee2d87fce3bea110e346d","author":{"email":"007bef6a47fb57dff675bfa1b0b798b4bdfbfc25@efficios.com","name":"Mathieu Desnoyers"},"message":"Thread local ABI cpu_id: x86 32/64 resume notifier\n\nCall the tlabi_cpu_id_handle_notify_resume() function on return to\nuserspace if TIF_NOTIFY_RESUME thread flag is set.\n\nSigned-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\nCC: Russell King <linux@arm.linux.org.uk>\nCC: Catalin Marinas <catalin.marinas@arm.com>\nCC: Will Deacon <will.deacon@arm.com>\nCC: Thomas Gleixner <tglx@linutronix.de>\nCC: Paul Turner <pjt@google.com>\nCC: Andrew Hunter <ahh@google.com>\nCC: Peter Zijlstra <peterz@infradead.org>\nCC: Andy Lutomirski <luto@amacapital.net>\nCC: Andi Kleen <andi@firstfloor.org>\nCC: Dave Watson <davejwatson@fb.com>\nCC: Chris Lameter <cl@linux.com>\nCC: Ingo Molnar <mingo@redhat.com>\nCC: \"H. Peter Anvin\" <hpa@zytor.com>\nCC: Ben Maurer <bmaurer@fb.com>\nCC: Steven Rostedt <rostedt@goodmis.org>\nCC: \"Paul E. McKenney\" <paulmck@linux.vnet.ibm.com>\nCC: Josh Triplett <josh@joshtriplett.org>\nCC: Linus Torvalds <torvalds@linux-foundation.org>\nCC: Andrew Morton <akpm@linux-foundation.org>\nCC: Boqun Feng <boqun.feng@gmail.com>\nCC: linux-api@vger.kernel.org","distinct":true,"url":"https://api.github.com/repos/compudj/linux-percpu-dev/commits/c6c86a1d547e9339db0ee2d87fce3bea110e346d"},{"sha":"5674b06cf90c7ca3320a40422387cd536efa2501","author":{"email":"007bef6a47fb57dff675bfa1b0b798b4bdfbfc25@efficios.com","name":"Mathieu Desnoyers"},"message":"Thread local ABI: wire up x86 32/64 system call\n\nWire up the thread-local ABI system call on x86 32/64.\n\nThis provides an ABI improving the speed of a getcpu operation\non x86 by removing the need to perform a function call, \"lsl\"\ninstruction, or system call on the fast path.\n\nThe thread-local ABI can be extended to add features in the future.\n\nSigned-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\nCC: Russell King <linux@arm.linux.org.uk>\nCC: Catalin Marinas <catalin.marinas@arm.com>\nCC: Will Deacon <will.deacon@arm.com>\nCC: Thomas Gleixner <tglx@linutronix.de>\nCC: Paul Turner <pjt@google.com>\nCC: Andrew Hunter <ahh@google.com>\nCC: Peter Zijlstra <peterz@infradead.org>\nCC: Andy Lutomirski <luto@amacapital.net>\nCC: Andi Kleen <andi@firstfloor.org>\nCC: Dave Watson <davejwatson@fb.com>\nCC: Chris Lameter <cl@linux.com>\nCC: Ingo Molnar <mingo@redhat.com>\nCC: \"H. Peter Anvin\" <hpa@zytor.com>\nCC: Ben Maurer <bmaurer@fb.com>\nCC: Steven Rostedt <rostedt@goodmis.org>\nCC: \"Paul E. McKenney\" <paulmck@linux.vnet.ibm.com>\nCC: Josh Triplett <josh@joshtriplett.org>\nCC: Linus Torvalds <torvalds@linux-foundation.org>\nCC: Andrew Morton <akpm@linux-foundation.org>\nCC: Boqun Feng <boqun.feng@gmail.com>\nCC: linux-api@vger.kernel.org","distinct":true,"url":"https://api.github.com/repos/compudj/linux-percpu-dev/commits/5674b06cf90c7ca3320a40422387cd536efa2501"}]}