of the SIMD (Single Instruction Multiple Data) and MIMD (Multiple Instruction Multiple Data), where the first reigned in the 80’s with the vector computers, and the later is the commonplace of almost every computing device nowadays, from cellphones to supercomputers.

Using SIMT paradigm, the parallel algorithm development changes greatly since it is possible to code in a one-thread-per-cell fashion. The thread creation, switching and destruction have such a low performance impact that doing a matrix scaling reduces to launch one kernel per matrix cell, even if the matrix is $32768\times 32768$ of single precision floating point numbers summing up 1 GThread all proceeding in parallel. In fact, for the implementation, the more threads the better, since the high memory latency to global memory (in the order of 200 cycles) is hidden by swapping out warps (vectors of 32 threads that execute synchronously) waiting for the memory to become available.

It is important to emphasize the role of blocks in the SIMT model. Threads are divided into blocks, where each block of threads have two special features: a private shared memory and the ability to barrier synchronize. Using these capabilities, the shared memory can be used as a manually-managed cache that 