title: How to assign numpy.ndarray to temporary variable under nogil loop in cython?
body: <p>I'm trying to implement <a href="https://github.com/benfred/implicit" rel="nofollow noreferrer">implicit</a> recommender model and have issues with code run time calculating top 5 suggestions to ~11kk users over ~100k items.</p>
  
  <p>I was able to partly solve the problem by numpy with some cython sparkles (in jupyter notebook). Lines with numpy sorting are still using single core:</p>
  
  <pre><code>%%cython -f
  # cython: language_level=3
  # cython: boundscheck=False
  # cython: wraparound=False
  # cython: linetrace=True
  # cython: binding=True
  # distutils: define_macros=CYTHON_TRACE_NOGIL=1
  from cython.parallel import parallel, prange
  import numpy as np
  from tqdm import tqdm
  
  def test(users_items=np.random.rand(11402139//1000, 134751//100)
          , int N=5, show_progress=True, int num_threads=1):
      # Define User count and loops indexes
      cdef int users_c = users_items.shape[0], u, i
      # Predefine zero 2-D C-ordered array for recommendations
      cdef int[:,::1] users_recs = np.zeros((users_c, N), dtype=np.intc)
      for u in tqdm(range(users_c), total=users_c, disable=not show_progress):
          # numpy .dot multiplication using multiple cores
          scores = np.random.rand(134751//1000, 10).dot(np.random.rand(10))
          # numpy partial sort
          ids_partial = np.argpartition(scores, -N)[-N:]
          ids_top = ids_partial[np.argsort(scores[ids_partial])]
          # Fill predefined 2-D array
          for i in range(N):
              users_recs[u, i] = ids_top[i]
      return np.asarray(users_recs)
  # Working example
  tmp = test()
  </code></pre>
  
  <p>I profiled it - np.argpartition consumes 60% of function time and uses onde core. I'm trying to make it parallel, cause I have a server with 80 cores. So, I perform .dot operation on a subset of users (uses multiple cores) and plan to fill empty predefined array by numpy sorting results (which use single core) in parallel, but I'm stuck with error from question title:</p>
  
  <pre><code>%%cython -f
  # cython: language_level=3
  # cython: boundscheck=False
  # cython: wraparound=False
  # cython: linetrace=True
  # cython: binding=True
  # distutils: define_macros=CYTHON_TRACE_NOGIL=1
  from cython.parallel import parallel, prange
  import numpy as np
  from tqdm import tqdm
  from math import ceil
  def test(int N=10, show_progress=True, int num_threads=1):
      # Define User and Item count and loops indexes
      cdef int users_c = 11402139//1000, items_c = 134751//100, u, i, u_b
      # Predefine zero 2-D C-ordered array for recommendations
      cdef int[:,::1] users_recs = np.zeros((users_c, N), dtype=np.intc)
      # Define memoryview var
      cdef float[:,::1] users_items_scores_mv
      progress = tqdm(total=users_c, disable=not show_progress)
      # For a batch of Users
      for u_b in range(5):
          # Use .dot operation which use multiple cores
          users_items_scores = np.random.rand(num_threads, 10).dot(np.random.rand(134751//100, 10).T)
          # Create memory view to 2-D array, which I'm trying to sort row wise
          users_items_scores_mv = users_items_scores
          # Here it starts, try to use numpy sorting in parallel
          for u in prange(num_threads, nogil=True, num_threads=num_threads):
              ids_partial = np.argpartition(users_items_scores_mv[u], items_c-N)[items_c-N:]
              ids_top = ids_partial[np.argsort(users_items_scores_mv[u][ids_partial])]
              # Fill predefined 2-D array
              for i in range(N):
                  users_recs[u_b + u, i] = ids_top[i]
          progress.update(num_threads)
      progress.close()
      return np.asarray(users_recs)
  </code></pre>
  
  <p>and got this (<a href="https://gist.githubusercontent.com/H0bbitBaron/df91614c5b129e5dd92ae009fc30069d/raw/0e20079170f023b45bc4070b3b35fa932d35945c/error.txt" rel="nofollow noreferrer">full error</a>):</p>
  
  <pre><code>Error compiling Cython file:
  ------------------------------------------------------------
  ...
          # Create memory view to 2-D array,
          # which I'm trying to sort row wise
          users_items_scores_mv = users_items_scores
          # Here it starts, try to use numpy sorting in parallel
          for u in prange(num_threads, nogil=True, num_threads=num_threads):
              ids_partial = np.argpartition(users_items_scores_mv[u], items_c-N)[items_c-N:]
             ^
  ------------------------------------------------------------
  
  /datascc/enn/.cache/ipython/cython/_cython_magic_201b296cd5a34240b4c0c6ed3e58de7c.pyx:31:12: Assignment of Python object not allowed without gil
  </code></pre>
  
  <p>I read about memory views and malloc-ating but haven't found example applicable to my situation.</p>