original_docstring:  iterate over the lines of x in blocks of size bs
docstring: iterate over the lines of x in blocks of size bs
docstring_tokens: iterate over the lines of x in blocks of size bs
code: def dataset_iterator(x, preproc, bs):
      """ iterate over the lines of x in blocks of size bs"""
  
      nb = x.shape[0]
      block_ranges = [(i0, min(nb, i0 + bs))
                      for i0 in range(0, nb, bs)]
  
      # def prepare_block((i0, i1)):
      def prepare_block(i0, i1):
          xb = sanitize(x[i0:i1])
          return i0, preproc.apply_py(xb)
  
      return rate_limited_imap(prepare_block, block_ranges)
code_tokens: def dataset_iterator ( x , preproc , bs ) : nb = x . shape [ 0 ] block_ranges = [ ( i0 , min ( nb , i0 + bs ) ) for i0 in range ( 0 , nb , bs ) ] def prepare_block ( i0 , i1 ) : xb = sanitize ( x [ i0 : i1 ] ) return i0 , preproc . apply_py ( xb ) return rate_limited_imap ( prepare_block , block_ranges )
short_docstring: iterate over the lines of x in blocks of size bs
short_docstring_tokens: iterate over the lines of x in blocks of size bs
comment: """ iterate over the lines of x in blocks of size bs"""$SEP$# def prepare_block((i0, i1)):