<p>I have been searching around a bit and I think the following is a candidate answer to your question.</p>

<p>Quoting N. Bell and J. Hoberock, "Thrust: a productivity-oriented library for CUDA", in GPU Computing Gems Jade Edition:</p>

<blockquote>
  <p>Thrust statically selects a highly-optimized <strong>Radix Sort</strong> algorithm for sorting primitive types (<code>char</code>, <code>int</code>, <code>float</code> and <code>double</code>) with the standard <code>less</code> and <code>greater</code> comparison operators. For all other types (e.g., user-defined data types) and comparison operators, Thrust uses a general <strong>Merge Sort</strong> algorithm. Because sorting primitives with Radix Sort is considerably faster than Merge Sort, this static optimization has significant value.</p>
</blockquote>

<p>Now, <strong>Merge Sort</strong> requires <code>O(N)</code> memory space, see <a href="https://stackoverflow.com/questions/2967153/space-requirements-of-a-merge-sort">Space requirements of a merge-sort</a>.</p>

<p>Furthermore, <strong>Radix Sort</strong> still requires <code>O(N)</code> memory space, see <a href="http://arxiv.org/pdf/1206.3511.pdf" rel="nofollow noreferrer">Comparison of Bucket Sort and RADIX Sort</a>.</p>

<p>Which of the two consumes more memory is not defined and depends on the input sequence to be sorted as well as on algorithm tuning parameters, see the comments to one of the answers to <a href="https://stackoverflow.com/questions/3539265/why-quicksort-is-more-popular-than-radix-sort">Why quicksort is more popular than radix-sort?</a>.</p>

<p>Opposite to that, <strong>Quick Sort</strong> requires <code>O(logN)</code> memory space if performed <em>in-place</em>, otherwise it needs <code>O(N)</code>. For a CUDA implementation of the Quick Sort algorithm, you may have a look at <a href="http://blogs.nvidia.com/blog/2012/09/12/how-tesla-k20-speeds-up-quicksort-a-familiar-comp-sci-code/" rel="nofollow noreferrer">How Tesla K20 Speeds Quicksort</a>.</p>

<p>For other <em>in-place</em> sorting algorithms (the <em>in-place</em> strategy is worth to be explored as it saves memory as compared to the <em>non-in-place</em> counterpart), have a look at the <strong>Bitonic Sort</strong>, see <a href="http://www.informatik.uni-kiel.de/fileadmin/arbeitsgruppen/comsys/files/public/ppam09.pdf" rel="nofollow noreferrer">Fast in-place sorting with CUDA based on bitonic sort</a>.</p>
Memory saving alternatives to CUDA Thrust sort_by_key?
<p>In <a href="https://stackoverflow.com/questions/13306793/cuda-thrust-trying-to-sort-by-key-2-8gb-of-data-in-6gb-of-gpu-ram-throws-bad-al">cuda/thrust: Trying to sort_by_key 2.8GB of data in 6GB of GPU RAM throws bad_alloc</a>, I have read that <code>sort_by_key</code> consumed most of the memory for the test case considered therein.</p>

<p>Is there an alternative that can do exactly what <code>sort_by_key</code> is doing even if it is a little bit slower but that can sort bigger datasets?</p>