<h2>Best use the one that best <a href="https://arxiv.org/pdf/1704.00726.pdf" rel="nofollow noreferrer">"Fair Divides a Cake with Burnt Parts ... "</a></h2>

<p>As per your domain expertise, the art of performance motivated scheduling is similarly constrained.</p>

<p>Given a few facts above,<br>
the approach to a problem solution is also heavily scale-dependent.</p>

<p>Given a Machine Learning has been added,<br>
the problem is not just a common and trivial <code>[SEQ]-[PAR]-[SEQ]</code> pipeline to process 10 different arrays and sum the partial results.</p>

<h2>The bigger the ML-DataSET gets,<br>the larger will be the "Burnt Parts of the Cake"</h2>

<p>Being inside a realm of <strong><code>numpy</code></strong> / <strong><code>python</code></strong> and their tools for  Machine Learning, your driving forces will not be a will to "parallelise", <strong>but rather the Costs of trying to do so.</strong> Machine Learning pipelines for real-world problems may easily take large units to small tens of hours to get trained, using all, yes ALL localhost available CPU-cores, just for one <strong><code>train</code></strong>-part of the DataSET. The efficiency, professionally built into the <strong><code>numpy</code></strong> / <strong><code>numba</code></strong> / <strong><code>scikit</code></strong> tools is not easy to get much increased further on, so rather do not expect any low hanging fruit to be anywhere near these grounds. For more <a href="https://stackoverflow.com/a/46931460">technical <strong>details on this</strong>, you might like to <strong>read this post</strong></a>.</p>

<p><strong>Memory matters the most.</strong></p>

<p>For small DataSETs, the <code>[PSPACE]</code>-costs in terms of memory footprint will remain easier, but that will also mean, that there will be harder to pay all the <code>[PTIME]</code>-costs, that will have to be paid per each parallel-process, yes the non-productive overheads fee, one always has to pay for just entering the show:
<code>{ instantiation + setup + control + finalisation + dismantling }</code>, which in this very case ( of small DataSETs ) will be hardly justified by any increased processing-performance, potentially coming out from an attempt to operate parallel-processing on too small-scale problems.</p>

<p><strong>Available Processing Resources matter almost the same</strong>  </p>

<p>In case, where a process is ordered to be operated on 10 process instances concurrently, the governing factor is the free-capacity of all resources-classes, that are relevant for such a process smooth execution -- a free ( and best uninterrupted ) CPU-core ( HT-off ), having non-blocked access to sufficient amount of ( free / private >> working set ) RAM memory ( no memory paging, no virtual-memory swaps ) plus minimum ( or fine-tuned ) disk-IO.</p>

<p>In cases, where such resources' capacities are not allowing all the named 10 instances of the Machine Learning process to smoothly execute indeed in <a href="https://stackoverflow.com/revisions/18374629/3"><strong>parallel</strong>, another, "just"-<strong><code>[CONCURRENT]</code></strong></a>, scheduling takes place and <a href="https://stackoverflow.com/a/46124635"><strong>one may straight forget any and all the Amdahl's Law theoretical promises</strong> ( even from the original, "classical", overhead-naive formulation )</a>.</p>

<p><strong>While it is obvious</strong> for anyone, that if a bus, full of tourists, arrives to a five stars Marriott Hotel, equipped with just a pair of lifts from a reception floor, it is simply principally impossible for all such new guests to reach their room at the same time -- <strong>the same problem is not so crystal-clear</strong> in almost exactly the same setup in true-<strong><code>[PARALLEL]</code></strong>-process scheduling and many users start in the very same way - asking "<em>Why my parallel code is slower than serial run?</em>" or "<em>Why is my parallel-code speedup not scaling?</em>".</p>

<p>This said, you will have to maximise your hardware resources-pools and <strong>very carefully balance the parallel-processing</strong> ( setup + termination )-<strong>overhead costs</strong>, because without having at least a chance to pay less than you will potentially gain ( justify the overheads ), going parallel AT ANY COSTS, is simply a devastating policy, whoever has advised you to try to go into this a-priori lost game ( costs/benefits-wise ) direction.</p>
What multiprocessing method should I use for Machine Learning training + accuracy tests?
<p>I have a function that accepts an array and returns a number.  </p>

<p>I want to run this function on 10 different input arrays, and then return the sum of all results.<br>
What is the best way to tell Python to run these 10 calculations in parallel on a computer with 4 cores?</p>

<p>In the documentation there are <a href="https://docs.python.org/3/library/concurrency.html" rel="nofollow noreferrer">many different ways for concurrent execution</a>, and also <a href="https://wiki.python.org/moin/ParallelProcessing" rel="nofollow noreferrer">many different packages for parallel processing</a>. With so many different methods, which of these should I use for the task above?</p>

<p><strong>EDIT:</strong><br>
the function is a Machine-Learning function:<br>
it partitions the input array to two parts - a "<code>train</code>" and a "<code>test</code>" part of the DataSET. It trains a certain classifier on the "<code>train</code>" part, and returns a prediction accuracy of such a trained classifier on the "<code>test</code>" part.</p>