<p>Hadoop doesn't really do what you want it to do.  There might be a way to define your own <a href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/InputFormat.html" rel="nofollow">InputFormat</a> (and probably override some other classes as well) to force it to do what you want, but I can't really recommend that.</p>

<p>The <a href="http://en.wikipedia.org/wiki/Map_%28parallel_pattern%29" rel="nofollow">map()</a> part of <a href="http://en.wikipedia.org/wiki/MapReduce" rel="nofollow">MapReduce</a> fundamentally relies on being able to decompose a problem "into independent subtasks, requiring no communication/synchronization between the subtasks".  If your problem has input data that's a single record that can grow arbitrarily large, and cannot be broken up, MapReduce is fundamentally the wrong conceptual approach because you're not doing any decomposition.</p>

<p>The way I'd think this could be decomposed (if you're speaking of the normal evolutionary algorithm) is to break it up by record (individual, in this case), and your file would be a collection of records.  You could then split the file up by record.  Depending on your file format, you can create an <code>InputFormat</code>, if necessary, so that it knows how to split the file up.  Potentially this will result in rather large records, so you may want to tune your block size to be around the average size of your records, for better distribution.</p>

<p>It looked like others did their generations either as separate jobs or in the reducer, and not in the mapper as you are proposing.  You might read these papers on the topic.</p>

<ul>
<li><a href="http://www0.cs.ucl.ac.uk/staff/F.Sarro/resource/papers/C2.pdf" rel="nofollow">A Parallel Genetic Algorithm Based on Hadoop MapReduce for the
Automatic Generation of JUnit Test Suite - Linda Di Geronimo, Filomena Ferrucci, Alfonso Murolo, Federica Sarro</a></li>
<li><a href="http://www.researchgate.net/profile/Abdulhamit_Subasi/publication/258858471_Parallelization_of_genetic_algorithms_using_Hadoop_MapReduce/links/0046352945f2358517000000.pdf" rel="nofollow">Parallelization of genetic algorithms using Hadoop Map/Reduce - Dino Kečo, Abdulhamit Subasi</a></li>
<li><a href="http://arxiv.org/pdf/1312.0086.pdf" rel="nofollow">A Framework for Genetic Algorithms Based on Hadoop - Filomena Ferrucci, M-Tahar Kechadi, Pasquale Salza, Federica Sarro
</a></li>
<li><a href="http://ieeexplore.ieee.org/xpl/login.jsp?tp=&amp;arnumber=5362925" rel="nofollow">Scaling Genetic Algorithms Using MapReduce - Verma, A.; Llora, X.; Goldberg, D.E.; Campbell, R.H.</a></li>
</ul>

<p>Alternatively, you could use an existing framework.  <a href="https://www.safaribooksonline.com/library/view/apache-mahout-cookbook/9781849518024/ch10.html" rel="nofollow">Apache Mahout Cookbook, Chapter 10</a> describes how the <a href="http://watchmaker.uncommons.org/" rel="nofollow">Watchmaker Framework</a> can be used in <a href="http://mahout.apache.org/" rel="nofollow">Mahout</a> (Hadoop's machine learning library) for evolutionary computation.</p>

<p>You may also find that <a href="http://spark.apache.org/" rel="nofollow">Spark</a> better suits your needs since it has better iterative computation since it keeps more in memory.  There's even native support for evolution algorithms <a href="https://issues.apache.org/jira/browse/SPARK-3830" rel="nofollow">being built</a> for their machine learning library (<a href="http://spark.apache.org/mllib/" rel="nofollow">MLlib</a>).</p>

<p>I hope this doesn't ruin your thesis.</p>
Can I duplicate a data in hdfs (hadoop)
<p>I am new in hadoop (v 2.6.0)
I work in my thesis with genetic algorithm in hadoop (Linux).
My problem :</p>

<p>1:I want to duplicate the file input (text) in hdfs location for all the slaves (not partitioned the file)
for example I have a file (200 Mb) I want to sent all the file to the slaves.(200 for slave 1 and 200 for slave 2 ...etc)
is that possible?if that is possible what is the keys to doing that?</p>

<p>2: second question:I have 2 slaves and one master ... when I start is my program by default execute in all slaves? or hadoop decide what slave will execute the program?if hadoop decide that how i make My program executable in all slaves without exception?i hope that is possible.
because when my program launched I see that it is executed just in slave 2 (not the slave 1.)</p>

<h3>Edit 1 with text from his comment-answer</h3>

<p>thank you for this detaills my data cannot grow arbitrarily large: if i inderstend you right ...well if i have in my data 200 individuals ...it stile 200 indivual with this algorithme...</p>

<p>inside the algorithme if i specific 30 chromosomes .well the algorithme will excute in every node with 200 individuals (in the data base in input file) and with 30 chromosom inside the execution...this parameters will specifie by me before starting my code. they are not a parameter that will be grow arbitrarily large in my algorithme.</p>

<p>can you give me more detaill about InputFormat to start my algorithme)</p>

<h3>Edit 2 with text from his second comment-answer</h3>

<p><a href="http://arxiv.org/pdf/1312.0086.pdf" rel="nofollow">In this document</a> you can find in c:related work in this part said: <code>The existent literature proposes some parallel version of GAs using the MapReduce paradigm. The first is an extension, by adding a second Reducer, of MapReduce named “MRPGA” [6] based on .Net. In this implementation a coordinator client manages the executions of the parallel GA iterations. The chosen model is the island model in which each participating node computes GAs operations for a portion of the entire population. In the first phase, each Mapper node receives its own portion of population and computes the fitness value for each of its individuals. The Reducer nodes of the first reduce phase receive the individuals of the correspondent island and apply the selection function. The final Reducer computes the global selection and the other following GAs functions.</code></p>

<p>That only detaills in this documentabout this approach. the portion of population here is mean the number of chromosoms.(a group of chromosoms named population if you decide de work with 2000 chromosoms and 5 slaves then just specifie 400 chromosoms in the code and every slave will work with just 400 (400*5 =2000)...that s my point.. because if you specifie a 2000 for one node this is very much and take a huge timefor the fitness.you inderstood?the real data that i will partionne it is the chromosoms not the data of input file ,and i want to use a huge number of chromosoms because when you use a big number of chromosoms you will get the approximative solution that u need.</p>