title: How do I ensure that the (small) data set is replicated to all nodes?
body: <p>I have a small data set <code>mysmall</code> which I need when streaming-processing a large set <code>mylarge</code>.</p>
  
  <p>Right now I do</p>
  
  <pre><code>hadoop fs -get mysmall
  hadoop jar hadoop-streaming.jar -files mysmall,myscript.py \
    -reducer ... -input mylarge -output ... \
    -mapper "python myscript.py mysmall"
  </code></pre>
  
  <p>which seems suboptimal - I collect <code>mysmall</code> from hadoop and then
  redistribute it to all the nodes using the <code>-files</code> option.</p>
  
  <p>It would seem that there should be a way to tell hadoop to replicate <code>mysmall</code> to all the nodes and then access it from <code>myscript.py</code> using
  hadoop path.</p>
  
  <p>Is it possible to do that?</p>
  
  <p>PS. When I say "<em>small data set</em>", I mean a range from a single record (less than 1kB) to 200k records (4MB).</p>