title: Hadoop: Having threads inside map function
body: <p>Can i have threads inside map function? I have a task were having threads could really help me. I need to concurrently add values to a hashmap for every input line. My input line becomes an array of string and for every value of this array, i need to add it to the hash map. I later use this hashmap in the cleanup function.</p>
  
  <p>I am doing this with a for loop and it seems that is the bottleneck for my project. So i thought of using a concurrent hash map and splitting the array of strings into several smaller arrays. So every thread would be responsible for adding the corresponding "smaller" array inside the hashmap. The thing is that i have implement it in a local java application and it works. When i use it inside hadoop, the results are not the ones expected. I am using Thread.join() for every thread so that for every line of input i make sure that the threads have finished before the next line. Well that is what i thought i did. Does hadoop treats threads with a special way?</p>
  
  <pre><code>   edits for duffymo
  </code></pre>
  
  <p>Here is the google citation <a href="http://research.google.com/pubs/pub36296.html" rel="nofollow">http://research.google.com/pubs/pub36296.html</a> .</p>
  
  <p>Algorithm 2 is the part that i am talking about. As you can see there is a for loop for every attribute and that for every attribute i need to update the in memory structure. They only have to predict one value in their approach (single label learning), were in mine i might have many values to predict (multi label learning). So what google says y value, for them is a 3 value array. For me it might be up to thousands. Aggregating two 3-dimension vectors is a lot faster than aggregating two 10000-dimension vectors.</p>
  
  <p>If i put only a single label in my algorithm, i have no problem at all. The 45 seconds i mentioned, are reduced to less than 5. So yes, it is working correctly for one label only.</p>
  
  <p>The 45 seconds i mentioned are for the for-loop only. I didn't count the parsing and all the other things. The for loop is the bottleneck for sure since this is the only thing that i am timing and it takes about 45 seconds, while the whole task takes about 1 minute (including task initialization and many more). I want to try and brake that for-loop into 2 or 3 smaller for loops and process them concurrently. Trying means that it might work and that it might not work. Sometimes crazy stuff like the one i mentioned, could be a necessity. Well that is  what a well respected programmer told me in a previous thread of mine about hadoop.</p>
  
  <p>I didn't provide these many details earlier, since i thought that i only wanted an opinion about hadoop and threads inside the map function. Didn't think that someone would question me so much :P.</p>