title: Boost asio io_service mutithreading poor performance
body: <p>I'm playing with cellular automatons and trying to boost my application performance by using multi threading. But I've some interesting results. I'm not really know why this is happening and what I'm missing...</p>
  
  <p>So my aim is to process a big buffer of data as fast as I can. In my example I have a big (20000 x 20000) bool array and transforming it into an image (one bool value to one pixel). This process can be done in parallel; there isn't any dependency between the pixels. I divide the bool array into <code>threadCount</code> number of blocks, start a new thread for each block, let them run, and wait for them to finish.</p>
  
  <p>I assumed that with more threads I`ll get a bit better runtime. (I do not use an unrealistic thread count, just between one and the logical core number.)</p>
  
  <p>So I wrote this:</p>
  
  <pre><code>typedef std::size_t Size;
  typedef std::vector&lt;bool&gt; Data;
  typedef std::vector&lt;Data&gt; History;
  
  class RenderTask
  {
    public:
      typedef void result_type;
  
    public:
      RenderTask(Ppm&amp; ppm,
                 const Ppm::Pixel&amp; fColor)
        : mPpm(ppm),
          mForegroundColor(fColor)
      {
      }
  
      void operator()(const History&amp; history,
                      const Size minIdxX,
                      const Size countX,
                      const Size minIdxY,
                      const Size countY)
      {
        const Size maxIdxX(minIdxX + countX);
        const Size maxIdxY(minIdxY + countY);
        for(Size y(minIdxY); y &lt; maxIdxY; ++y)
        {
          for(Size x(minIdxX); x &lt; maxIdxX; ++x)
          {
            if(history[y][x])
            {
              mPpm.setPixel(x, y, mForegroundColor);
            }
          }
        }
      }
  
    private:
      Ppm&amp; mPpm;
      const Ppm::Pixel mForegroundColor;
  };
  
  void render(const History&amp; history,
              Ppm&amp; ppm,
              const Ppm::Pixel&amp; fColor,
              const Size threadCount)
  {
    boost::asio::io_service io_service;
    boost::thread_group threads;
    for(Size i(0); i &lt; threadCount; ++i)
    {
      threads.create_thread(boost::bind(&amp;boost::asio::io_service::run,
                                        &amp;io_service));
    }
  
    RenderTask task(ppm, fColor);
    io_service.reset();
  
    const Size count(history.size() / threadCount);
    const Size rem(history.size() % threadCount);
    Size minIdxY(0);
    for(Size i(0); i &lt; threadCount; ++i)
    {
      const bool addRemainders(rem &amp;&amp; i == threadCount - 1);
      io_service.post(boost::bind(task,
                                  boost::cref(history),
                                  0,
                                  history.front().size(),
                                  minIdxY,
                                  addRemainders ? count + rem : count));
      minIdxY += count;
    }
  
    threads.join_all();
  }
  
  int main(int argc, char* argv[])
  {
    const Size rule(parseNumber&lt;Size&gt;(argv[1]));
    const Size size(parseNumber&lt;Size&gt;(argv[2]));
    const Size iteration(parseNumber&lt;Size&gt;(argv[3]));
    const Size threadCount(clamp(1,
                                 static_cast&lt;Size&gt;(boost::thread::physical_concurrency())
                                 parseNumber&lt;Size&gt;(argv[4])));
    ...
    History history(iteration, Data(size, false));
    history.front()[size / 2] = true;
    ...
    process(history, rule, threadCount);
    ...
    Ppm ppm(history.front().size(), history.size(), Ppm::Pixel(30, 30, 30));
    std::cout &lt;&lt; "rendering... ";
    t.start();
    render(history, ppm, Ppm::Pixel(200, 200, 200),  threadCount);
    t.stop();
    std::cout &lt;&lt; t.ms() &lt;&lt; " ms" &lt;&lt; std::endl;
  }
  </code></pre>
  
  <p>But when I ran the program with different number of threads I got the following:</p>
  
  <p>I have no idea why the more cores could not result in better performance. With two cores it's a bit better, but interestingly with three cores it almost as the same as with one core... These value are averages:</p>
  
  <pre><code>ThreadingTest.exe 110 20000 20000 1 test.ppm
  rendering... 554.95 ms
  
  ThreadingTest.exe 110 20000 20000 2 test.ppm
  rendering... 289.75 ms
  
  ThreadingTest.exe 110 20000 20000 3 test.ppm
  rendering... 555.37 ms
  
  ThreadingTest.exe 110 20000 20000 4 test.ppm
  rendering... 554.23 ms
  
  ThreadingTest.exe 110 20000 20000 5 test.ppm
  rendering... 564.23 ms
  
  ThreadingTest.exe 110 20000 20000 6 test.ppm
  rendering... 551.82 ms
  
  ThreadingTest.exe 110 20000 20000 7 test.ppm
  rendering... 555.22 ms
  
  ThreadingTest.exe 110 20000 20000 8 test.ppm
  rendering... 510.12 ms
  </code></pre>
  
  <p>What can causes this? Am I using the io_service in a wrong fashion? There is no I/O operation involved, just pure memory.</p>
  
  <p>I have eight cores in my machine with 16 GB of RAM.</p>
  
  <p>For additional details, here is the outline of the Ppm class:</p>
  
  <pre><code>class Ppm
  {
    public:
      struct Pixel
      {
        typedef unsigned char ChannelType;
        ChannelType  mRed, mGreen, mBlue;
        ...
      };
  
      typedef std::vector&lt;Pixel&gt; ImageData;
  
      Ppm( const SizeType width
           , const SizeType height
           , const Pixel&amp; color = Pixel() )
        : mWidth( width )
        , mHeight( height )
        , mImageData( mWidth * mHeight, color )
      { }
  
      void setPixel( SizeType x, SizeType y, const Pixel&amp; p )
      {
        mImageData[x + y * mWidth] = p;
      }
      ...
    private:
      const SizeType mWidth;
      const SizeType mHeight;
      ImageData mImageData;
  };
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>After your valuable comments I changed the approach a lot and wrote this:
  Now I'm using pure c++'11 stuff and there is no boost involved anymore..</p>
  
  <pre><code>class ThreadPool
  {
    public:
      ThreadPool(const Size threadCount);
      ~ThreadPool();
  
    public:
      template&lt;class T&gt;
      void addTask(T task);
      void wait();
  
    private:
      bool mStopped;
      Size mRunningCount;
      std::vector&lt;std::thread&gt; mWorkers;
      std::deque&lt;std::function&lt;void()&gt;&gt; mTasks;
      std::mutex mMutex;
      std::condition_variable mCondition;
      std::condition_variable mFinishCondition;
  };
  
  ThreadPool::ThreadPool(const Size threadCount)
    : mStopped(false),
      mRunningCount(0)
  {
    for (Size i(0); i &lt; threadCount; ++i)
    {
      mWorkers.push_back(std::thread([this]()
        {
          std::function&lt;void()&gt; task;
          while(true)
          {
            {
              std::unique_lock&lt;std::mutex&gt; lock(this-&gt;mMutex);
  
              this-&gt;mCondition.wait(lock, [this] { return this-&gt;mStopped ||
                                                          !this-&gt;mTasks.empty();
                                                 });
              if(this-&gt;mStopped)
              {
                return;
              }
  
              ++this-&gt;mRunningCount;
              task = this-&gt;mTasks.front();
              this-&gt;mTasks.pop_front();
            }
  
            task();
  
            {
              std::unique_lock&lt;std::mutex&gt; lock(this-&gt;mMutex);
              --this-&gt;mRunningCount;
            }
  
            this-&gt;mFinishCondition.notify_all();
        }
      }));
    }
  }
  
  ThreadPool::~ThreadPool()
  {
    {
      std::unique_lock&lt;std::mutex&gt; lock(mMutex);
      mStopped = true;
      mCondition.notify_all();
    }
  
    for(auto&amp; worker : mWorkers)
    {
      worker.join();
    }
  }
  
  template&lt;class T&gt;
  void ThreadPool::addTask(T task)
  {
    {
      std::unique_lock&lt;std::mutex&gt; lock(mMutex);
      mTasks.push_back(std::function&lt;void()&gt;(task));
    }
    mCondition.notify_one();
  }
  
  void ThreadPool::wait()
  {
    std::unique_lock&lt;std::mutex&gt; lock(mMutex);
    mFinishCondition.wait(lock, [this]()
      {
        return mTasks.empty() &amp;&amp; mRunningCount == 0;
      });
  }
  </code></pre>
  
  <p>Now the performance is OK; using more threads the runtime gets faster...
  But something is not good with the wait method. I'm using it in this way:</p>
  
  <pre><code>ThreadPool pool(threadCount);
  for(Size i(1); i &lt; iteration; ++i)
  {
    Size count(history.front().size() / threadCount);
    Size rem(history.front().size() % threadCount);
    Size minIdx(0);
    for(Size n(0); n &lt; threadCount; ++n)
    {
      pool.addTask(std::bind(ECATask(rule),
                             std::cref(history[i-1]),
                             std::ref(history[i]),
                             minIdx,
                             (rem &amp;&amp; n == threadCount - 1) ?
                               count + rem :
                               count));
      minIdx += count;
    }
    pool.wait();
  }
  </code></pre>
  
  <p>The problem with this in not clear yet, but it seems the <code>pool.wait()</code> sometimes does not wait for all the current tasks to be finished and code starts a new iteration... Can you please do a code review for me? :)</p>