title: How does caffe compute gradient when there are multiple branches?
body: <p>I'm now reading <code>Caffe</code> source code, and the question occurred to me.</p>
  
  <p>Take <code>caffe/relu_layer.cpp</code> for example. When computing gradient, from</p>
  
  <pre><code>void ReLULayer&lt;Dtype&gt;::Backward_cpu(const vector&lt;Blob&lt;Dtype&gt;*&gt;&amp; top,
      const vector&lt;bool&gt;&amp; propagate_down,
      const vector&lt;Blob&lt;Dtype&gt;*&gt;&amp; bottom) {
    if (propagate_down[0]) {
      const Dtype* bottom_data = bottom[0]-&gt;cpu_data();
      const Dtype* top_diff = top[0]-&gt;cpu_diff();
      Dtype* bottom_diff = bottom[0]-&gt;mutable_cpu_diff();
      const int count = bottom[0]-&gt;count();
      Dtype negative_slope = this-&gt;layer_param_.relu_param().negative_slope();
      for (int i = 0; i &lt; count; ++i) {
        bottom_diff[i] = top_diff[i] * ((bottom_data[i] &gt; 0)
            + negative_slope * (bottom_data[i] &lt;= 0));
      }
    }
  }
  </code></pre>
  
  <p>we can see a value is finally assigned to <code>bottom_diff</code>, indicating that value is the gradient of the corresponding bottom blob.</p>
  
  <p>However, when multiple layers take one blob as inputs, e.g., stacking multiple <code>ReLU</code> layers on one blob, how does <code>Caffe</code> handle the gradient computation? The first <code>ReLU</code> layer modifies <code>bottom_diff</code>, and it seems that the second <code>ReLU</code> layer just overrides it, instead of adding two gradients.</p>
  
  <p>I didn't see anywhere performing gradient summation, and I am confuses. Please inform me if I have missed something important, and thanks a lot.</p>