How big artificial neural networks can we run now (either with full train-backprop cycle or just evaluating network outputs) if our total energy budget for computation is equivalent to human brain energy budget (12.6 watts)?

Let assume one cycle per second, which seems to roughly match the firing rate of biological neurons.

  • $\begingroup$ I was going to ask this question. I am happy someone else already did :) $\endgroup$ – Eka Sep 8 '16 at 15:47

126 million artificial neurons at 12.6 Watts, with IBM's True North

Back in 2014, IBM's True North chip was pushing 1 million neurons at less than 100mW.

So that's roughly 126 million artificial neurons at 12.6 Watts.

A mouse has 70 million neurons.

IBM believes they can build a human-brain scale True North mainframe at a "mere" 4kW.

Once 3D transistors come to market, I think we'll catch up to animal brain efficiency pretty fast.

  • 1
    $\begingroup$ > So that's roughly 126 million artificial neurons at 12.6 Watts. > A mouse has 70 million neurons. Problem is, we don't know if a single artificial "neuron" is truly a one to one equivalent for a real biological neuron. $\endgroup$ – mindcrime Sep 2 '16 at 22:52
  • $\begingroup$ In addition to mindcrime, keep in mind that for roughly 60 years, neuroscientists/AI researchers have consistently badly underestimated the processing power of the human brain. I saw early estimates by big names that our minds could only store a few 100 MB! We have yet to factor in things like bit depth, proper architecture, possible quantum states in brain cells, etc. IBM may likely build a power-efficient 70-million-neuron neural net that can't even approximate the behavior of a mouse's parasites. AI researchers have historically always badly overestimated their own capabilities. $\endgroup$ – SQLServerSteve Sep 2 '16 at 23:24
  • $\begingroup$ Indeed, forgot about IBM chips. I didn't even take them into consideration when writing the question, as there is known criticism on their efficiency (see e.g. facebook.com/yann.lecun/posts/10152184295832143). Though technically they still implement neural networks, so +1 from me. $\endgroup$ – liori Sep 2 '16 at 23:51
  • $\begingroup$ @mindcrime & SQLServerSteve Well, the question could have been specifically about vanilla feed forward ANNs running on stock hardware. That's not the impression I got from the question though. $\endgroup$ – Doxosophoi Sep 3 '16 at 0:37
  • $\begingroup$ Nor did it seem to be about artificial neural networks that have yet to be imagined. $\endgroup$ – Doxosophoi Sep 3 '16 at 0:48

If you limited yourself to 12.6 watts, you wouldn't get much done. Just lookup the power consumption for a modern GPU, look at the size networks people are training on those, and then scale down. For reference, modern GPU's appear to consume between 52-309 watts under heavy use.

Clearly energy efficiency is one area where the human brain is still far head of ANN's.


Your Answer

