Why are Eight Bits Enough for Deep Neural Networks?


Picture by Retronator





How does it work?



Why does it work?




What does this mean?




25 responses

  1. Some sort of “8 bit proof” could be taken just from notebook. Open any video recording tool and look at yourself. I hope you recognize yourself pretty fast. 🙂 The explanation of the proof starts from the fact your notebook camera very likely uses no more than 8 bits per BGR channel. And it appears enough for the task. Another consideration: 1% value is not generally significant, so 1 bit value of 8 bit range counts just as 0.4%, and so fits quite well. Of course, math guys will put a lot of critics on these “proofs”, but I think as coarse assessment it does work.
    About memory requirements, you’re right. After evaluation of some my test, I discovered machine needs about 3 terabytes of RAM just to mimic neocortex of HS with 8 bit precision. Having 64 bit computing cuts a range by 8. Sounds bad.

  2. Binary ANNs use just one bit for weights. I have done some testing on binary ANNs, using the output of many of them to provide a consensus decision. This has usefulness in minimizing te memory of te ANN, but allowing them to run in parallel. Kanerva’s Sparse Distributed memory uses 8-nit weights, althought I suspect it will work with fewer bits.

    I suspect that as te number of connections increases, the smaller the individual bit-size needs to be. At the limit, it may be just 1. Perhaps a better approach is to consider what is the aggregate number of bits needed across a given network to reach some arbitray accuracy. Are 10 connections at 32 bits about as accurate as 40 at 8-bits or 320 at 1 bit each?

  3. Pingback: nl comments on “Intel Buys Altera for $16.7B” | Exploding Ads

  4. As Xueseng Qian mentioned in “Engineering Control Theory”, Deviation of the system may be under control even that of components are large, since deviation of composite components may be counteracted.

  5. Pingback: Five Deep Links « Pete Warden's blog

  6. Pingback: LessThunk.com « Why are Eight Bits Enough for Deep Neural Networks? | Pete Warden’s blog

  7. You can reduce the numerical precision of a neuron model, from floating point to discrete value/ integer networks, simplify the sigmoid function, use power-of-2 weights, etc. and still get good results on many tasks. Techniques like this were used to successfully run neural networks on non-desktop CPUs that didn’t have a floating point unit e.g. mobile robotics platforms, FPGAs etc. There is a list of citations on this topic dating back to 1988 at https://goo.gl/6nltlB page 68.

  8. Pingback: One Weird Trick for Faster Android Multithreading « Pete Warden's blog


    Um.. no. Integer computation, when it can be limited to 16-bits or less are typically 2+ times faster on neon or SSE4.1 or later. GPUs on the other hand are heavily geared towards float and may in fact be faster with float than integer arithmetic.

  10. On a modern x86 chip with AVX2 you have instructions to operations to deal with 32 8bit numbers. There is even an 8bit macc instruction, which conveniently gives a 16bit result which would be a great fit for this kind of algorithm.

    • From what I see on Wikipedia, Cannonlake is expected to have byte and word instructions and “some Skylake Xeons” have them. But they are listed as AVX-512 instructions, not AVX2. What are the AVX2 instructions you are referring to?

  11. Pingback: How to Quantize Neural Networks with TensorFlow « Pete Warden's blog

  12. Pingback: Deep Learning & Pieces of Eight - OrionX.net A New Model for Strategy, Marketing, PR

  13. Pingback: Bookmarks for October 17th from 15:10 to 20:12 : Extenuating Circumstances

  14. You say “their almost-magical effectiveness at a lot of AI tasks” (‘their’ being Deep NNs).
    I find that perplexing. What tasks have these hyped models been (relatively) successful at, besides basic pattern recognition tasks (sound and image recognition) the kind of which even cats and mice to much better – where’ the huge AI success, it beats me!

  15. Hi Pete, I found your post because of a project I’m working on and I’d like to thank you. As a hardware engineer I’ve had the same feeling about redundant memory — more so because chip manufacturers would never recommend using less RAM. I’m glad to see it confirmed. From now on my network will be 8-bit fixed point. (RGBA!)

  16. Pingback: What Machine Learning needs from Hardware « Pete Warden's blog

  17. Pingback: Quantization Screencast « Pete Warden's blog

Leave a Reply


WordPress.com Logo


Google photo


Twitter picture


Facebook photo


Connecting to %s

%d bloggers like this: