Our conclusions on ReLU networks can be extended to any other networks that use piecewise linear activation functions with finite breakpoints such as leaky ReLU and ReLU-6 immediately, as one can replace a ReLU network by an equivalent one using these activation functions while only increasing the number of units and weights by constant factors [@yarotsky2017error].

We denote the finite number of distinct weight values as $\lambda$ ($ \lambda \in \mathbb{Z}^+$ and $\lambda \geq2$), for both linear quantization and nonlinear quantization. For linear quantization, without loss of generality, we assume the finite number of distinct weight values are given as $ \{-1, \frac{1}{\lambda},\frac{2}{\lambda},\dots,\frac{\lambda-1}{\lambda}\}$, where $\{\frac{1}{\lambda},\frac{2}{\lambda},\dots,\frac{\lambda-1}{\lambda}\}$ are uniformly spaced (hence called “linear”) in $(0,1)$ and $-1$ is used to obtain the negative weight values. For nonlinear quantization, we assume the finite number of distinct weight values are not constrained to any specific values, i.e., they can take any values as needed. To store each weight, we only need $\log (\lambda)$ [^2] bits to encode the index, i.e. the bit-width is $\log (\lambda)$. The overhead to store sparse structures can be ignored because it varies depending on the implementation and can be easily reduced to the same order as the weight storage 