not affect the trends too much. Note that $\lambda$ is the number of distinct weight values and thus $\log(\lambda)$ is the corresponding bit-width. It can be seen that $\epsilon$ and $n$ only affect $\log(\lambda_{opt})$ when $d$ is small ($<10^2$). We also mark the input dimension $d$ of various image data set and their corresponding $\log(\lambda_{opt})$. It shows that the optimal bit-width increases very slowly with $d$.](ICLRBitwidth.pdf "fig:"){width="\linewidth"}

Since the bounds are derived for fully connected networks and depend on the construction approach, the interesting proximity between $\log(\lambda_{opt})$ and the empirical results cannot be viewed as a strict theoretical explanation. Regardless, we show that the complexity bound may be a viable approach to understand the optimal bit-width problem, thus potentially accelerating the hyper-parameter optimization of deep neural networks. We defer such a thorough investigation of the optimal bit-width or optimal hybrid bit-width configuration across the network to our future work.

Discussion {#sec:comparison}
==========

In this section, we further discuss the bound of nonlinear quantization with a function-independent structure as the generality of nonlinear quantization. The availability of unquantized function-independent structures in literature also makes it an excellent reference for comparison.

**Comparison with the Upper Bound:** The quality of an upper bound 