 of $h$ consist of two parts, namely, $\mr{Lip}(f_1)=\mr{K}_1$ and $\mr{Lip}(f_2)=\mr{K}_2$. Therefore, the Lipschitz constant of layer $h$ can be expressed as:
\begin{equation} \label{eq:lip_block}
    \mathrm{Lip}(h) = \sqrt[\leftroot{-1}\uproot{1}]{(\mr{K}_1^2 +\mr{K}_2^2)}.
\end{equation}
With spectral normalization of Equation~\eqref{eq:spec_norm}, we know that we can enforce (convolutional) weights $W_i$ to be at most $1$-Lipschitz. Hence, for all $n$ dense layers we apply the spectral normalization on the lower part which locally enforces $\mr{Lip}(f_2)=\mr{K}_2 < 1$. Further, since we enforce each layer $h_i$ to be at most $1$-Lipschitz and we start with $h_1$, where $f_1(x)=x$, we know that $\mr{Lip}(f_1)=1$. Therefore, the Lipschitz constant of an entire layer can be at most $\mr{Lip}(h) < \sqrt[\leftroot{-1}\uproot{1}]{1^2+1^2} = \sqrt[\leftroot{-1}\uproot{1}]{2}$, thus dividing by this limit enforces each layer to be at most 1-Lipschitz.

\subsection{Learnable weighted concatenation}
\begin{wrapfigure}{r}{0.22\textwidth} 
    \vspace{-22pt}
    \begin{center}
    \includegraphics[width=0.21\textwidth]{figures/learnable_concat.pdf}
    \end{center}
    \caption{Range of the possible normalized parameters $\hat{\eta}_1$ and $\hat{\eta}_2$.}
    \vspace{-10pt}
    \label{fig:learnable_params}
\end{wrapfigure}
We have shown that we can enforce an entire dense layer to have $\mr{Lip}(h_i) < 1$ by applying a spectral norm on the (convolutional) weights $W_i$ and then divide the layer $h_i$ by $\sqrt{2}$. Although learning a weighting between the upper and lower part would barely affect a standard dense layer, it matters in this case because the layers are regularized to be 1-Lipschitz. To optimize and learn the importance of the concatenated representations, we introduce learnable parameters $\eta_1$ and $\eta_2$ for, respectively, the upper and lower part of each layer $h_i$. 

Since the upper and lower part of the layer can be at most 1-Lipschitz, multiplication by these factors results in functions that are at most $\eta_1$-Lipschitz and $\eta_2$-Lipschitz. As indicated by Equation~(\ref{eq:lip_block}), the layer is then at most $\sqrt{\eta_1^2 + \eta_2^2}-$Lipschitz. Dividing by this factor results in a bound that is at most $1$-Lipschitz. 

In practice, we initialize $\eta_1$ and $\eta_2$ at value $1$ and during training use a softplus function to avoid them being negative. The range of the normalized parameters is between $\hat{\eta}_{1}, \hat{\eta}_{2} \in [0, 1]$ and can be expressed on the unit circle as shown in Figure~\ref{fig:learnable_params}. In the special case where $\eta_1 = \eta_2$, the normalized parameters are $\hat{\eta}_1=\hat{\eta}_2=\frac{1}{2} \sqrt{2}$. This case corresponds to the situation in Section~\ref{sec:enforcing_lip} where the concatenation is not learned. An additional advantage is that the normalized $\hat{\eta}_1$ and $\hat{\eta}_2$ express the importance of the upper and lower signal. For example, when $\hat{\eta}_1 > \hat{\eta}_2$, the input signal is of more importance than the transformed signal.

\subsection{CLipSwish}\label{sec:clipswish}
When a deep neural network is bounded to be $1$-Lipschitz, in practice each consecutive layer reduces the Jacobian-norm. As a result, the Jacobian-norm of the entire network is becoming much smaller than $1$ and the expressive power is getting lost. This is known as \textit{gradient norm attenuation }\cite{anil2019sorting, li2019preventing}. This problem arises in activation functions in regions where the derivative is small, such as the left tail of the ReLU and the LipSwish.
Non-linearities $\phi$ modeled in i-DenseNets are required to be at most $1$-Lipschitz and, thus, face gradient-norm attenuation issues. For this reason we introduce a new activation function, which mitigates these issues.

Recall that Residual Flows use the LipSwish activation function \citep{chen2019residualflows}:
\begin{equation} \label{eq:lipswish}
    \mr{LipSwish}(x)= x \sigma(\beta x) /1.1, 
\end{equation}
where $\sigma(\beta x) = 1/(1+\exp(-x \beta))$ is the sigmoid, $\beta$ is a learnable constant, initialized at 0.5 and is passed through a softplus to be strictly positive. This activation function is not only $\mr{Lip}(\mr{LipSwish})=1$ but also resolves the derivative saturation problem \citep{chen2019residualflows}. However, the LipSwish function has large ranges on the negative axis where its derivative is close to zero.

Therefore, we propose the Concatenated LipSwish (CLipSwish) which concatenates two LipSwish functions with inputs $x$ and $-x$. This is a concatenated activation function as in \citep{shang2016understanding} but using a LipSwish instead of a ReLU. Intuitively, even if an input lies in the tail of the upper part, it will have a larger derivative in the bottom part and thus suffer less from gradient norm attenuation. Since using CLipSwish increases the channel growth and to stay inline with the channel growth that non-concatenated activation functions use, we use a lower channel growth when using CLipSwish. To utilize the CLipSwish, we need to derive Lipschitz continuity of the activation function $\Phi$ defined below and enforce it to be $1$-Lipschitz. 
We could use the result obtained in Equation~\eqref{eq:lip_block} to obtain a $\sqrt{2}$-bound, however, by using knowledge about the activation function $\Phi$, we can derive a tighter $1.004 < \sqrt{2}$ bound. In general, a tighter bound is preferred since more expressive power will be preserved in the network.
To start with, we define function $\Phi: \mathbb{R} \rightarrow  \mathbb{R}^2$ for a point $x$ as:
\begin{equation}
    \Phi(x) = 
    \begin{bmatrix}
    \phi_1(x) \\ \phi_2(x)
    \end{bmatrix}=
    \begin{bmatrix}
    \mr{LipSwish}(x) \\ \mr{LipSwish}(-x)
    \end{bmatrix}, \quad \mr{CLipSwish(x) = \Phi(x) / \mr{Lip}(\Phi)},
\end{equation}
where the LipSwish is given by Equation~\eqref{eq:lipswish} and the derivative of $\Phi(x)$ exists. To find $\mr{Lip}(\Phi)$ we use that for a differentiable $\ell_2$-Lipschitz bounded function $\Phi$, the following identity holds:
\begin{equation} \label{eq:lip_2norm}
    \mr{Lip}(\Phi) = \sup_x ||J_\Phi(x)||_2,
\end{equation}
where $J_\Phi(x)$ is the Jacobian of $\Phi$ at $x$ and $||\cdot||_2$ represents the induced matrix norm which is equal to the spectral norm of the matrix. Rewriting the spectral norm results in solving: $\mr{det}(J_{\Phi}(x)^T J_{\Phi}(x) - \lambda I_n)=0$, which gives us the final result (see Appendix~\ref{apd:B_1clipswish} for the full derivation):%
\begin{equation} \label{eq:ub_clipswish}
    \sup_x ||J_{\Phi}(x)||_2 = \sup_x \sigma_{\max}(J_{\Phi}(x)) = \sup_x \sqrt{\left(\frac{\partial \phi_1(x)}{\partial x}\right)^2 + \left( \frac{\partial \phi_2(x)}{\partial x}\right)^2},
\end{equation}
where $\sigma_{\max}(\cdot)$ is the largest singular value