|
<!doctype html> |
|
<html> |
|
<head> |
|
<meta charset="utf-8" /> |
|
<meta name="viewport" content="width=device-width" /> |
|
<title>My static Space</title> |
|
<link rel="stylesheet" href="style.css" /> |
|
</head> |
|
<body> |
|
|
|
<div align="center"> |
|
<h1>Adaptive Parametric Activation </h1> |
|
|
|
<p><a href="https://kostas1515.github.io/">Konstantinos Panagiotis Alexandridis</a><sup>1</sup>, |
|
<a href="https://jiankangdeng.github.io/">Jiankang Deng</a><sup>1</sup>, |
|
<a href="https://cgi.csc.liv.ac.uk/~anguyen/">Anh Nguyen</a><sup>2</sup>, |
|
<a href="https://shanluo.github.io/">Shan Luo</a><sup>3</sup>,</p> |
|
<p><sup>1</sup> Huawei Noah's Ark Lab, |
|
<sup>2</sup> University of Liverpool, |
|
<sup>3</sup> King's College London</p> |
|
</div> |
|
|
|
<div class="row"> |
|
<div class="col-sm"> |
|
<figure class="block dark:hidden"> |
|
<img alt="APA activation" src="https://huggingface.co/spaces/konsa15/AGLU/resolve/main/assets/unified_activations_combined.jpg" width="600"> |
|
<figcaption> Figure 1: APA unifies most activation functions under the same formula.</figcaption> |
|
</figure> |
|
</div> |
|
</div> |
|
|
|
|
|
<h3>Abstract</h3> |
|
|
|
<p>The activation function plays a crucial role in model optimisation, yet the optimal choice remains unclear. For example, the Sigmoid activation is the de-facto activation in balanced classification tasks, however, in imbalanced classification, it proves inappropriate due to bias towards frequent classes. In this work, we delve deeper in this phenomenon by performing a comprehensive statistical analysis in the classification and intermediate layers of both balanced and imbalanced networks and we empirically show that aligning the activation function with the data distribution, enhances the performance in both balanced and imbalanced tasks. To this end, we propose the Adaptive Parametric Activation (APA) function, a novel and versatile activation function that unifies most common activation functions under a single formula. APA can be applied in both intermediate layers and attention layers, significantly outperforming the state-of-the-art on several imbalanced benchmarks such as ImageNet-LT, iNaturalist2018, Places-LT, CIFAR100-LT and LVIS and balanced benchmarks such as ImageNet1K, COCO and V3DET.</p> |
|
<p>The arxiv link is <a href="https://arxiv.org/abs/2407.08567">here</a> </p> |
|
<h3>Definition</h3> |
|
|
|
<p>The Adaptive Parametric Activation APA is defined as: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>A</mi><mi>P</mi><mi>A</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo><mo>=</mo><mo stretchy="false">(</mo><mi>λ</mi><mi>e</mi><mi>x</mi><mi>p</mi><mo stretchy="false">(</mo><mo>−</mo><mi>κ</mi><mi>z</mi><mo stretchy="false">)</mo><mo>+</mo><mn>1</mn><msup><mo stretchy="false">)</mo><mfrac><mn>1</mn><mrow><mo>−</mo><mi>λ</mi></mrow></mfrac></msup></mrow>APA(z,λ,κ) = (λ exp(−κz) + 1) ^{\frac{1}{−λ}}</math></span>. APA unifies most activation functions under the same formula as shwon in Figure 1.</p> |
|
<p>APA can be used insed the intermediate layers using Adaptive Generalised Linear Unit (AGLU): <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>A</mi><mi>G</mi><mi>L</mi><mi>U</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo><mo>=</mo><mi>z</mi><mi>A</mi><mi>P</mi><mi>A</mi><mo stretchy="false">(</mo><mi>z</mi><mo separator="true">,</mo><mi>λ</mi><mo separator="true">,</mo><mi>κ</mi><mo stretchy="false">)</mo></mrow>AGLU(z,λ,κ) = z APA(z,λ,κ)</math></span>. |
|
The derivatives of AGLU with respect to κ (top), λ (middle) and z (bottom) are shown in Figure 2:</p> |
|
|
|
<div class="row"> |
|
<div class="col-sm"> |
|
<figure class="block dark:hidden"> |
|
<img alt="AGLU derivatives" src="https://huggingface.co/spaces/konsa15/AGLU/resolve/main/assets/derivative_visualisations.jpg" width="600"> |
|
<figcaption> Figure 2: The derivatives of AGLU with respect to κ (top), λ (middle) and z (bottom).</figcaption> |
|
</figure> |
|
</div> |
|
</div> |
|
|
|
|
|
<h3> Simple Code implementation </h3> |
|
|
|
<pre><code class="language-python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Unified</span>(nn.Module): |
|
<span class="hljs-string">"""Unified activation function module."""</span> |
|
|
|
<span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, device=<span class="hljs-literal">None</span>, dtype=<span class="hljs-literal">None</span></span>) -> <span class="hljs-literal">None</span>: |
|
<span class="hljs-string">"""Initialize the Unified activation function."""</span> |
|
factory_kwargs = {<span class="hljs-string">"device"</span>: device, <span class="hljs-string">"dtype"</span>: dtype} |
|
<span class="hljs-built_in">super</span>().__init__() |
|
lambda_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs)) |
|
kappa_param = torch.nn.init.uniform_(torch.empty(<span class="hljs-number">1</span>, **factory_kwargs)) |
|
self.softplus = nn.Softplus(beta=-<span class="hljs-number">1.0</span>) |
|
self.lambda_param = nn.Parameter(lambda_param) |
|
self.kappa_param = nn.Parameter(kappa_param) |
|
|
|
<span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, <span class="hljs-built_in">input</span>: torch.Tensor</span>) -> torch.Tensor: |
|
<span class="hljs-string">"""Compute the forward pass of the Unified activation function."""</span> |
|
l = torch.clamp(self.lambda_param, <span class="hljs-built_in">min</span>=<span class="hljs-number">0.0001</span>) |
|
p = torch.exp((<span class="hljs-number">1</span> / l) * self.softplus((self.kappa_param * <span class="hljs-built_in">input</span>) - torch.log(l))) |
|
<span class="hljs-keyword">return</span> p <span class="hljs-comment"># for AGLU simply return p*input</span> |
|
</code></pre> |
|
<h2 class="relative group flex items-center"> |
|
<a rel="nofollow" href="#bibtex" class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" id="bibtex"> |
|
<span class="header-link"><svg viewBox="0 0 256 256" preserveAspectRatio="xMidYMid meet" height="1em" width="1em" role="img" aria-hidden="true" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4"><path fill="currentColor" d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z"></path></svg></span> |
|
</a> |
|
<span> |
|
BibTeX |
|
</span> |
|
</h2> |
|
<pre><code class="language-bibtex">@inproceedings{alexandridis2024adaptive, |
|
title={Adaptive Parametric Activation}, |
|
author={Alexandridis, Konstantinos Panagiotis and Deng, Jiankang and Nguyen, Anh and Luo, Shan}, |
|
booktitle={European Conference on Computer Vision}, |
|
pages={455--476}, |
|
year={2024}, |
|
organization={Springer} |
|
} |
|
</code></pre> |
|
</div> |
|
</div> |
|
</body> |
|
</html> |
|
|
