<h1>What is inductive bias?</h1>

<p>Pretty much every design choice in machine learning signifies some sort of inductive bias. <a href="https://arxiv.org/pdf/1806.01261.pdf" rel="noreferrer">"Relational inductive biases, deep learning, and graph networks" (Battaglia et. al, 2018)</a> is an amazing  read, which I will be referring to throughout this answer.</p>

<blockquote>
  <p>An <strong>inductive bias</strong> allows a learning algorithm to <strong>prioritize one solution (or interpretation) over another</strong>, independent of the observed data. [...] Inductive biases can express assumptions about either the data-generating process or the space of solutions.</p>
</blockquote>

<h2>Examples in deep learning</h2>

<p>Concretely speaking, the very <strong>composition of layers</strong>  in deep learning provides a  type of relational inductive bias: <em>hierarchical processing</em>. The <strong>type of layer</strong> imposes further relational inductive biases:</p>

<p><a href="https://i.stack.imgur.com/QbD58.png" rel="noreferrer"><img src="https://i.stack.imgur.com/QbD58.png" alt="Various relational inductive biases in standard deep learning components (Battaglia et. al, 2018)"></a></p>

<p>More generally, non-relational inductive biases used in deep learning include:</p>

<ul>
<li>activation non-linearities,</li>
<li>weight decay,</li>
<li>dropout,</li>
<li>batch and layer normalization,</li>
<li>data augmentation,</li>
<li>training curricula,</li>
<li>optimization algorithms,</li>
<li><em>anything that imposes constraints on the learning trajectory</em>.</li>
</ul>

<h2>Examples outside of deep learning</h2>

<p>In a Bayesian model, inductive biases are typically expressed through the choice and parameterization of the prior distribution. Adding a Tikhonov regularization penalty to your loss function implies assuming that simpler hypotheses are more likely.</p>

<h1>Conclusion</h1>

<p>The stronger the inductive bias, the better the sample efficiency--this can be understood in terms of the <strong>bias-variance tradeoff</strong>. Many modern deep learning methods follow an “end-to-end” design philosophy which emphasizes minimal <em>a priori</em> representational and computational assumptions, which explains why they tend to be so <strong>data-intensive</strong>. On the other hand, there is a lot of research into baking stronger relational inductive biases into deep learning architectures, e.g. with graph networks.</p>

<h2>An aside about the word "inductive"</h2>

<p>In philosophy, inductive reasoning refers to <strong>generalization</strong> from specific observations to a conclusion. This is a counterpoint to deductive reasoning, which refers to <strong>specialization</strong> from general ideas to a conclusion.</p>
What is inductive bias in machine learning?
<p>What is inductive bias in machine learning? Why is it necessary?</p>