<p>The trend these days is to perform end-to-end learning rather than having a model learn some abstract representation and then feed this representation to some other model (e.g. SVM).</p>

<p>The intuition behind this trend is the following: if you optimise a model <code>A</code> on some subtask <code>S1</code> and a model <code>B</code> on another subtask <code>S2</code>, both models will converge to some local optimum solution. The combination of the two local optima is expected to be suboptimal compared to an optimum that would have been obtained by optimising on the <em>full</em> task <code>S = (S1 + S2)</code>. When optimising a model end-to-end, you can adjust all the parts of your model together to better solve the task. However, when you split your model and train separately its different parts, you <em>break</em> the direct <em>signal</em> between the parts and make it harder to improve the output of model <code>A</code> for the specific task of improving the results from model <code>B</code> as you do not have a direct way to optimise the two models alongside.</p>

<p>What you're suggesting was quite popular in the past. For instance, the original <a href="https://arxiv.org/abs/1311.2524" rel="nofollow noreferrer">RCNN</a> paper by Girshick was using a pretrained convolutional neural network to extract features that were then fed to an SVM for classification.</p>

<p>However, this approach was abandoned in the following iteration of R-CNN, <a href="https://arxiv.org/pdf/1504.08083.pdf" rel="nofollow noreferrer">Fast RCNN</a>, the SVM step being replaced by a softmax. In table 8 section 5.4 of <em>Fast R-CNN</em>, the authors compare the same model with SVM vs. softmax and they come to the conclusion that softmax slightly outperforms the SVM version.</p>
Tensorflow Inception model retraining, why not feed bottleneck to an SVM?
<p>The common way to do "transfer-learning", or "retraining" on inception model is to take the bottleneck layer from the model, squeezing the bottleneck tensor as a flat 2048 neuron layer, then add a final layer with the number neurons matching the number of categories to classify (and eventually the softmax).</p>

<p>My question is: instead training this bottleneck layer as a neuron-network, why not feed these highly abstracted 2048 features to an SVM, which probably could achieve a better result ?</p>

<p>Many thanks!</p>