<h2>Transforming a map into a feature vector:</h2>

<p>In your case, you could turn the 7x7x128 map into an array with 6727 dimensions. In case you have an intermediate map, whose flattened array has a high-dimentionality (e.g., 100k-d), you may want to reduce its dimensions (through PCA, for example) because it probably contains a lot of redundancy in its features.</p>

<h2>Combining the output of your layers:</h2>

<p>As for combining the features, there are many ways of doing it. An option that you've mentioned is to create a single vector concatenating the desired layers' outputs and train a classifier on top of that. This is known as <em>early fusion</em> [1], where you combine your features for a single representation before training. </p>

<p>Another possibility is to train a separate classifier for each feature (the output of each intermediate layer, in your case) and then, for a testing image, you combine the outputs/scores of those separate classifiers. This is known as <em>late fusion</em> [1].</p>

<h2>Extras:</h2>

<p>An exploration you could perform is to investigate which layers you should select (either for early- or late-fusion) prior to training your SVM. This [2] is an interesting paper, where the authors explore something similar (analyzing the performance when using the output from the last few layers as features separately). As far as I remember, their investigation is within the context of transfer learning (using a model pre-trained in a similar problem to tackle/solve another task).</p>

<p>[1] <a href="https://www.researchgate.net/profile/Marcel_Worring/publication/221571875_Early_versus_late_fusion_in_semantic_video_analysis/links/0912f508932602eb9a000000.pdf" rel="nofollow noreferrer"><em>"Early versus Late Fusion in Semantic Video Analysis"</em></a></p>

<p>[2] <a href="https://arxiv.org/pdf/1403.6382.pdf" rel="nofollow noreferrer"><em>"CNN Features off-the-shelf: an Astounding Baseline for Recognition"</em></a></p>
extracting CNN features from middle layers
<p>I want to test the performance of each convolutional layer of my Convolutional Neural Network(CNN) architecture using SVM. I am using MatConvNet Matlab toolbox.</p>

<p>Layers are like that:
Conv1 Relu1 Pool1  (3x3, 32 features) -> Conv2 Relu2 Pool2  (3x3, 64 features) -> Conv3 Relu3 Pool3  (3x3, 128 features) ->Conv4 Relu4   (1x1, 256 features) -> Conv5  (1x1, 4 features)-> Softwamxloss</p>

<p>After training, I removed the loss layer</p>

<pre><code>net.layers=net.layers(1 : end - 1);
</code></pre>

<p>I have the network looks like that
<a href="https://i.stack.imgur.com/ij9ik.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ij9ik.jpg" alt="enter image description here"></a></p>

<p>I can extract the features like that:</p>

<pre><code>feats = vl_simplenn(net, im) ;
Feautre_L1(fea,:) = squeeze(feats(end).x);
</code></pre>

<p>similarly, I remove 2 more layers and extract 256 features from Conv4.
But when I moved to Conv3 the output feature is 7x7x128. 
I want to know that how can I used these features<br>
i) Making a single vector 
ii) Averaging the values in depth?</p>