<p>There are mainly two kind of metrics/loss function in generative models/image restoration models</p>
<ol>
<li>reference based
<ul>
<li>l2 loss (~rmse loss)</li>
<li>l1 loss</li>
<li>PSNR : peak Signal-to-Noise ratio</li>
</ul>
</li>
<li>non-reference based
<ul>
<li>SSIM : structural similarity index,</li>
<li>MS-SSIM : Multi-scale SSIM
<br />
*Also there are metrics that are combination of both of classes (Ex. Mixed SM-SSIM)<br />
For more detailed understanding you can read <a href="https://arxiv.org/pdf/1511.08861.pdf" rel="nofollow noreferrer">this paper</a>.</li>
</ul>
</li>
</ol>
<p>I think for the error you got can be resolve by,<br />
<code>mse(vec1, vec2)</code><br />
*It could also depends on library you are using.</p>
How to evaluate autoencoder performance by mse?
<p>I want to train autoencoder on mnist dataset to generate images similar to input. That's code I found:</p>
<pre><code>% Load the training data. 
XTrain = digitTrainCellArrayData;


%% 
% Train an autoencoder with a hidden layer containing 50 neurons. 
hiddenSize = 50;
autoenc = trainAutoencoder(XTrain,hiddenSize,...
        'MaxEpochs', 200, ...
        'L2WeightRegularization', 10^(-3),...
        'SparsityRegularization',1,...
        'SparsityProportion',0.45);  

% Load the test data. 
XTest = digitTestCellArrayData; 

% Reconstruct the test image data using the trained autoencoder. 
xReconstructed = predict(autoenc, XTest);  
</code></pre>
<p>And now I want to somehow evaluate performance of my autoencoder and I'm not sure how to do it. I found <a href="https://nl.mathworks.com/help/deeplearning/ref/trainautoencoder.html" rel="nofollow noreferrer">here</a> that MSE can be used between real and reconstructed data set. However when I tried this:</p>
<pre><code>mse(XTest - xReconstructed)
</code></pre>
<p>I obtain error</p>
<pre><code>Operator '-' is not supported for operands of type 'cell'.
</code></pre>
<p>How can I properly perform evaluation of this autoencoder using mse?</p>