<p>Generally, the idea is to measure the reconstruction and classify anomalies as those datapoints that cause a significant deviation from the input. Thus, one can other other norms such as <code>mae</code>. However, the results will probably be very similar.</p>
<p>I would suggest different flavors of the auto encoder. First of all, if your are not already using it, the <a href="https://www.jeremyjordan.me/variational-autoencoders/" rel="nofollow noreferrer">variational autoencoder</a> is better than a standard auto encoder in all aspects.</p>
<p>Second, the performance of a variational autoencoder can be significantly improved by using the <a href="http://dm.snu.ac.kr/static/docs/TR/SNUDM-TR-2015-03.pdf" rel="nofollow noreferrer">reconstruction probability</a>. The idea is to output the parameters for probability distributions not only for the latent space but also for the feature space. This means that the decoder would output a mean and a variance to parameterize a normal distribution when used with continuous data. Then the reconstruction probability is basically the negative log likehood of the normal distribution <code>N(x; decoder_mu, decoder_var)</code>. Using the 2-sigma rule, the variance can be interpreted as confidence intervall and thus even small errors can lead to an high error.</p>
<p>Other than that, there are other flavors like <a href="https://www.mdpi.com/1424-8220/20/13/3738" rel="nofollow noreferrer"><code>vae-gan</code></a>, which combines a vae and gan uses a combined anomaly score with the reconstruction error and the discriminator prediction. Also depending on your problem type, you can also go into the route of a <a href="https://arxiv.org/pdf/2103.12998.pdf" rel="nofollow noreferrer"><code>vae-sl</code></a> that adds an additional classifier in the bottleneck. The model is then trained on mixed data which can be fully or sparsed labelled. Then the classifier can be used for anomaly detection.</p>
Picking up the anomalies using autoencoder
<p>Other than mean square error, are there  other quantities that we can use to detect anomalies using autoencoder in keras?</p>