<p>There is a simple approach for this. create your clusters with kmeans, then for each clusters, set some good radius with respect to center of that cluster, if some point lie out of that radius, it is an outlier.</p>

<p>Try looking at this: <a href="https://arxiv.org/pdf/1402.6859.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1402.6859.pdf</a></p>

<hr>

<p>There is some outlier detection Technics like: <strong>OneClassSvm</strong> or <strong>AngleBaseOutlierDetection</strong> and so on. Try looking at this: <a href="http://scikit-learn.org/stable/modules/outlier_detection.html" rel="nofollow noreferrer">http://scikit-learn.org/stable/modules/outlier_detection.html</a></p>
How to use the K-Means Algorithm to find anomalies/outliers
<p>I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and I decided to use the KMeans method to solve the project. </p>

<p>The main project consists on analyzing daily files and detecting fluctuating changes in some of the records and reporting them as possible anomalies (if they are considered one based on the model). The files are generated at the end of a day and my program needs to check them on the morning of the next day to see if there is an anomaly. However, I need to check anomalies file vs file, NOT within the file. This means that I have to compare the data of every file and see if it fits to the model I would create following the specific algorithm. What I'm trying to say is that I have some valid data that I will apply the algorithm to in order to train my model. Then I have to apply this same model to other files of the same format but, obviously, different data. I'm not looking for a prediction column but rather detecting anomalies in these other files. If there is an anomaly the program should tell me which row/column has the anomaly and then I have to program it to send an email saying that there is a possible anomaly in the specific file.</p>

<p>Like I said I am new to machine learning. I want to know how I can use the KMeans algorithm to detect outliers/anomalies on a file. </p>

<p>So far I have created the model: </p>

<pre><code>SparkConf conf = new SparkConf().setAppName("practice").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);


SparkSession spark = SparkSession
          .builder()
          .appName("Anomaly Detection")
          .getOrCreate();

String day1txt = "C:\\Users\\User\\Documents\\day1.txt";
String day2txt = "C:\\Users\\User\\Documents\\day2.txt";

Dataset&lt;Row&gt; day1 = spark.read().
        option("header", "true").
        option("delimiter", "\t").
        option("inferSchema", "true").
        csv(day1txt);

day1 = day1.withColumn("Size", day1.col("Size").cast("Integer"));
day1 = day1.withColumn("Records", day1.col("Records").cast("Integer"));

VectorAssembler assembler = new VectorAssembler()
        .setInputCols(new String[]{"Size", "Records"})
        .setOutputCol("features");

Dataset&lt;Row&gt; day1vector = assembler.transform(day1);

KMeans kmeans = new KMeans().setK(5).setSeed(1L);
KMeansModel model = kmeans.fit(day1vector);
</code></pre>

<p>I don't know what to do from this point on to detect outliers. I have several other .txt files that should have "normalized" data, and also I have a couple of files that have "tampered/not-normalized" data. Do I need to train my model with all the test data I have available, and if so, how can I train a model using different datasets? Or can I only train it with one dataset and test it with the others? </p>

<p>EDIT:</p>

<p>This is a sample of the file (day1.txt) I will be using (dummy data of course / top 10)</p>

<pre><code>Name    Size    Records
File1   1000    104370
File2   990     101200
File3   1500    109123
File4   2170    113888
File5   2000    111974
File6   1820    110666
File7   1200    106771
File8   1500    108991
File9   1000    104007
File10  1300    107037
</code></pre>

<p>This is considered normal data, and I will have different files with the same format but different values around the same range. Then I have some files where I purposely added an outlier, like Size: 1000, Records: 50000. </p>

<p>How can I detect that with KMeans? Or if KMeans is not the perfect model, which model should I use and how should I go around it? </p>