title: Applying transformation to new data Spark
body: <p>I am using Spark (core/mlib) with Java, version 2.3.1.</p>
  <p>I am applying three transformations to a Dataset - StringIndexer, OneHotEncoderEstimator, VectorAssember - this is to a transform a categorical variable in my dataset into individual columns of 1 and 0 for each category. On my train data, this transformation works with no issues, everything is as expected, and I am saving this model to file.</p>
  <p>My issue comes when I try to use this model on a new datapoint:</p>
  <pre><code>public static double loadModel(Obj newData) {
      SparkSession spark = Shots.buildSession();
  
      //Function which applies transformations
      Dataset&lt;Row&gt; data = buildDataset(spark, Arrays.asList(newData));
  
      LogisticRegressionModel lrModel = LogisticRegressionModel.load(modelPath);
      //Error is thrown here as the model doesn't seem to understand the input
      Dataset&lt;Row&gt; preds = lrModel.transform(data);
      preds.show();
  }
  </code></pre>
  <p>The issue, I believe, is that the transformation is now being applied to only one row of data which outputs only one category for the categorical feature and a vector with only one element after transformation. This causes an error when the LogisticRegressionModel transform is applied, which is expecting a vector with length greater than one for that feature... I think.</p>
  <p>I know my error is not knowing how to apply the train transform to the new data... but I am unsure where exactly the error is and, as a result, do not know where to find the answer (is the issue with saving the model, do I need to save something else like the pipeline, etc.).</p>
  <p>The actual error being thrown is -</p>
  <pre><code>java.lang.IllegalArgumentException: requirement failed: BLAS.dot(x: Vector, y:Vector) was given Vectors with non-matching sizes: x.size = 7, y.size = 2 - the reason why I have come to the conclusions above is a visual examination of the data.
  </code></pre>
  <p>An example may help explain: I have a categorical feature with 3 values [Yes, No, Maybe]. My train data includes all three values, I end up with a vector feature of length 3 signifying the category.</p>
  <p>I am then using the same pipeline on a single data point to predict a value but the categorical feature can only be Yes, No, or Maybe as there is only one data point. Therefore, when you apply the same transformation as above you ended up with a vector with one element, as opposed to three, causing the model transform to throw an error.</p>