title: PYSPARK - Predicted Target Label MISSING one class - Logistic regression
body: <p>I have just started my Pyspark journey building a Logistic Regression model that predicts users device type (tablet, phone, tv, pad, and desktop).</p>
  
  <p>I have 4 features: </p>
  
  <ol>
  <li>total_minutes  </li>
  <li>vendor_version (iOs, Web, Android, etc)</li>
  <li>day_of_week</li>
  <li>time_of_day</li>
  </ol>
  
  <hr>
  
  <p><strong>Accuracy</strong></p>
  
  <p>The algorithm are working and I have got a good accuracy ~ 83% and when I do final checking to see my predicted target label (device_type), I can see that I am missing one class (pad). </p>
  
  <p><strong>Prototype</strong></p>
  
  <p>At first, I am prototyping with small data, and thought it might need more data. So I grabbed more data (1 million rows) and ran the Log Reg model. </p>
  
  <p>When it comes to check the prediction target class label, I am still missing one class (pad). Does any one have encountered the same issue?</p>
  
  <p><strong>Hypothesis</strong></p>
  
  <p>My hypothesis at first is that, the 'pad' class label is so small (3312 rows in full data, not split into train and test just yet vs 150,000 rows of data and above for other target class labels) that Log Reg didnt even bother to predict for it.</p>
  
  <p><strong>Question</strong> </p>
  
  <p>Could this be the case? Or is there other thing that I should check?</p>
  
  <p>My colleague ran the same algorithm but in Scala but have all the target class labels in his predicted table. So I am not sure what is the issue here.</p>
  
  <p>Let me know if you need more info.</p>
  
  <p>Here is a sample of how the data looks like:</p>
  
  <pre><code>device_id   vendor_version  total_minutes   dow hour    device_type
  1   TV           100                5               22          tv
  2   Web           20                6               10          desktop
  3   iOs            4                3               9           phone
  4   Android    12030                7               15          pad
  5   Chromecast  2300                2               12          tablet
  6   Playstation  587                1               3           tv
  </code></pre>
  
  <p>Best,</p>
  
  <p>Sugi</p>