title: UserWarning: One or more of the test scores are non-finite: [nan nan nan nan 0. 0.] when running HalvingRandomSearchCV
body: <p>I am trying to search for optimal parameters by using the faster version of RandomizedSearchCV---&gt;HalvingRandomSearchCV class from sklearn. My dataset is large (~1million records), it has 10 classes and it is imbalanced.</p>
  <p>My code looks like this:</p>
  <pre><code># Define parameters for model 
  params_cat = {'loss_function': 'MultiClass',  
                'eval_metric': 'AUC',
                'random_state': seed,
                'iterations': 20000,
                'early_stopping_rounds' : 5000}
  
  # Define Parameters for Grid 
  params_grid = {'max_depth': [8, 10],
                 'l2_leaf_reg': [3, 5], #5, 10, 15
                 # 'random_strength' : [3], #5
                 # 'border_count': [128], #254
                 #'iterations': [20000, 30000] #40000 }
  
  cat = CatBoostClassifier(**params_cat)
  
  
  mcc_scorer = make_scorer(matthews_corrcoef) cv = StratifiedShuffleSplit(n_splits=2, test_size=0.01, random_state=42) gridsearch = HalvingRandomSearchCV(cat, params_grid, n_jobs=-1, cv=cv, verbose=3,
                                     scoring=mcc_scorer, return_train_score = True)
  
  gridsearch.fit(X_train_val, y_train_val)
  </code></pre>
  <p>When running it, I get the following error:</p>
  <pre><code>/home/ec2-user/anaconda3/lib/python3.6/site-packages/sklearn/metrics/_classification.py:873: RuntimeWarning: invalid value encountered in double_scalars
    mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
  [CV 1/2] END l2_leaf_reg=5, max_depth=9;, score=(train=1.000, test=0.000) total time= 6.0min
  /home/ec2-user/anaconda3/lib/python3.6/site-packages/sklearn/metrics/_classification.py:873: RuntimeWarning: invalid value encountered in double_scalars
    mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
  [CV 2/2] END l2_leaf_reg=5, max_depth=9;, score=(train=1.000, test=0.000) total time= 6.0min
  /home/ec2-user/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_search.py:925: UserWarning: One or more of the test scores are non-finite: [nan nan nan nan  0.  0.]
    category=UserWarning
  </code></pre>
  <p>I don't understand whether the cause of the error is the HalvingRandomSearchCV, the scorer I'm using or the values for the parameters. Does anyone know what could be the problem?</p>
  <p>Thank you!</p>