title: Retrieving prediction result in negative
body: <p>I have the following sample data using regression algorithm to predict time.
  I trained the model using lightgbm and xgboost too, however, I am retrieving very bad prediction result and prediction results are negative. I am not sure what I am doing wrong here. Thanks</p>
  <pre><code>size channels time
  3    3      4.980278
  3    16     4.972054
  3    64     4.899884
  3    256    5.499221
  3    512    5.599495
  3    1024   5.936933
  16   3      5.221653
  16   16     5.994821
  16   64     6.648254
  16   256    7.176828
  16   512    8.1707
  16   1024   8.651496
  64   3      7.801533
  64   16     7.398248
  64   64     8.395648
  64   256    17.49494
  64   512    26.43354
  64   1024   49.55192
  256  3      12.36093
  256  16     20.50781
  256  64     46.49553
  256  256    170.5452
  256  512    333.8809
  256  1024   675.9459
  512  3      22.44313
  512  16     53.82643
  512  64     164.3493
  512  256    659.4345
  512  512    1306.881
  512  1024   3122.403 
  </code></pre>
  <p>lightbgm code</p>
  <pre><code>x_train, x_val, y_train, y_val = train_test_split(X,Y, train_size=0.8)
  print(f&quot;Number of training examples {len(x_train)}&quot;)
  print(f&quot;Number fo testing examples {len(x_val)}&quot;)
  regressor = lightgbm.LGBMRegressor()
  regressor.fit(x_train,y_train)
  train_pred = regressor.predict(x_train)
  train_rmse = mean_squared_error(train_pred, y_train) ** 0.5
  print(f&quot;Train RMSE is {train_rmse}&quot;)
  val_pred = regressor.predict(x_val)
  val_rmse = mean_squared_error(val_pred, y_val)**0.5 
  print(f&quot;Test RMSE is {val_rmse}&quot;)
  R_squared = r2_score(val_pred,y_val)
  print('R2',R_squared)
  </code></pre>
  <p>Result</p>
  <pre><code>Train RMSE is 5385.50, Test RMSE is 1245.1,R2 -2.9991290197894976e+31
  </code></pre>
  <p>XGBoost code optimised using optuna</p>
  <pre><code>def optimize(trial,x,y,regressor):
    max_depth = trial.suggest_int(&quot;max_depth&quot;,3,10)
    n_estimators = trial.suggest_int(&quot;n_estimators&quot;,5000,10000)
    max_leaves= trial.suggest_int(&quot;max_leaves&quot;,1,10)
    learning_rate = trial.suggest_loguniform('learning_rate', 0.001, 0.1)
    colsample_bytree = trial.suggest_uniform('colsample_bytree', 0.0, 1.0) 
    min_child_weight = trial.suggest_uniform('min_child_weight',1,3)
    subsample = trial.suggest_uniform('subsample', 0.5, 1)
  
    model = xgb.XGBRegressor(
      objective ='reg:squarederror',
      n_estimators=n_estimators,
      max_depth=max_depth,
      learning_rate=learning_rate,
      colsample_bytree=colsample_bytree,
      min_child_weight=min_child_weight,
      max_leaves=max_leaves,
      subsample = subsample
  )
    kf=model_selection.KFold(n_splits=5)
    error=[]
    for idx in kf.split(X=x , y=y):
      train_idx , test_idx= idx[0],idx[1]
      xtrain=x[train_idx]
      ytrain=y[train_idx]
      xtest=x[test_idx]
      ytest=y[test_idx]   
      model.fit(xtrain,ytrain)
      y_pred = model.predict(xtest)
      fold_err = metrics.mean_squared_error(ytest,y_pred)
      error.append(np.sqrt(fold_err))
    return np.mean(error)
    best_params={{'max_depth': 9, 'n_estimators': 9242, 'max_leaves': 7, 'learning_rate': 0.0015809052065858954, 'colsample_bytree': 0.4908644884609704, 'min_child_weight': 2.3502876962874435, 'subsample': 0.5927926099148189}
  def optimize_xgb(X,y):
    list_of_y = [&quot;Target 1&quot;]
    for i,m in zip(range(y.shape[1]),list_of_y):
      print(&quot;{} optimized Parameters on MSE Error&quot;.format(m))
      optimization_function = partial(optimize , x=X,y=y[:,i],regressor=&quot;random_forest&quot;)
      study = optuna.create_study(direction=&quot;minimize&quot;)
      study.optimize(optimization_function,n_trials=50)
  optimize_xgb(X_train, y_train)
  def modeling(X,
               y,
               optimize = &quot;no&quot;,
               max_depth=50,
               n_estimators=3000,
               max_leaves=30,
               learning_rate=0.01,
               colsample_bytree=1.0,
               gamma=0.0001,
               min_child_weight=2,
               reg_lambda=0.0001):
    if optimize == &quot;no&quot;:
      model = xgb.XGBRegressor(objective='reg:squarederror')
    else:
      model = xgb.XGBRegressor(objective='reg:squarederror',
                            **best_params)
    if y.shape[1] ==1:
      model_xgb = model.fit(X, y)
  
    cv = RepeatedKFold(n_splits=5, n_repeats=3, random_state=1)
    scores = []
    for i in range(y.shape[1]):
      scores.append(np.abs(cross_val_score(model, X, y[:,i], scoring='neg_mean_squared_error', cv=cv, n_jobs=-1)))
      print('Mean MSE of the {} target : {}  ({})'.format(i,scores[i].mean(), scores[i].std()) )
    return model_xgb
  model_xgb = modeling(X_train,y_train, optimize=&quot;yes&quot;)
  model_xgb.fit(X_train, y_train)
  y_pred = model_xgb.predict(X_test)
  MSE = mse(y_pred,y_test)
  RMSE = np.sqrt(MSE)
  print(&quot;TEST MSE&quot;,MSE)
  R_squared = r2_score(y_pred,y_test)
  print(&quot;RMSE: &quot;, np.round(RMSE, 2))
  print(&quot;R-Squared: &quot;, np.round(R_squared, 2))
  </code></pre>
  <p>The result for xgboost</p>
  <pre><code>TEST MSE 2653915.139388934,RMSE:  1629.08,R-Squared:  -1.69
  </code></pre>