<p>I think you might want to look at the following: <a href="https://arxiv.org/pdf/1610.02757.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1610.02757.pdf</a></p>

<p>By quoting the authors "Notice that XGBoost does not work with the exact hessian but with its diagonal approximation."</p>
How can I optimize boosted trees on Brier score for classification?
<p>I have a binary classification task and use the xgboost package to solve it. Basically, I just use boosted trees to do so. But I am being evaluated on the Brier score, so I thought I would optimize the Brier loss function (defined as the brier score applied on top of logistic classification) which led me to define the gradient and the hessian of the Brier loss like so : </p>

<pre><code>def brier(preds, dtrain):
   labels = dtrain.get_label()
   preds = 1.0 / (1.0 + np.exp(-preds))
   grad = 2*(preds-labels)*preds*(1-preds)
   hess = 2*(2*(labels+1)*preds-labels-3*preds*preds)*preds*(1-preds)
   return grad, hess

def evalerror(preds, dtrain):
   preds = 1.0 / (1.0 + np.exp(-preds))
   labels = dtrain.get_label()
   errors = (labels - preds)**2
   return 'brier-error', float(np.sum(errors)) / len(labels)

param = {'eta':0.01,
'max_depth': 6,  # the maximum depth of each tree
#'objective': 'binary:logistic',
'booster' : 'gbtree',
'eval_metric':['rmse', 'auc']}

bst = xgb.train(param,dtrain, num_boost_round=999,early_stopping_rounds=10,obj=brier, feval=evalerror,evals=[(dtrain,'train'),(dtest,'test')])
</code></pre>

<p>The only problem is that by doing so, I get negative values for my prediction on my test set, which suggests that the output of the xgboost model is not the logistic probability as expected. 
Does anyone know what I am missing here or if there is a better way to optimize the brier score? </p>

<p>Any help would be really appreciated!! </p>

<p>Thanks,</p>