<p>If you plan to use neural networks with TensorFlow or Pytorch, it will be easy. As long as you can express the function <code>U</code> within the framework and the utility function is reasonably close to being continuous, you can back-propagate the utility to the network. You just ask the optimizer to maximize the utility and that's it.</p>

<p>If the utility function is discrete, it gets tricky, but there are several tricks you might try. One of them is the <a href="http://www.cs.toronto.edu/~tingwuwang/REINFORCE.pdf" rel="nofollow noreferrer">REINFORCE algorithm</a> (Monte-Carlo policy gradient). Another trick that is getting quite popular is <a href="https://casmls.github.io/general/2017/02/01/GumbelSoftmax.html" rel="nofollow noreferrer">Gubmle softmax</a> that allows sampling of discrete action and propagating the error to the network.</p>

<p>If you plan to use different classifiers (like decision forests or whatever), you might try something based on imitation learning like the <a href="https://arxiv.org/abs/0907.0786" rel="nofollow noreferrer">SEARN algorithim</a>.</p>
Classification with custom utility function
<p>I have a problem which involves optimization of actions over time:</p>

<ul>
<li>Lets assume I have a set of input variables <code>X</code> where each <code>X_i_t</code> has a
value at each point in time <code>t = 0 ... T</code>.</li>
<li>For each point in time, I would like to choose an action <code>a_t</code> of a set of
actions <code>A</code>,</li>
<li>such that a utility function <code>U(a0, ..., a_T)</code> is maximized.</li>
</ul>

<p>Note, the utility function does not have a closed-form solution and its value depends on the entire sequence of actions <code>a_0 ... a_T</code>.</p>

<p><strong>How would I implement something like this?</strong> I am perfectly happy with a keyword I can use to look up relevant literature. I do not need a full solution. - Though if somebody can point me to a python sklearn function which does this, I would definitly not say no...</p>

<p>My first intuition was "logistic regression" but there is no way to assign "correct labels" to an action <code>a_t</code> at time <code>t</code>, since the utility depends on the actions taken earlier and later in the time series.</p>