<p>This would be problematic, regardless of what framework you did this with. Specifically, we have from the <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">ADAM paper</a> the relevant lines:</p>

<pre><code>g_t = d Cost / d weights
v_t = beta2 * v_{t-1} + (1 - beta2) g_t^2
</code></pre>

<p>Now, if you were to include v_t into Cost, this would be an implicit equation:</p>

<pre><code>g_t = d Cross Entropy / d weights + d (v_t*some_function) / d weights
v_t = beta2 * v_{t-1} + (1 - beta2) g_t^2
</code></pre>

<p>Notice how v_t appears in both equations. We can expand it as such for greater clarity</p>

<pre><code>v_t = beta2 * v_{t-1} + (1 - beta2) [d Cross Entropy / d weights + d (v_t*some_function) / d weights]^2
</code></pre>

<p>You could attempt to solve this exactly, but in doing so you would have to use some form of implicit solver, which will be very computationally costly. One way would be <a href="https://en.wikipedia.org/wiki/Fixed-point_iteration" rel="nofollow noreferrer">fixed point iteration</a>.</p>
Using second momentum as part of new cost function? (Tensorflow and/or keras)
<p>I'm currently trying to take Adam's second order moment term, v_t, and use that as an additional term in my cost function. How can I implement something like this:</p>

<pre><code>Cost = Cross Entropy + v_t*some_function(weights)
</code></pre>

<p>Can this be accomplished within python? Or do I have to write my own C++ code to accomplish this? Also is this easily accomplished in a framework like Keras? 
Here's the code for the cost function that I'm trying to add into keras:</p>

<pre><code>def my_loss(y_pred, y_true, current_weights, v_t):
     normal_loss=K.categorial_cross_entropy(y_pred,y_true)
     additional_term=K.dot(K.square(current_weights - K.some_function(current_weights)), v_t)
     return normal_loss + additional_term
</code></pre>