<p>This could happen if you have a high learning rate. Maybe, in the gradient-descent even if you reach close to a good local optimum, due to the high learning rate you get shot out of it soon. Try reducing the learning rate and see if this behavior goes away. Also, use Adam if you're not using already.</p>

<p><a href="https://i.stack.imgur.com/BIoUW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BIoUW.png" alt="Image from KDnuggets"></a>
Validation loss is just following the overall trend of the training loss, that's nothing interesting.</p>

<p>A similar strategy is used in cyclic learning, where learning is suddenly increased to get models with multiple local minima. (In your case, the model is just not going to converge)</p>

<p><a href="https://arxiv.org/pdf/1506.01186.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1506.01186.pdf</a></p>
Loss graph in neural networks
<p>I've made a neural network and I plotted the loss of training and validation set. And for validation I get like a step function type of loss and for training, I get these weird spikes. Now I know my model is learning nothing because my loss is so high but I still wonder what does these spikes actually mean. I mean why am I getting these spikes. I've been looking to literature but haven't been able to find an explanation. Could it be that during gradient descent my model moves close to some local optimum but then takes a giant step and moves away but then these spikes seem to happen periodically. And I have no idea what's causing a step function for validation. 
I've attached a image as well.</p>

<p><a href="https://i.stack.imgur.com/Tj34C.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Tj34C.png" alt=""></a></p>