<p>As of now, it is not completely clear in theory what is the best way to initialize the weights of a neural network. As you have mentioned, the loss surface is highly non-convex and different things can happen based on initialization. </p>

<p>Current popular and empirically verified techniques for initialization include the Glorot initialization (<a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf?hc_location=ufi" rel="nofollow noreferrer">http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf?hc_location=ufi</a>) or the He initialization (<a href="https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf" rel="nofollow noreferrer">https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf</a>) which aim to stabilize the training process.</p>

<p>There are some new theoretical guarantees on the well-behavedness of the network with a certain random initialization: <a href="https://arxiv.org/abs/1901.03611" rel="nofollow noreferrer">https://arxiv.org/abs/1901.03611</a></p>

<p>Touching a bit closer on your question, it has been recently showed that for some networks, if trained via SGD, it will converge to a near-closest global optimum of the loss surface: <a href="https://arxiv.org/abs/1902.04674" rel="nofollow noreferrer">https://arxiv.org/abs/1902.04674</a></p>

<p>To conclude, there is no universally accepted answer what is the best initialization for deep neural networks, however there are empirically verified 'good' initializations and there are some theoretical results recently, but this is a very active field of research currently.</p>
Regarding gradient descent-based optimization for neural networks
<p>Is it possible to determine the best starting point for the gradient descent optimization algorithm regarding neural networks?</p>

<p>For example, looking at an example loss surface containing local AND global minima in the link below, it is clear (1) that some starting points are better than other in the sense that the global optimum would be reached faster than other starting points, (2) that some starting points will cause descent into LOCAL, rather than GLOBAL optima and (3) that some starting points will probably never converge at all.</p>

<p><a href="https://www.researchgate.net/profile/Klaus_Raizer/publication/278036660/figure/fig7/AS:294224927969287@1447160097730/Error-surface-in-the-weigth-space-for-two-weights.png" rel="nofollow noreferrer">https://www.researchgate.net/profile/Klaus_Raizer/publication/278036660/figure/fig7/AS:294224927969287@1447160097730/Error-surface-in-the-weigth-space-for-two-weights.png</a></p>

<p>Thanks in advance for any contributions :) </p>