<p>I am the author of the R package <b>optimParallel</b>, which could be helpful in your case. The package provides parallel versions of the gradient-based optimization methods of <code>optim()</code>. The main function of the package is <code>optimParallel()</code>, which has the same usage and output as <code>optim()</code>. Using <code>optimParallel()</code> can significantly reduce optimization times as illustrated in the following figure (<code>p</code> is the number of paramters).
<a href="https://i.stack.imgur.com/PiEZM.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PiEZM.png" alt="enter image description here"></a>
See <a href="https://cran.r-project.org/package=optimParallel" rel="noreferrer">https://cran.r-project.org/package=optimParallel</a> and <a href="http://arxiv.org/abs/1804.11058" rel="noreferrer">http://arxiv.org/abs/1804.11058</a> for more information. </p>
Moving beyond R's optim function
<p>I am trying to use R to estimate a multinomial logit model with a manual specification. I have found a few packages that allow you to estimate MNL models <a href="http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/VGAM/html/multinomial.html" rel="noreferrer">here</a> or <a href="http://cran.r-project.org/web/packages/mlogit/index.html" rel="noreferrer">here</a>.  </p>

<p>I've found some other writings on "rolling" your own MLE function <a href="http://www.mayin.org/ajayshah/KB/R/documents/mle/mle.html" rel="noreferrer">here</a>. However, from my digging around - all of these functions and packages rely on the internal <code>optim</code> function. </p>

<p>In my benchmark tests, <code>optim</code> is the bottleneck. Using a simulated dataset with ~16000 observations and 7 parameters, R takes around 90 seconds on my machine. The equivalent model in <a href="http://transp-or.epfl.ch/page63023.html" rel="noreferrer">Biogeme</a> takes ~10 seconds. A colleague who writes his own code in <a href="http://www.doornik.com/" rel="noreferrer">Ox</a> reports around 4 seconds for this same model.</p>

<p>Does anyone have experience with writing their own MLE function or can point me in the direction of something that is optimized beyond the default <code>optim</code> function (no pun intended)? </p>

<p>If anyone wants the R code to recreate the model, let me know - I'll glady provide it. I haven't provided it since it isn't directly relevant to the problem of optimizing the <code>optim</code> function and to preserve space...</p>

<p><em>EDIT: Thanks to everyone for your thoughts. Based on a myriad of comments below, we were able to get R in the same ballpark as Biogeme for more complicated models, and R was actually faster for several smaller / simpler models that we ran. I think the long term solution to this problem is going to involve writing a separate maximization function that relies on a fortran or C library, but am certainly open to other approaches.</em></p>