Another comparison of heuristic optimizers

A herd of heuristic algorithms is compared using a portfolio optimization.




Test case






The optimizers


Three methods were added.

pso package

The psoptim function from pso implements a particle swarm algorithm.

nloptr package



cmaes package










Private correspondence with Enrico Schumann improved this section.




Applications matter


Experimentation is useful


What is good enough?







  • a problem
  • an algorithm
  • its control parameters
  • the amount of time it runs


One star



Appendix R

The functions and data to reproduce this analysis are in heuristic_objs2.R (4 MB). The utility that Portfolio Probe achieves is -0.35776787228876. The non-zero optimal weights (to six decimal places) are in optimalWeightsAll.csv.



This entry was posted in optimization, R language and tagged , , , . Bookmark the permalink.

3 Responses to Another comparison of heuristic optimizers

  1. Andrew says:

    Just curious as to why time was used as you stopping criteria? I’ve done some work with a Matlab counterpart of a few of the above mentioned optimization algorithms, (PSO, GA, NLOpt package, and implicit filtering to name a few). Often we use some relative difference stopping tolerance, ie. stop when the previous x function values are within some small tolerance of each other. This method better guarantees convergence to the global optimum. We then proceed to use the number of function evaluations as a way to measure the algorithm’s efficiency. I find computer clocks can be a very unreliable tool, whereas the number of function evaluations is a more concrete value for measuring performance (averages can be taken for stochastic methods while deterministic algorithm’s will always require the same number of function evaluations given identical initial conditions). Also, using time provides no insight as to how certain algorithms might perform if they were operated in parallel. An explanation as to why time was chosen would be great!


    • Pat says:


      Thanks for your comments.

      The two main reasons to use time were:

      * people care about how long a computation takes
      * using time was easy

      Sometimes there is an explicit budget for time. Even if there isn’t, people are likely to have a limited tolerance for how long an optimization takes.

      The number of function evaluations is quite a good measure to use. But I don’t think all of the optimizers have that as a stopping criterion (‘genopt’ does). However, there can be a problem with that if the function evaluations are very fast. In that case, some optimizers may be slower because they use a lot of time relative to the amount of time evaluating the objective function. (But that is also a problem for what I have done.)

      In this comparison, I think using some sort of population convergence would not have been a good criterion. But if you have a real application that you care about, then you probably would want to use that. You could adjust control parameters for the algorithms you are testing so that they tend to converge within your time tolerance. Then pick the one that seems to be best in that setting.

      I’m not convinced we can get much insight into how algorithms are likely to perform in a parallelized setting without doing the parallelization. How you parallelize an algorithm is sure to matter.

      In summary, I think Andrew’s criticism is worth noting. Figuring out which optimizer will work best is hard.

  2. Pingback: A comparison of some heuristic optimization methods | Portfolio Probe | Generate random portfolios. Fund management software by Burns Statistics

Leave a Reply
