{"action":"created","issue":{"url":"https://api.github.com/repos/scipy/scipy/issues/8373","repository_url":"https://api.github.com/repos/scipy/scipy","labels_url":"https://api.github.com/repos/scipy/scipy/issues/8373/labels{/name}","comments_url":"https://api.github.com/repos/scipy/scipy/issues/8373/comments","events_url":"https://api.github.com/repos/scipy/scipy/issues/8373/events","html_url":"https://github.com/scipy/scipy/issues/8373","id":295021200,"number":8373,"title":"scipy.optimize has broken my trust.","user":{"login":"brandondube","id":13191497,"avatar_url":"https://avatars2.githubusercontent.com/u/13191497?v=4","gravatar_id":"","url":"https://api.github.com/users/brandondube","html_url":"https://github.com/brandondube","followers_url":"https://api.github.com/users/brandondube/followers","following_url":"https://api.github.com/users/brandondube/following{/other_user}","gists_url":"https://api.github.com/users/brandondube/gists{/gist_id}","starred_url":"https://api.github.com/users/brandondube/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brandondube/subscriptions","organizations_url":"https://api.github.com/users/brandondube/orgs","repos_url":"https://api.github.com/users/brandondube/repos","events_url":"https://api.github.com/users/brandondube/events{/privacy}","received_events_url":"https://api.github.com/users/brandondube/received_events","type":"User","site_admin":false},"labels":[{"id":37334597,"url":"https://api.github.com/repos/scipy/scipy/labels/defect","name":"defect","color":"e11d21","default":false},{"id":37341011,"url":"https://api.github.com/repos/scipy/scipy/labels/scipy.optimize","name":"scipy.optimize","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":34,"created_at":"2018-02-07T06:26:39Z","updated_at":"2018-02-08T20:53:22Z","closed_at":null,"author_association":"NONE","body":"This submodule has broken my trust.  \r\n\r\nI am genuinely in awe of how it was included in scipy 1.0.  I am tagging two heavy hitters to attempt to bring discussion to this issue since, in general, scipy.optimize issues seem to go without discussion or fixes.  @pv @teoliphant\r\n\r\nAs I understand, an effort was made in the last few version of scipy leading up to and including 1.0 to unify the interface of the different optimization routines and eventually deprecate fmin_xxxx.\r\n\r\nA fairly large issue with this is that many of the inputs and outputs in the scipy documentation for these functions are simply wrong.\r\n\r\nTake, for example the [BFGS method](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html#optimize-minimize-bfgs).\r\n\r\nthe documentation for disp is incorrect; convergence messages are not printed, only a blurb at the end.  return_all appears in the call signature at the top and is undocumented.  I looked to [fmin_bfgs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_bfgs.html) hoping its more specific documentation would be superior.  The documentation for that is even more misleading.  I hoped that the specified type for the constituents of allvecs being OptimizationResult would mean they hold cost function values and input vector values.  They are simply input vector values.  The documentation here is also misleading.\r\n\r\nI also understand that many of the algorithms are wrapped code in another language, e.g. L-BFGS-B is written in FORTRAN77, and this may make their modification difficult.  Earlier today I opened an issue that L-BFGS-B does not respect the maxiter input, #8372.  There are other issues, e.g. lack of respect for user's input dtype because of hard coded doubles in the fortran code, for example.\r\n\r\nHowever, the docs being misleading is a great travesty for the users of this module, which I believe is among the more important/widely used of scipy.  I would work to improve them myself, but I do not feel that my knowledge of the inner workings of the module is great enough to do so without myself making mistakes.\r\n\r\nReturning to L-BFGS-B, the wrapped fortran code appears to contain a variety of errors or behaviors that have a varied level of harmfulness.  For example, this is an example output:\r\n\r\n```\r\nRUNNING THE L-BFGS-B CODE\r\n\r\n           * * *\r\n\r\nMachine precision = 2.220D-16\r\n N =            4     M =           10\r\n\r\nAt X0         0 variables are exactly at the bounds\r\n\r\nAt iterate    0    f=  2.75370D+01    |proj g|=  1.06893D+02\r\n\r\nAt iterate    1    f=  2.27149D+01    |proj g|=  1.76623D+02\r\n\r\nAt iterate    2    f=  1.20196D+01    |proj g|=  1.20733D+02\r\n\r\nAt iterate    3    f=  8.05081D+00    |proj g|=  8.74848D+01\r\n\r\nAt iterate    4    f=  5.00344D+00    |proj g|=  8.42253D+01\r\n\r\nAt iterate    5    f=  3.70940D+00    |proj g|=  1.01991D+02\r\n\r\nAt iterate    6    f=  1.57818D+00    |proj g|=  7.10948D+01\r\n\r\nAt iterate    7    f=  4.81956D-01    |proj g|=  3.76945D+01\r\n\r\nAt iterate    8    f=  6.81301D-02    |proj g|=  2.20645D+01\r\n\r\nAt iterate    9    f=  1.34639D-02    |proj g|=  8.79078D+00\r\n\r\n           * * *\r\n\r\nTit   = total number of iterations\r\nTnf   = total number of function evaluations\r\nTnint = total number of segments explored during Cauchy searches\r\nSkip  = number of BFGS updates skipped\r\nNact  = number of active bounds at final generalized Cauchy point\r\nProjg = norm of the final projected gradient\r\nF     = final function value\r\n\r\n           * * *\r\n\r\n   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F\r\n    4      9     12      1     0     0   8.791D+00   1.346D-02\r\n  F =  1.346394518877836E-002\r\n\r\nCONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH             \r\n\r\n Cauchy                time 0.000E+00 seconds.\r\n Subspace minimization time 0.000E+00 seconds.\r\n Line search           time 0.000E+00 seconds.\r\n\r\n Total User time 0.000E+00 seconds.\r\n```\r\nthe code writes \"iterate\" instead of \"iteration\" (a typo).  This is an annoyance as it is not very professional, but it does not break anything.  The separate timings for Cauchy, subspace minimization, and line search are also always zero.  Same for user time.  It would be nice for these to not be a filler values since this is misleading, but same thing - this is creature comfort problem, not an accuracy problem.\r\n\r\nMy application is in a topic known as wavefront sensing, or [phase retrieval](https://en.wikipedia.org/wiki/Phase_retrieval).  We put in some number of parameters, and the landscape of the cost function is highly evolutionary as the optimizer traverses the space.  As a result, when designing a cost function or debugging code it is desirable to know the state of the optimizer, even without knowing its internal mechanisms, as it walks the space.  To do this I do the following:\r\n\r\n1.  Save this output text from L-BFGS-B, and scrape the f= values from it.\r\n2.  Use a callback that stores parameter vectors.\r\n\r\nI expect these two things to have the same length always, and depend on cost[n] being associated with all_x[n] without registration errors between these two blocks of data.\r\n\r\nIn my particular application (I can share code on request, but 1000-2000 lines are called to implement my routine split across a few modules; there is no easy \"def foo()\" analog that I have found to reproduce this) I am having trouble understanding why my cost function appears to not be correlated, based on the output of l-bfgs-b, to the closeness of the current iteration input to truth:\r\n\r\n![cost function is not correlated with true error metric](https://i.imgur.com/Br9OHCg.jpg)\r\n\r\nThis has cost me dozens of hours of debugging time, only to learn that **something is wrong with L-BFGS-B that causes the output of options={'disp': True} to not be the the cost function at a given parameter vector.**\r\n\r\nBecause I capture stdout, I am not able to use print.  I have embedded a line in my cost function that appends the current parameter vector and cost to a .txt file and resorted to matching values in that to L-BFGS-B print output to see what is wrong.  The text from that is below, with numbered lines\r\n\r\n```\r\n00. params = [ 0.  0.  0.  0.] \t f(x) = 27.5369873872\r\n\r\n01. params = [  1.00000000e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00] \t f(x) = 27.5369884562\r\n\r\n02. params = [  0.00000000e+00   1.00000000e-08   0.00000000e+00   0.00000000e+00] \t f(x) = 27.5369867324\r\n\r\n03. params = [  0.00000000e+00   0.00000000e+00   1.00000000e-08   0.00000000e+00] \t f(x) = 27.5369870096\r\n\r\n04. params = [  0.00000000e+00   0.00000000e+00   0.00000000e+00   1.00000000e-08] \t f(x) = 27.5369875808\r\n\r\n05. params = [-0.80769268  0.49476373  0.28537275 -0.14630029] \t f(x) = 86.4418490547\r\n\r\n06. params = [-0.80769267  0.49476373  0.28537275 -0.14630029] \t f(x) = 86.4418490135\r\n\r\n07. params = [-0.80769268  0.49476374  0.28537275 -0.14630029] \t f(x) = 86.441849366\r\n\r\n08. params = [-0.80769268  0.49476373  0.28537276 -0.14630029] \t f(x) = 86.4418499781\r\n\r\n09. params = [-0.80769268  0.49476373  0.28537275 -0.14630028] \t f(x) = 86.4418482602\r\n\r\n10. params = [-0.16406864  0.1005026   0.05796848 -0.02971834] \t f(x) = 22.714940612\r\n\r\n11. params = [-0.16406863  0.1005026   0.05796848 -0.02971834] \t f(x) = 22.7149388457\r\n\r\n12. params = [-0.16406864  0.10050261  0.05796848 -0.02971834] \t f(x) = 22.7149405671\r\n\r\n13. params = [-0.16406864  0.1005026   0.05796849 -0.02971834] \t f(x) = 22.7149404753\r\n\r\n14. params = [-0.16406864  0.1005026   0.05796848 -0.02971833] \t f(x) = 22.7149421094\r\n\r\n15. params = [-0.10087717  0.09158391  0.05575493 -0.06632038] \t f(x) = 12.0195875306\r\n\r\n16. params = [-0.10087716  0.09158391  0.05575493 -0.06632038] \t f(x) = 12.0195863232\r\n\r\n17. params = [-0.10087717  0.09158392  0.05575493 -0.06632038] \t f(x) = 12.0195878653\r\n\r\n18. params = [-0.10087717  0.09158391  0.05575494 -0.06632038] \t f(x) = 12.0195883001\r\n\r\n19. params = [-0.10087717  0.09158391  0.05575493 -0.06632037] \t f(x) = 12.0195867159\r\n\r\n20. params = [ 0.15842029 -0.01548538 -0.01632799 -0.07159999] \t f(x) = 48.7174210299\r\n\r\n21. params = [ 0.1584203  -0.01548538 -0.01632799 -0.07159999] \t f(x) = 48.7174233933\r\n\r\n22. params = [ 0.15842029 -0.01548537 -0.01632799 -0.07159999] \t f(x) = 48.717421743\r\n\r\n23. params = [ 0.15842029 -0.01548538 -0.01632798 -0.07159999] \t f(x) = 48.7174193993\r\n\r\n24. params = [ 0.15842029 -0.01548538 -0.01632799 -0.07159998] \t f(x) = 48.7174245544\r\n\r\n25. params = [-0.05512618  0.07269238  0.04303647 -0.06725192] \t f(x) = 8.05081234605\r\n\r\n26. params = [-0.05512617  0.07269238  0.04303647 -0.06725192] \t f(x) = 8.05081192026\r\n\r\n27. params = [-0.05512618  0.07269239  0.04303647 -0.06725192] \t f(x) = 8.0508114712\r\n\r\n28. params = [-0.05512618  0.07269238  0.04303648 -0.06725192] \t f(x) = 8.05081281076\r\n\r\n29. params = [-0.05512618  0.07269238  0.04303647 -0.06725191] \t f(x) = 8.05081206742\r\n\r\n30. params = [-0.02149653  0.09172596  0.0267816  -0.06652566] \t f(x) = 5.0034401551\r\n\r\n31. params = [-0.02149652  0.09172596  0.0267816  -0.06652566] \t f(x) = 5.00344002938\r\n\r\n32. params = [-0.02149653  0.09172597  0.0267816  -0.06652566] \t f(x) = 5.00343999403\r\n\r\n33. params = [-0.02149653  0.09172596  0.02678161 -0.06652566] \t f(x) = 5.00344079846\r\n\r\n34. params = [-0.02149653  0.09172596  0.0267816  -0.06652565] \t f(x) = 5.00343931285\r\n\r\n35. params = [ 0.03012534  0.12108955 -0.01347029 -0.04719621] \t f(x) = 3.70939759272\r\n\r\n36. params = [ 0.03012535  0.12108955 -0.01347029 -0.04719621] \t f(x) = 3.70939819132\r\n\r\n37. params = [ 0.03012534  0.12108956 -0.01347029 -0.04719621] \t f(x) = 3.70939842652\r\n\r\n38. params = [ 0.03012534  0.12108955 -0.01347028 -0.04719621] \t f(x) = 3.70939740919\r\n\r\n39. params = [ 0.03012534  0.12108955 -0.01347029 -0.0471962 ] \t f(x) = 3.70939657281\r\n\r\n40. params = [ 0.00300894  0.11308784  0.00337076 -0.03858502] \t f(x) = 1.57818376805\r\n\r\n41. params = [ 0.00300895  0.11308784  0.00337076 -0.03858502] \t f(x) = 1.578184007\r\n\r\n42. params = [ 0.00300894  0.11308785  0.00337076 -0.03858502] \t f(x) = 1.57818379941\r\n\r\n43. params = [ 0.00300894  0.11308784  0.00337077 -0.03858502] \t f(x) = 1.57818414645\r\n\r\n44. params = [ 0.00300894  0.11308784  0.00337076 -0.03858501] \t f(x) = 1.5781830571\r\n\r\n45. params = [-0.01561316  0.12143126  0.00100096 -0.01330919] \t f(x) = 0.481955650202\r\n\r\n46. params = [-0.01561315  0.12143126  0.00100096 -0.01330919] \t f(x) = 0.481955273258\r\n\r\n47. params = [-0.01561316  0.12143127  0.00100096 -0.01330919] \t f(x) = 0.481955840221\r\n\r\n48. params = [-0.01561316  0.12143126  0.00100097 -0.01330919] \t f(x) = 0.481955839908\r\n\r\n49. params = [-0.01561316  0.12143126  0.00100096 -0.01330918] \t f(x) = 0.481955373719\r\n\r\n50. params = [-0.00041042  0.12401147 -0.00485423  0.00207093] \t f(x) = 0.0681301442419\r\n\r\n51. params = [-0.00041041  0.12401147 -0.00485423  0.00207093] \t f(x) = 0.0681301690496\r\n\r\n52. params = [-0.00041042  0.12401148 -0.00485423  0.00207093] \t f(x) = 0.0681301089522\r\n\r\n53. params = [-0.00041042  0.12401147 -0.00485422  0.00207093] \t f(x) = 0.0681299235971\r\n\r\n54. params = [-0.00041042  0.12401147 -0.00485423  0.00207094] \t f(x) = 0.0681302701249\r\n\r\n55. params = [-0.00144057  0.12278377 -0.00124594 -0.00264166] \t f(x) = 0.0134639451888\r\n\r\n56. params = [-0.00144056  0.12278377 -0.00124594 -0.00264166] \t f(x) = 0.0134639411113\r\n\r\n57. params = [-0.00144057  0.12278378 -0.00124594 -0.00264166] \t f(x) = 0.0134638572809\r\n\r\n58. params = [-0.00144057  0.12278377 -0.00124593 -0.00264166] \t f(x) = 0.0134639506651\r\n\r\n59. params = [-0.00144057  0.12278377 -0.00124594 -0.00264165] \t f(x) = 0.0134639172132\r\n```\r\n\r\nThe callback creates the following list:\r\n```\r\n[array([ 0.,  0.,  0.,  0.]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166]),\r\n  array([-0.00144057,  0.12278377, -0.00124594, -0.00264166])]\r\n```\r\nI have reproduced this for completeness, but the important lines are 00, the initial guess, which matches iterate 0 in the output text from the optimizer.  Iteration 1 matches any of line 12-14.  The parameter vectors are similar to:\r\n```\r\n[-0.16406864  0.10050261  0.05796848 -0.02971834]\r\n```\r\nThe callback contains this input array at that iteration:\r\n```\r\n[-0.00144057,  0.12278377, -0.00124594, -0.00264166]\r\n```\r\nWhich would be associated with a cost function 0.013, not 22 and change.  As a result, I am unable to trust the tools I have to diagnose a problem with them.  The output of the L-BFGS-B print simply is not the value of the cost function for the parameter vector at that iteration.  I believe the problem is something like as follows:\r\n\r\n1.  The callback is called at the end of a line search before beginning a new gradient hunt\r\n2.  The f= is written at the start of a line search\r\n\r\nBut I cannot trust the documentation of scipy to be correct that the callback is called at the end of an iteration (~= end of line search in BFGS), and I cannot trust lbfgsb.f to be error free in this way.\r\n\r\nAre there plans to improve the optimization module in a \"months not years\" timespan?  Is anyone willing to rewrite lbfgsb.f or even port it to C/C++ or pure python to solve these problems?  In commercialized applications, it is often desirable for a user to keep their finger on the pulse by monitoring convergence and potentially stopping optimization, hand tuning the inputs, and then continuing the work.  The architecture of optimize.minimize is prohibitive to this.  It would be nice to see a context manager based implementation that allows interruption of the optimizer without losing the current state, as this sort of behavior is only possible at the moment through use of a complicated callback function."},"comment":{"url":"https://api.github.com/repos/scipy/scipy/issues/comments/364244894","html_url":"https://github.com/scipy/scipy/issues/8373#issuecomment-364244894","issue_url":"https://api.github.com/repos/scipy/scipy/issues/8373","id":364244894,"user":{"login":"matthew-brett","id":67612,"avatar_url":"https://avatars2.githubusercontent.com/u/67612?v=4","gravatar_id":"","url":"https://api.github.com/users/matthew-brett","html_url":"https://github.com/matthew-brett","followers_url":"https://api.github.com/users/matthew-brett/followers","following_url":"https://api.github.com/users/matthew-brett/following{/other_user}","gists_url":"https://api.github.com/users/matthew-brett/gists{/gist_id}","starred_url":"https://api.github.com/users/matthew-brett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthew-brett/subscriptions","organizations_url":"https://api.github.com/users/matthew-brett/orgs","repos_url":"https://api.github.com/users/matthew-brett/repos","events_url":"https://api.github.com/users/matthew-brett/events{/privacy}","received_events_url":"https://api.github.com/users/matthew-brett/received_events","type":"User","site_admin":false},"created_at":"2018-02-08T20:53:22Z","updated_at":"2018-02-08T20:53:22Z","author_association":"CONTRIBUTOR","body":"To follow up on what @stefanv is saying - there's a really strong culture in the open-source projects I know, not to block or suppress people, no matter how annoying you may find them or their comments.   Open-source - like Scipy - runs on the fuel of 'everyone is welcome'.   That means it appears to us old-timers as ugly and unhappy if you block someone.   I see you can do it, but we're humbly asking you to be patient; we'll try and help if it gets heated."}}
{"actor":{"display_login":"matthew-brett"}}