{"action":"created","issue":{"url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/46","repository_url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl","labels_url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/46/comments","events_url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/46/events","html_url":"https://github.com/robertfeldt/BlackBoxOptim.jl/pull/46","id":150134915,"number":46,"title":"[RFC/WIP] Asynchronous ParallelEvaluator","user":{"login":"alyst","id":348591,"avatar_url":"https://avatars.githubusercontent.com/u/348591?v=3","gravatar_id":"","url":"https://api.github.com/users/alyst","html_url":"https://github.com/alyst","followers_url":"https://api.github.com/users/alyst/followers","following_url":"https://api.github.com/users/alyst/following{/other_user}","gists_url":"https://api.github.com/users/alyst/gists{/gist_id}","starred_url":"https://api.github.com/users/alyst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alyst/subscriptions","organizations_url":"https://api.github.com/users/alyst/orgs","repos_url":"https://api.github.com/users/alyst/repos","events_url":"https://api.github.com/users/alyst/events{/privacy}","received_events_url":"https://api.github.com/users/alyst/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-04-21T17:06:54Z","updated_at":"2016-04-22T13:05:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/pulls/46","html_url":"https://github.com/robertfeldt/BlackBoxOptim.jl/pull/46","diff_url":"https://github.com/robertfeldt/BlackBoxOptim.jl/pull/46.diff","patch_url":"https://github.com/robertfeldt/BlackBoxOptim.jl/pull/46.patch"},"body":"The PR changes the `ParallelEvaluator` to be asynchronous:\r\n 1. The candidate is submitted to the queue by `async_update_fitness()`, which immediately returns the fitness calculation job Id and the job gets submitted to one of the available worker processes.\r\n 2. The job status could be tested by the `isready()` function. Also, the candidates with the recently calculated fitness could be processed by calling `process_completed() do job_id, candidate <custom code> end` routine. (Note: if `isready(job_id)` is called and it returns true, this job would not be enumerated by a call to `process_completed()` anymore and it's up for the caller to take actions).\r\n\r\nThe old synchronous API (used by NES) is still supported.\r\n\r\n`BorgMOEA` is updated to support asynchronous `ParallelEvaluator`: the algorithm runs on master, the new individual is generated by recombination and sent to the parallel evaluation, the further processing of the individual (updating the population and the archive) is postponed until its fitness is evaluated. This should improve the performance for the problems with computationally intensive fitness functions. I see the speed up for my problems, although it's not linear.\r\n\r\nThere are several reasons why it's RFC/WIP:\r\n  1. Unfortunately, the current julia parallelization scheme (remote channels/remote refs implemented using the messages sent through pipes) doesn't allow for an efficient intensive communication between the parallel processes (the profiling shows that the overhead of serializing/deserializing the message and writing/reading to UV pipe stream is too high), so after trying several implementations (this journey could be tracked as I haven't squashed the commits yet) I've ended up using shared arrays for the data and status exchange. Using threads (or maybe better messaging implementation) would be much more efficient, but it's not yet mainlined to Julia.\r\n  2. The logic of `async_update_fitness()`/`isready()`/`process_completed()` is somewhat confusing, but so far I had no better ideas given that `Future{T}` approach would create too much overhead.\r\n  3. As the communication is shmem-based, the workers use \"busy wait\" to continuously poll for the new jobs. To avoid 100% CPU usage, the workers have to be killed/suspended after the optimization is finished. This is done by the new `shutdown!()` call. However, `OptController/OptRunController` interface assumes the same optimization method (and its evaluator) could be reused in several `OptRunController` runs. With the `ParallelEvaluator` it is currently not possible, because all the workers are killed by the `shutdown!()` call at the end of the first run. I see two alternatives:\r\n    * introduce `start!(Evaluator)`/`shutdown!(Evaluator)` methods that needs to be called immediately before/after method iterations (no-ops for the normal evaluators);\r\n    * implement lazy initialization of the workers in `ParallelEvaluator`, detect idle periods, hybernate the workers when idle (using `wait()`), resume on the new fitness evaluation request. Given the state of the parallelism in julia it's not so easy to get it working nicely."},"comment":{"url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/comments/213421692","html_url":"https://github.com/robertfeldt/BlackBoxOptim.jl/pull/46#issuecomment-213421692","issue_url":"https://api.github.com/repos/robertfeldt/BlackBoxOptim.jl/issues/46","id":213421692,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars.githubusercontent.com/u/2354108?v=3","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2016-04-22T13:05:16Z","updated_at":"2016-04-22T13:05:16Z","body":"\n[![Coverage Status](https://coveralls.io/builds/5893106/badge)](https://coveralls.io/builds/5893106)\n\nCoverage increased (+4.4%) to 70.253% when pulling **88f9ff1adad3c9761adc3af8eb6e7cb0e05039ac on alyst:async_parallel_eval** into **bd0745c543ad4b5f5c03a6b459b3c9a97edef27f on robertfeldt:master**.\n"}}