{"action":"created","issue":{"url":"https://api.github.com/repos/tokio-rs/tokio/issues/653","repository_url":"https://api.github.com/repos/tokio-rs/tokio","labels_url":"https://api.github.com/repos/tokio-rs/tokio/issues/653/labels{/name}","comments_url":"https://api.github.com/repos/tokio-rs/tokio/issues/653/comments","events_url":"https://api.github.com/repos/tokio-rs/tokio/issues/653/events","html_url":"https://github.com/tokio-rs/tokio/pull/653","id":362334969,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE3MDgzNTAw","number":653,"title":"[WIP] Assign a home worker to each task","user":{"login":"stjepang","id":1303843,"node_id":"MDQ6VXNlcjEzMDM4NDM=","avatar_url":"https://avatars2.githubusercontent.com/u/1303843?v=4","gravatar_id":"","url":"https://api.github.com/users/stjepang","html_url":"https://github.com/stjepang","followers_url":"https://api.github.com/users/stjepang/followers","following_url":"https://api.github.com/users/stjepang/following{/other_user}","gists_url":"https://api.github.com/users/stjepang/gists{/gist_id}","starred_url":"https://api.github.com/users/stjepang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stjepang/subscriptions","organizations_url":"https://api.github.com/users/stjepang/orgs","repos_url":"https://api.github.com/users/stjepang/repos","events_url":"https://api.github.com/users/stjepang/events{/privacy}","received_events_url":"https://api.github.com/users/stjepang/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-20T19:36:06Z","updated_at":"2018-09-25T18:42:43Z","closed_at":"2018-09-25T18:42:43Z","author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/tokio-rs/tokio/pulls/653","html_url":"https://github.com/tokio-rs/tokio/pull/653","diff_url":"https://github.com/tokio-rs/tokio/pull/653.diff","patch_url":"https://github.com/tokio-rs/tokio/pull/653.patch"},"body":"This is still very much a WIP, but I'm submitting early to get feedback.\r\n\r\nSo far, performance improvements seem to be pretty good. The running time of `bench-poll` has gone down from 460 ms to 390 ms, but this is just the beginning.\r\n\r\nIn `reactor::Builder`, rather than creating one reactor that is running in a background thread, now we create one reactor per worker thread. Each `Reactor` (remember, it impls `Park`) is hooked up to the per-worker `Timer` (also impls `Park` but can embed another `Park` inside it).\r\n\r\nWhen we spawn a new task, it is submitted to a random worker's queue and that worker becomes its home worker. If we steal a task from another worker, poll the future, and it isn't completed yet, we *always* push the `Arc<Task>` back into its home worker's queue. In other words, every uncompleted task can only be in its home worker's queue. Tasks never move between different worker queues.\r\n\r\nSo what happens when we steal a task from another worker? Before polling the task, we must call `tokio_reactor::with_default` to begin using its home worker's reactor. Also recall that we cannot nest calls to `with_default`. Before this PR, we used to set up the reactor, clock, and timer inside the `around_worker` callback once to initialize the worker thread, but now we must be more clever.\r\n\r\nCurrently, this PR uses a rather simple and stupid strategy. The `around_worker` callback is now called just before executing each task. Previously, the `around_worker` callback was called only once before the worker thread started executing. Since that callback sets up the default reactor and we have multiple reactors, now we must call it before running each task in order to choose its appropriate reactor.\r\n\r\nCalling the `around_worker` callback every time we run a task is very slow. The good thing is that even despite the slowness we got performance wins. :) So anyways, how do we fix that? I think we should call the `around_worker` callback only once while we're popping tasks from our worker's queue. Once we run out of tasks and steal a task from another worker, that's when we should call the callback again to set up a different reactor. So in the common case we shouldn't call the callback very often.\r\n\r\nThe major challenge here (but not impossible to solve) is that due to the callback nature of `around_worker`, we need to keep some state in `Worker`. For example, in order to run a stolen task, we store the `Arc<Task>` and `Arc<Notifier>` in a `Worker`'s private field, run the `around_worker` callback, which in turn calls our `run` method. Inside `run` we read the `Arc<Task>` and `Arc<Notifier>` and poll the task. Once we're done we store the result (the result is what to do with the task: reschedule it, drop it, or do something else) into a local field, too. We return from `run`, return from the `around_worker` callback, and read the result, then figure out how to proceed with that.\r\n\r\nThings get much more hairy when we do the optimization mention above (which is call the `around_worker` callback only once while we're running tasks from our own queue).\r\n\r\nThere are several API changes that are not strictly necessary, but would make life much easier:\r\n\r\n* Make `with_default` functions reentrant (currently attempting to nest calls to `with_default` will panic). So when we set a new default reactor/timer/clock/whatever, we temporarily swap out the old one and put it back once we're done.\r\n\r\n* Make it possible to return a value in the `around_worker` callback, i.e. change `F: FnOnce(...)` to `F: FnOnce(...) -> R`.\r\n\r\n* Make the actual `usize` inside `WorkerId` public. See how we I use it in `reactor::Builder::build`. Rayon does something similar in the form of [`current_thread_index`](https://docs.rs/rayon/1.0.2/rayon/struct.ThreadPool.html#method.current_thread_index).\r\n\r\nWhat do you think, should we do that?\r\n\r\nAlso, what should we do about the getter method `Runtime::reactor` (and its deprecated equivalent `Runtime::handle`)? It doesn't make much sense once we go with multiple reactors."},"comment":{"url":"https://api.github.com/repos/tokio-rs/tokio/issues/comments/424456332","html_url":"https://github.com/tokio-rs/tokio/pull/653#issuecomment-424456332","issue_url":"https://api.github.com/repos/tokio-rs/tokio/issues/653","id":424456332,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDQ1NjMzMg==","user":{"login":"stjepang","id":1303843,"node_id":"MDQ6VXNlcjEzMDM4NDM=","avatar_url":"https://avatars2.githubusercontent.com/u/1303843?v=4","gravatar_id":"","url":"https://api.github.com/users/stjepang","html_url":"https://github.com/stjepang","followers_url":"https://api.github.com/users/stjepang/followers","following_url":"https://api.github.com/users/stjepang/following{/other_user}","gists_url":"https://api.github.com/users/stjepang/gists{/gist_id}","starred_url":"https://api.github.com/users/stjepang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stjepang/subscriptions","organizations_url":"https://api.github.com/users/stjepang/orgs","repos_url":"https://api.github.com/users/stjepang/repos","events_url":"https://api.github.com/users/stjepang/events{/privacy}","received_events_url":"https://api.github.com/users/stjepang/received_events","type":"User","site_admin":false},"created_at":"2018-09-25T18:42:43Z","updated_at":"2018-09-25T18:42:43Z","author_association":"COLLABORATOR","body":"Closing in favor of smaller PRs."}}
{"actor":{"display_login":"stjepang"}}