{"action":"created","issue":{"url":"https://api.github.com/repos/basho/yokozuna/issues/700","repository_url":"https://api.github.com/repos/basho/yokozuna","labels_url":"https://api.github.com/repos/basho/yokozuna/issues/700/labels{/name}","comments_url":"https://api.github.com/repos/basho/yokozuna/issues/700/comments","events_url":"https://api.github.com/repos/basho/yokozuna/issues/700/events","html_url":"https://github.com/basho/yokozuna/pull/700","id":186670190,"number":700,"title":"Performance fixes for Solr integration [JIRA: RIAK-2955]","user":{"login":"JeetKunDoug","id":682821,"avatar_url":"https://avatars.githubusercontent.com/u/682821?v=3","gravatar_id":"","url":"https://api.github.com/users/JeetKunDoug","html_url":"https://github.com/JeetKunDoug","followers_url":"https://api.github.com/users/JeetKunDoug/followers","following_url":"https://api.github.com/users/JeetKunDoug/following{/other_user}","gists_url":"https://api.github.com/users/JeetKunDoug/gists{/gist_id}","starred_url":"https://api.github.com/users/JeetKunDoug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeetKunDoug/subscriptions","organizations_url":"https://api.github.com/users/JeetKunDoug/orgs","repos_url":"https://api.github.com/users/JeetKunDoug/repos","events_url":"https://api.github.com/users/JeetKunDoug/events{/privacy}","received_events_url":"https://api.github.com/users/JeetKunDoug/received_events","type":"User","site_admin":false},"labels":[{"id":197402700,"url":"https://api.github.com/repos/basho/yokozuna/labels/JIRA:%20Needs%20Review","name":"JIRA: Needs Review","color":"ededed","default":false},{"id":348873018,"url":"https://api.github.com/repos/basho/yokozuna/labels/JIRA:%20PR%20Review","name":"JIRA: PR Review","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-11-01T22:56:29Z","updated_at":"2016-11-04T20:34:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/basho/yokozuna/pulls/700","html_url":"https://github.com/basho/yokozuna/pull/700","diff_url":"https://github.com/basho/yokozuna/pull/700.diff","patch_url":"https://github.com/basho/yokozuna/pull/700.patch"},"body":"# Several major performance improvements for Solr integration:\r\n- Remove Delete By Query calls whenever possible as it's a performance killer on Solr\r\n- Split hashtree update out of the \"Drain Timeout\" window as we were killing hashtree builds unnecessarily.\r\n- Fix issue where `yz_index_hashtree:fold_keys` was flooding the yz_index_hashtree mailbox by implementing the \"90 async/1 sync\" call pattern as other `index` and `delete` callers used.\r\n\r\n## Removed usages of Delete By Query\r\nPrior to this fix, whenever we would add a document to Solr that might contain siblings, we would add a delete by query operation to the batch we send to Solr (in a single HTTP request).  This had a significant impact on performance, especially when objects being updated had a lot of siblings.  (We found that deleting by query would contend for locks with the segment scheduler, and in general adding deletesByQuery to high workloads seems like a bad idea).\r\n\r\nThe fix involves changing deleteByQuery to deleting documents in Solr by their document ids, which has much better performance than deleting by query..  The challenge is that when adding an object, we didn't have the sibling vtags to generate a document id.  We therefore made a change to riak_kv (basho/riak_kv#1520) which, when updating an object, also sends the old object being replaced.  From that old object, we can extract any siblings and generate associated document ids to delete in Solr.\r\n\r\n## Hashtree update/drain timeout split\r\nThe process of doing a KV->Yokozuna hastree comparison takes several steps, including draining the relevant Solr queues before snapshotting the Yokozuna hashtree so that its state can be as close as possible to the KV hashtree. In order to not block the exchange indefinitely, a \"Drain Timeout\" was included in this process. However, as initially developed, the drain was not the only thing being measured in this timeout window. It also included the hashtree:update call, which, on a heavily-loaded system, can take several minutes. This would cause the \"drain\" to time out and be forcibly cancelled, even though the drain itself had only taken a few hundreds of milliseconds.\r\n\r\n`yz_drain_fsm`, `yz_drain_mgr`,  and `yz_exchange_fsm` were was reworked to split the Solr drain from the hashtree update, resolving the issue of drain timeouts under normal operating conditions.\r\n\r\n## Hashtree asyc/sync change\r\nDuring performance testing, it was discovered that the `yz_index_hashtree:fold_keys` \r\nfunction could swamp the mailbox of the `yz_index_hashtree` so that other processes \r\ncould not make progress. This commit makes `fold_keys` use the same\r\n\"90 async + 1 sync\" call pattern that `yz_kv:index` used. That logic\r\n(which lived in `yz_kv`) has been moved to `yz_index_hashtree` and\r\nshared by the new `index` and `delete` calls that do not take an explicit\r\n\"call mode\" parameter.\r\nFuture work is scheduled to figure out how to make this pattern more generic \r\n(as it's used in several places) and to eliminate the use of the caller's\r\nprocess dictionary for storing hashtree tokens.\r\n\r\n## Minor improvements included in this Pull request\r\n\r\n* Added logging for clear and exchange trees, for audit of administrative operations.\r\n* Modified the \"Will Repair\" log in the yz_exchange_fsm to track the \"direction\" of repair, viz., whether the repair resulted in a delete of Solr data or an add/update to Solr.\r\n* Corrected the list of `required_queues` in `yz_solrq_sup` to include both _current_ and _future_ indexes the node owned. This prevented premature shutdown of a solrq worker/helper pair that was started by writes to a _future_ vnode (one in the process of being handed off to the node in question).\r\n* Remove exometer statistics when an index is removed. Before, if an index was later re-added, the fuse creation would fail, eventually causing the node to crash. Test updated in 315b7464fa9996e56237de21050fd5f7b1f4eb6d\r\n* Monitor `yz_solrq_drain_fsm` from the queues being drained in case the drain_fsm crashes. Before, it was possible for a queue to get stuck in `wait_for_drain_complete` state if the drain fsm crashed before the drain complete messages were sent.\r\n\r\n## Review Checklist:\r\n\r\n1. [X] Observability\r\n    - Observability was not changed in this PR - stats already existed for the Solr operations, and there were no major changes in logging required.\r\n1. [X] Performance\r\n    - This PR has been driven by performance testing conducted on AWS over the course of several weeks, and has been shown to improve performance dramatically (In our testing, 4X when writing siblings in large numbers).\r\n1. [X] Fault-tolerance\r\n    - Added some additional monitoring around the `yz_solrq_drain_fsm` to prevent queues from being stuck in `wait_for_drain_complete` state.\r\n1. [X] Testing\r\n    - Updated many tests to account for different behavior.\r\n1. [X] Code Clarity/Quality\r\n    - Wherever possible, code was refactored along the way and reviewed on an ongoing basis for clarity and quality.\r\n1. [ ] Code Documentation\r\n    - TODO: Update `batching.md` and associated docs to incorporate changes.\r\n1. [X] External Documentation\r\n    - No impact\r\n1. [X] Release Notes\r\n    - No impact - added a strictly internal setting `{yokozuna, anti_entropy_max_async}` that mirrors a similar setting in riak_kv (and will take the KV value if the yokozuna value isn't provided), but the setting in riak_kv isn't documented and is essentially never changed."},"comment":{"url":"https://api.github.com/repos/basho/yokozuna/issues/comments/258540502","html_url":"https://github.com/basho/yokozuna/pull/700#issuecomment-258540502","issue_url":"https://api.github.com/repos/basho/yokozuna/issues/700","id":258540502,"user":{"login":"thumbot","id":20921557,"avatar_url":"https://avatars.githubusercontent.com/u/20921557?v=3","gravatar_id":"","url":"https://api.github.com/users/thumbot","html_url":"https://github.com/thumbot","followers_url":"https://api.github.com/users/thumbot/followers","following_url":"https://api.github.com/users/thumbot/following{/other_user}","gists_url":"https://api.github.com/users/thumbot/gists{/gist_id}","starred_url":"https://api.github.com/users/thumbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thumbot/subscriptions","organizations_url":"https://api.github.com/users/thumbot/orgs","repos_url":"https://api.github.com/users/thumbot/repos","events_url":"https://api.github.com/users/thumbot/events{/privacy}","received_events_url":"https://api.github.com/users/thumbot/received_events","type":"User","site_admin":false},"created_at":"2016-11-04T20:34:37Z","updated_at":"2016-11-04T20:34:37Z","body":"|||||\n------------ | -------------|------------ | ------------- | -------------\nfd-experimental-factor-deletes 4345b67 | :arrow_right: | develop-2.2 c7d3275 | :clock1: in_progress"}}
{"actor":{"display_login":"thumbot"}}