{"action":"created","comment":{"url":"https://api.github.com/repos/status-im/status-react/pulls/comments/187633204","pull_request_review_id":119457191,"id":187633204,"diff_hunk":"@@ -375,8 +328,14 @@\n (reg-sub\n  :get-last-message\n  (fn [[_ chat-id]]\n-   (subscribe [:get-ordered-chat-messages chat-id]))\n- first)\n+   (subscribe [:get-chat chat-id]))\n+ (fn [{:keys [messages message-groups]}]\n+   (->> (sort-message-groups message-groups messages)\n+        first\n+        second","path":"src/status_im/chat/subs.cljs","position":120,"original_position":120,"commit_id":"53eb050ed4ffd71eb65ce3eb85cefb3481970125","original_commit_id":"53eb050ed4ffd71eb65ce3eb85cefb3481970125","user":{"login":"rasom","id":2364994,"avatar_url":"https://avatars2.githubusercontent.com/u/2364994?v=4","gravatar_id":"","url":"https://api.github.com/users/rasom","html_url":"https://github.com/rasom","followers_url":"https://api.github.com/users/rasom/followers","following_url":"https://api.github.com/users/rasom/following{/other_user}","gists_url":"https://api.github.com/users/rasom/gists{/gist_id}","starred_url":"https://api.github.com/users/rasom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rasom/subscriptions","organizations_url":"https://api.github.com/users/rasom/orgs","repos_url":"https://api.github.com/users/rasom/repos","events_url":"https://api.github.com/users/rasom/events{/privacy}","received_events_url":"https://api.github.com/users/rasom/received_events","type":"User","site_admin":false},"body":"probably even `second first :message-id`","created_at":"2018-05-11T14:35:19Z","updated_at":"2018-05-11T14:35:19Z","html_url":"https://github.com/status-im/status-react/pull/4221#discussion_r187633204","pull_request_url":"https://api.github.com/repos/status-im/status-react/pulls/4221","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/status-im/status-react/pulls/comments/187633204"},"html":{"href":"https://github.com/status-im/status-react/pull/4221#discussion_r187633204"},"pull_request":{"href":"https://api.github.com/repos/status-im/status-react/pulls/4221"}},"in_reply_to_id":187628003},"pull_request":{"url":"https://api.github.com/repos/status-im/status-react/pulls/4221","id":187447959,"html_url":"https://github.com/status-im/status-react/pull/4221","diff_url":"https://github.com/status-im/status-react/pull/4221.diff","patch_url":"https://github.com/status-im/status-react/pull/4221.patch","issue_url":"https://api.github.com/repos/status-im/status-react/issues/4221","number":4221,"state":"open","locked":false,"title":"Fast message grouping/sorting","user":{"login":"janherich","id":1910609,"avatar_url":"https://avatars1.githubusercontent.com/u/1910609?v=4","gravatar_id":"","url":"https://api.github.com/users/janherich","html_url":"https://github.com/janherich","followers_url":"https://api.github.com/users/janherich/followers","following_url":"https://api.github.com/users/janherich/following{/other_user}","gists_url":"https://api.github.com/users/janherich/gists{/gist_id}","starred_url":"https://api.github.com/users/janherich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janherich/subscriptions","organizations_url":"https://api.github.com/users/janherich/orgs","repos_url":"https://api.github.com/users/janherich/repos","events_url":"https://api.github.com/users/janherich/events{/privacy}","received_events_url":"https://api.github.com/users/janherich/received_events","type":"User","site_admin":false},"body":"[comment]: # (Please replace ... with your information. Remove < and >)\r\n[comment]: # (To auto-close issue on merge, please insert the related issue number after # i.e fixes #566)\r\n\r\nThis PR implements significantly faster re-computation of chat messages stream.\r\nPreviously we always re-computed chat messages stream from scratch each time anything in `message-id->message` datastructure in `app-db` changed, that means even if just `:user-status` of one message was altered.\r\nThis computation was quite heavy especially because we needed to group messages according to their datemark (time/day-relative function) + need to re-sort all of them from scratch (grouping being ~ 3 more expensive then re-sorting).\r\nSo instead of doing that in subscriptions, this PR introduces new chat-specific `app-db` index called `:message-groups` which contains correctly grouped and sorted references to messages (+ additional expensive to compute data which is not \"live\", eq never changes, currently only `:timestamp-str`), like this:\r\n```clj\r\n{:message-groups {\"today\" '({:message-id 1 :timestamp-str \"12:00 AM\"}\r\n                            {:message-id 2 :timestamp-str \"3:00 PM\"})}}\r\n```\r\nEverytime message is added to db through `status-im.chat.models.message/add-message` function, this index is updated (no re-grouping each time necessary) and messages in the affected datetime group are re-sorted (that's unfortunately necessary because of the possibility of out-of-band messages, nevertheless it's still better then before because at least, only one day worth of messages is re-sorted, not the **whole** currently loaded message history).\r\n\r\nCare needs to be taken whenever the message is added/removed from db to update the `:message-group` index, but not when live message data (like statuses, etc.) is updated, that's exactly as easy as before.\r\n\r\nSubscriptions are now significantly leaner and don't need to do any grouping and much less sorting (just sorting the groups itself is necessary).\r\n\r\nDemonstrating the difference in time to re-compute between the \"old way\" and \"new way\", it's around 4x improvement (chat with ~300 messages, likely to be much more significant on slower platform then iOS simulator:)\r\n![screen shot 2018-05-11 at 13 31 36](https://user-images.githubusercontent.com/1910609/39923849-a61d56e0-5525-11e8-8f23-9ee2baeeb02b.png)\r\n\r\nFuture improvement can be done by utilising  priority-queue (with priority function being message-ref -> message -> `:clock-value` function) instead of plain sequence for values in `:message-groups` map, but unfortunately unlike clojure, clojurescript doesn't offer such data-structure out of the box.\r\n\r\nAdditional thing which occurred to me was that chat datemarks (eq \"today\") or not \"live\", eq they were recomputed only when something in messages changed, but it could happen that nothing in messages changed, only time passed (with app and some chat being open), and suddenly, as midnight passes \"today\" group becomes \"yesterday\" group :)\r\nIn this PR, the behaviour (re-computation of the datemarks) is hooked into `add-message` function, so it happens each time new message is received or sent, but the right place to hook it into would be some periodic event, like `:sync-state`.\r\n\r\n### Testing notes (optional):\r\nComplete regression testing with focus on performance with larger number of messages in chat (ideally chat history spanning multiple days) + making sure that chat messages are still being rendered correctly\r\n\r\nstatus: ready\r\n","created_at":"2018-05-11T12:17:38Z","updated_at":"2018-05-11T14:35:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b8ddb30f34365871403c7327ab05e2bb5b224d2d","assignee":{"login":"janherich","id":1910609,"avatar_url":"https://avatars1.githubusercontent.com/u/1910609?v=4","gravatar_id":"","url":"https://api.github.com/users/janherich","html_url":"https://github.com/janherich","followers_url":"https://api.github.com/users/janherich/followers","following_url":"https://api.github.com/users/janherich/following{/other_user}","gists_url":"https://api.github.com/users/janherich/gists{/gist_id}","starred_url":"https://api.github.com/users/janherich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janherich/subscriptions","organizations_url":"https://api.github.com/users/janherich/orgs","repos_url":"https://api.github.com/users/janherich/repos","events_url":"https://api.github.com/users/janherich/events{/privacy}","received_events_url":"https://api.github.com/users/janherich/received_events","type":"User","site_admin":false},"assignees":[{"login":"janherich","id":1910609,"avatar_url":"https://avatars1.githubusercontent.com/u/1910609?v=4","gravatar_id":"","url":"https://api.github.com/users/janherich","html_url":"https://github.com/janherich","followers_url":"https://api.github.com/users/janherich/followers","following_url":"https://api.github.com/users/janherich/following{/other_user}","gists_url":"https://api.github.com/users/janherich/gists{/gist_id}","starred_url":"https://api.github.com/users/janherich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janherich/subscriptions","organizations_url":"https://api.github.com/users/janherich/orgs","repos_url":"https://api.github.com/users/janherich/repos","events_url":"https://api.github.com/users/janherich/events{/privacy}","received_events_url":"https://api.github.com/users/janherich/received_events","type":"User","site_admin":false}],"requested_reviewers":[{"login":"cammellos","id":1017008,"avatar_url":"https://avatars0.githubusercontent.com/u/1017008?v=4","gravatar_id":"","url":"https://api.github.com/users/cammellos","html_url":"https://github.com/cammellos","followers_url":"https://api.github.com/users/cammellos/followers","following_url":"https://api.github.com/users/cammellos/following{/other_user}","gists_url":"https://api.github.com/users/cammellos/gists{/gist_id}","starred_url":"https://api.github.com/users/cammellos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cammellos/subscriptions","organizations_url":"https://api.github.com/users/cammellos/orgs","repos_url":"https://api.github.com/users/cammellos/repos","events_url":"https://api.github.com/users/cammellos/events{/privacy}","received_events_url":"https://api.github.com/users/cammellos/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":807362947,"url":"https://api.github.com/repos/status-im/status-react/labels/clojure","name":"clojure","color":"244ae2","default":false},{"id":783781483,"url":"https://api.github.com/repos/status-im/status-react/labels/performance","name":"performance","color":"dd3bd8","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/status-im/status-react/pulls/4221/commits","review_comments_url":"https://api.github.com/repos/status-im/status-react/pulls/4221/comments","review_comment_url":"https://api.github.com/repos/status-im/status-react/pulls/comments{/number}","comments_url":"https://api.github.com/repos/status-im/status-react/issues/4221/comments","statuses_url":"https://api.github.com/repos/status-im/status-react/statuses/53eb050ed4ffd71eb65ce3eb85cefb3481970125","head":{"label":"status-im:feature/fast-message-grouping","ref":"feature/fast-message-grouping","sha":"53eb050ed4ffd71eb65ce3eb85cefb3481970125","user":{"login":"status-im","id":11767950,"avatar_url":"https://avatars1.githubusercontent.com/u/11767950?v=4","gravatar_id":"","url":"https://api.github.com/users/status-im","html_url":"https://github.com/status-im","followers_url":"https://api.github.com/users/status-im/followers","following_url":"https://api.github.com/users/status-im/following{/other_user}","gists_url":"https://api.github.com/users/status-im/gists{/gist_id}","starred_url":"https://api.github.com/users/status-im/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/status-im/subscriptions","organizations_url":"https://api.github.com/users/status-im/orgs","repos_url":"https://api.github.com/users/status-im/repos","events_url":"https://api.github.com/users/status-im/events{/privacy}","received_events_url":"https://api.github.com/users/status-im/received_events","type":"Organization","site_admin":false},"repo":{"id":52308441,"name":"status-react","full_name":"status-im/status-react","owner":{"login":"status-im","id":11767950,"avatar_url":"https://avatars1.githubusercontent.com/u/11767950?v=4","gravatar_id":"","url":"https://api.github.com/users/status-im","html_url":"https://github.com/status-im","followers_url":"https://api.github.com/users/status-im/followers","following_url":"https://api.github.com/users/status-im/following{/other_user}","gists_url":"https://api.github.com/users/status-im/gists{/gist_id}","starred_url":"https://api.github.com/users/status-im/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/status-im/subscriptions","organizations_url":"https://api.github.com/users/status-im/orgs","repos_url":"https://api.github.com/users/status-im/repos","events_url":"https://api.github.com/users/status-im/events{/privacy}","received_events_url":"https://api.github.com/users/status-im/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/status-im/status-react","description":"a free (libre) open source, mobile OS for Ethereum","fork":false,"url":"https://api.github.com/repos/status-im/status-react","forks_url":"https://api.github.com/repos/status-im/status-react/forks","keys_url":"https://api.github.com/repos/status-im/status-react/keys{/key_id}","collaborators_url":"https://api.github.com/repos/status-im/status-react/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/status-im/status-react/teams","hooks_url":"https://api.github.com/repos/status-im/status-react/hooks","issue_events_url":"https://api.github.com/repos/status-im/status-react/issues/events{/number}","events_url":"https://api.github.com/repos/status-im/status-react/events","assignees_url":"https://api.github.com/repos/status-im/status-react/assignees{/user}","branches_url":"https://api.github.com/repos/status-im/status-react/branches{/branch}","tags_url":"https://api.github.com/repos/status-im/status-react/tags","blobs_url":"https://api.github.com/repos/status-im/status-react/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/status-im/status-react/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/status-im/status-react/git/refs{/sha}","trees_url":"https://api.github.com/repos/status-im/status-react/git/trees{/sha}","statuses_url":"https://api.github.com/repos/status-im/status-react/statuses/{sha}","languages_url":"https://api.github.com/repos/status-im/status-react/languages","stargazers_url":"https://api.github.com/repos/status-im/status-react/stargazers","contributors_url":"https://api.github.com/repos/status-im/status-react/contributors","subscribers_url":"https://api.github.com/repos/status-im/status-react/subscribers","subscription_url":"https://api.github.com/repos/status-im/status-react/subscription","commits_url":"https://api.github.com/repos/status-im/status-react/commits{/sha}","git_commits_url":"https://api.github.com/repos/status-im/status-react/git/commits{/sha}","comments_url":"https://api.github.com/repos/status-im/status-react/comments{/number}","issue_comment_url":"https://api.github.com/repos/status-im/status-react/issues/comments{/number}","contents_url":"https://api.github.com/repos/status-im/status-react/contents/{+path}","compare_url":"https://api.github.com/repos/status-im/status-react/compare/{base}...{head}","merges_url":"https://api.github.com/repos/status-im/status-react/merges","archive_url":"https://api.github.com/repos/status-im/status-react/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/status-im/status-react/downloads","issues_url":"https://api.github.com/repos/status-im/status-react/issues{/number}","pulls_url":"https://api.github.com/repos/status-im/status-react/pulls{/number}","milestones_url":"https://api.github.com/repos/status-im/status-react/milestones{/number}","notifications_url":"https://api.github.com/repos/status-im/status-react/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/status-im/status-react/labels{/name}","releases_url":"https://api.github.com/repos/status-im/status-react/releases{/id}","deployments_url":"https://api.github.com/repos/status-im/status-react/deployments","created_at":"2016-02-22T21:48:48Z","updated_at":"2018-05-11T13:42:47Z","pushed_at":"2018-05-11T14:16:28Z","git_url":"git://github.com/status-im/status-react.git","ssh_url":"git@github.com:status-im/status-react.git","clone_url":"https://github.com/status-im/status-react.git","svn_url":"https://github.com/status-im/status-react","homepage":"https://status.im","size":54242,"stargazers_count":2186,"watchers_count":2186,"language":"Clojure","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":532,"mirror_url":null,"archived":false,"open_issues_count":579,"license":{"key":"mpl-2.0","name":"Mozilla Public License 2.0","spdx_id":"MPL-2.0","url":"https://api.github.com/licenses/mpl-2.0"},"forks":532,"open_issues":579,"watchers":2186,"default_branch":"develop"}},"base":{"label":"status-im:develop","ref":"develop","sha":"89632bea1bb9a7413fb8854abec49205f9a68b5b","user":{"login":"status-im","id":11767950,"avatar_url":"https://avatars1.githubusercontent.com/u/11767950?v=4","gravatar_id":"","url":"https://api.github.com/users/status-im","html_url":"https://github.com/status-im","followers_url":"https://api.github.com/users/status-im/followers","following_url":"https://api.github.com/users/status-im/following{/other_user}","gists_url":"https://api.github.com/users/status-im/gists{/gist_id}","starred_url":"https://api.github.com/users/status-im/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/status-im/subscriptions","organizations_url":"https://api.github.com/users/status-im/orgs","repos_url":"https://api.github.com/users/status-im/repos","events_url":"https://api.github.com/users/status-im/events{/privacy}","received_events_url":"https://api.github.com/users/status-im/received_events","type":"Organization","site_admin":false},"repo":{"id":52308441,"name":"status-react","full_name":"status-im/status-react","owner":{"login":"status-im","id":11767950,"avatar_url":"https://avatars1.githubusercontent.com/u/11767950?v=4","gravatar_id":"","url":"https://api.github.com/users/status-im","html_url":"https://github.com/status-im","followers_url":"https://api.github.com/users/status-im/followers","following_url":"https://api.github.com/users/status-im/following{/other_user}","gists_url":"https://api.github.com/users/status-im/gists{/gist_id}","starred_url":"https://api.github.com/users/status-im/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/status-im/subscriptions","organizations_url":"https://api.github.com/users/status-im/orgs","repos_url":"https://api.github.com/users/status-im/repos","events_url":"https://api.github.com/users/status-im/events{/privacy}","received_events_url":"https://api.github.com/users/status-im/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/status-im/status-react","description":"a free (libre) open source, mobile OS for Ethereum","fork":false,"url":"https://api.github.com/repos/status-im/status-react","forks_url":"https://api.github.com/repos/status-im/status-react/forks","keys_url":"https://api.github.com/repos/status-im/status-react/keys{/key_id}","collaborators_url":"https://api.github.com/repos/status-im/status-react/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/status-im/status-react/teams","hooks_url":"https://api.github.com/repos/status-im/status-react/hooks","issue_events_url":"https://api.github.com/repos/status-im/status-react/issues/events{/number}","events_url":"https://api.github.com/repos/status-im/status-react/events","assignees_url":"https://api.github.com/repos/status-im/status-react/assignees{/user}","branches_url":"https://api.github.com/repos/status-im/status-react/branches{/branch}","tags_url":"https://api.github.com/repos/status-im/status-react/tags","blobs_url":"https://api.github.com/repos/status-im/status-react/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/status-im/status-react/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/status-im/status-react/git/refs{/sha}","trees_url":"https://api.github.com/repos/status-im/status-react/git/trees{/sha}","statuses_url":"https://api.github.com/repos/status-im/status-react/statuses/{sha}","languages_url":"https://api.github.com/repos/status-im/status-react/languages","stargazers_url":"https://api.github.com/repos/status-im/status-react/stargazers","contributors_url":"https://api.github.com/repos/status-im/status-react/contributors","subscribers_url":"https://api.github.com/repos/status-im/status-react/subscribers","subscription_url":"https://api.github.com/repos/status-im/status-react/subscription","commits_url":"https://api.github.com/repos/status-im/status-react/commits{/sha}","git_commits_url":"https://api.github.com/repos/status-im/status-react/git/commits{/sha}","comments_url":"https://api.github.com/repos/status-im/status-react/comments{/number}","issue_comment_url":"https://api.github.com/repos/status-im/status-react/issues/comments{/number}","contents_url":"https://api.github.com/repos/status-im/status-react/contents/{+path}","compare_url":"https://api.github.com/repos/status-im/status-react/compare/{base}...{head}","merges_url":"https://api.github.com/repos/status-im/status-react/merges","archive_url":"https://api.github.com/repos/status-im/status-react/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/status-im/status-react/downloads","issues_url":"https://api.github.com/repos/status-im/status-react/issues{/number}","pulls_url":"https://api.github.com/repos/status-im/status-react/pulls{/number}","milestones_url":"https://api.github.com/repos/status-im/status-react/milestones{/number}","notifications_url":"https://api.github.com/repos/status-im/status-react/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/status-im/status-react/labels{/name}","releases_url":"https://api.github.com/repos/status-im/status-react/releases{/id}","deployments_url":"https://api.github.com/repos/status-im/status-react/deployments","created_at":"2016-02-22T21:48:48Z","updated_at":"2018-05-11T13:42:47Z","pushed_at":"2018-05-11T14:16:28Z","git_url":"git://github.com/status-im/status-react.git","ssh_url":"git@github.com:status-im/status-react.git","clone_url":"https://github.com/status-im/status-react.git","svn_url":"https://github.com/status-im/status-react","homepage":"https://status.im","size":54242,"stargazers_count":2186,"watchers_count":2186,"language":"Clojure","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":532,"mirror_url":null,"archived":false,"open_issues_count":579,"license":{"key":"mpl-2.0","name":"Mozilla Public License 2.0","spdx_id":"MPL-2.0","url":"https://api.github.com/licenses/mpl-2.0"},"forks":532,"open_issues":579,"watchers":2186,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/status-im/status-react/pulls/4221"},"html":{"href":"https://github.com/status-im/status-react/pull/4221"},"issue":{"href":"https://api.github.com/repos/status-im/status-react/issues/4221"},"comments":{"href":"https://api.github.com/repos/status-im/status-react/issues/4221/comments"},"review_comments":{"href":"https://api.github.com/repos/status-im/status-react/pulls/4221/comments"},"review_comment":{"href":"https://api.github.com/repos/status-im/status-react/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/status-im/status-react/pulls/4221/commits"},"statuses":{"href":"https://api.github.com/repos/status-im/status-react/statuses/53eb050ed4ffd71eb65ce3eb85cefb3481970125"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"rasom"}}