{"action":"created","comment":{"url":"https://api.github.com/repos/matrix-org/dendrite/pulls/comments/195031766","pull_request_review_id":128315454,"id":195031766,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NTAzMTc2Ng==","diff_hunk":"@@ -0,0 +1,247 @@\n+// Copyright 2018 New Vector Ltd\n+//\n+// Licensed under the Apache License, Version 2.0 (the \"License\");\n+// you may not use this file except in compliance with the License.\n+// You may obtain a copy of the License at\n+//\n+//     http://www.apache.org/licenses/LICENSE-2.0\n+//\n+// Unless required by applicable law or agreed to in writing, software\n+// distributed under the License is distributed on an \"AS IS\" BASIS,\n+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+// See the License for the specific language governing permissions and\n+// limitations under the License.\n+\n+package storage\n+\n+import (\n+\t\"context\"\n+\t\"database/sql\"\n+\t\"fmt\"\n+\t\"time\"\n+\n+\t\"github.com/matrix-org/gomatrixserverlib\"\n+\tlog \"github.com/sirupsen/logrus\"\n+)\n+\n+const appserviceEventsSchema = `\n+-- Stores events to be sent to application services\n+CREATE TABLE IF NOT EXISTS appservice_events (\n+\t-- An auto-incrementing id unique to each event in the table\n+\tid BIGSERIAL NOT NULL PRIMARY KEY,\n+\t-- The ID of the application service the event will be sent to\n+\tas_id TEXT NOT NULL,\n+\t-- The ID of the event\n+\tevent_id TEXT NOT NULL,\n+\t-- Unix seconds that the event was sent at from the originating server\n+\torigin_server_ts BIGINT NOT NULL,\n+\t-- The ID of the room that the event was sent in\n+\troom_id TEXT NOT NULL,\n+\t-- The type of the event (e.g. m.text)\n+\ttype TEXT NOT NULL,\n+\t-- The ID of the user that sent the event\n+\tsender TEXT NOT NULL,\n+\t-- The JSON representation of the event's content. Text to avoid db JSON parsing\n+\tevent_content TEXT,\n+\t-- The ID of the transaction that this event is a part of\n+\ttxn_id BIGINT NOT NULL\n+);\n+\n+CREATE INDEX IF NOT EXISTS appservice_events_as_id ON appservice_events(as_id);\n+`\n+\n+const selectEventsByApplicationServiceIDSQL = \"\" +\n+\t\"SELECT id, event_id, origin_server_ts, room_id, type, sender, event_content, txn_id \" +\n+\t\"FROM appservice_events WHERE as_id = $1 ORDER BY txn_id DESC LIMIT $2\"","path":"src/github.com/matrix-org/dendrite/appservice/storage/appservice_events_table.go","position":55,"original_position":55,"commit_id":"af841cdcc54abe6985a8fffd5a28d10b71097b36","original_commit_id":"af841cdcc54abe6985a8fffd5a28d10b71097b36","user":{"login":"anoadragon453","id":1342360,"node_id":"MDQ6VXNlcjEzNDIzNjA=","avatar_url":"https://avatars1.githubusercontent.com/u/1342360?v=4","gravatar_id":"","url":"https://api.github.com/users/anoadragon453","html_url":"https://github.com/anoadragon453","followers_url":"https://api.github.com/users/anoadragon453/followers","following_url":"https://api.github.com/users/anoadragon453/following{/other_user}","gists_url":"https://api.github.com/users/anoadragon453/gists{/gist_id}","starred_url":"https://api.github.com/users/anoadragon453/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anoadragon453/subscriptions","organizations_url":"https://api.github.com/users/anoadragon453/orgs","repos_url":"https://api.github.com/users/anoadragon453/repos","events_url":"https://api.github.com/users/anoadragon453/events{/privacy}","received_events_url":"https://api.github.com/users/anoadragon453/received_events","type":"User","site_admin":false},"body":"Yes, that is supported and will prevent events from going out of order :)","created_at":"2018-06-13T10:27:34Z","updated_at":"2018-06-13T10:27:34Z","html_url":"https://github.com/matrix-org/dendrite/pull/477#discussion_r195031766","pull_request_url":"https://api.github.com/repos/matrix-org/dendrite/pulls/477","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/comments/195031766"},"html":{"href":"https://github.com/matrix-org/dendrite/pull/477#discussion_r195031766"},"pull_request":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/477"}},"in_reply_to_id":194834540},"pull_request":{"url":"https://api.github.com/repos/matrix-org/dendrite/pulls/477","id":192089057,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyMDg5MDU3","html_url":"https://github.com/matrix-org/dendrite/pull/477","diff_url":"https://github.com/matrix-org/dendrite/pull/477.diff","patch_url":"https://github.com/matrix-org/dendrite/pull/477.patch","issue_url":"https://api.github.com/repos/matrix-org/dendrite/issues/477","number":477,"state":"open","locked":false,"title":"Send Application Service Events","user":{"login":"anoadragon453","id":1342360,"node_id":"MDQ6VXNlcjEzNDIzNjA=","avatar_url":"https://avatars1.githubusercontent.com/u/1342360?v=4","gravatar_id":"","url":"https://api.github.com/users/anoadragon453","html_url":"https://github.com/anoadragon453","followers_url":"https://api.github.com/users/anoadragon453/followers","following_url":"https://api.github.com/users/anoadragon453/following{/other_user}","gists_url":"https://api.github.com/users/anoadragon453/gists{/gist_id}","starred_url":"https://api.github.com/users/anoadragon453/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anoadragon453/subscriptions","organizations_url":"https://api.github.com/users/anoadragon453/orgs","repos_url":"https://api.github.com/users/anoadragon453/repos","events_url":"https://api.github.com/users/anoadragon453/events{/privacy}","received_events_url":"https://api.github.com/users/anoadragon453/received_events","type":"User","site_admin":false},"body":"Adds the database backend for the application service component, stores filtered events for goroutines for each app service to grab them and send them off to the transactions API.\r\n\r\nThe database consists of two tables, one for events, along with the app service they're destined for and some metadata. The second table is a simple relation between an app service and the last transaction ID that was used to send data to it.\r\n\r\nThe full bullet-point list of changes is below:\r\n\r\n* Modify INSTALL.md and dendrite-config.yaml for the new appservice database.\r\n* Correct all instances of casing on 'application service' to align with\r\nspec.\r\n* Store incoming events that an app service is interested in in the\r\ndatabase to be later read by transaction workers.\r\n* Retrieve these events from transaction workers, one per AS.\r\n* Minimal transaction ID data is stored as well to recover after\r\nserver failure.\r\n* Send events to AS and exponentially backoff on failure.\r\n\r\n\r\nFixes: #450 ","created_at":"2018-06-01T16:52:50Z","updated_at":"2018-06-13T10:27:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2ce9c519dcf903097d0140f047fa98d42ff13a5b","assignee":{"login":"anoadragon453","id":1342360,"node_id":"MDQ6VXNlcjEzNDIzNjA=","avatar_url":"https://avatars1.githubusercontent.com/u/1342360?v=4","gravatar_id":"","url":"https://api.github.com/users/anoadragon453","html_url":"https://github.com/anoadragon453","followers_url":"https://api.github.com/users/anoadragon453/followers","following_url":"https://api.github.com/users/anoadragon453/following{/other_user}","gists_url":"https://api.github.com/users/anoadragon453/gists{/gist_id}","starred_url":"https://api.github.com/users/anoadragon453/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anoadragon453/subscriptions","organizations_url":"https://api.github.com/users/anoadragon453/orgs","repos_url":"https://api.github.com/users/anoadragon453/repos","events_url":"https://api.github.com/users/anoadragon453/events{/privacy}","received_events_url":"https://api.github.com/users/anoadragon453/received_events","type":"User","site_admin":false},"assignees":[{"login":"anoadragon453","id":1342360,"node_id":"MDQ6VXNlcjEzNDIzNjA=","avatar_url":"https://avatars1.githubusercontent.com/u/1342360?v=4","gravatar_id":"","url":"https://api.github.com/users/anoadragon453","html_url":"https://github.com/anoadragon453","followers_url":"https://api.github.com/users/anoadragon453/followers","following_url":"https://api.github.com/users/anoadragon453/following{/other_user}","gists_url":"https://api.github.com/users/anoadragon453/gists{/gist_id}","starred_url":"https://api.github.com/users/anoadragon453/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anoadragon453/subscriptions","organizations_url":"https://api.github.com/users/anoadragon453/orgs","repos_url":"https://api.github.com/users/anoadragon453/repos","events_url":"https://api.github.com/users/anoadragon453/events{/privacy}","received_events_url":"https://api.github.com/users/anoadragon453/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/matrix-org/dendrite/pulls/477/commits","review_comments_url":"https://api.github.com/repos/matrix-org/dendrite/pulls/477/comments","review_comment_url":"https://api.github.com/repos/matrix-org/dendrite/pulls/comments{/number}","comments_url":"https://api.github.com/repos/matrix-org/dendrite/issues/477/comments","statuses_url":"https://api.github.com/repos/matrix-org/dendrite/statuses/af841cdcc54abe6985a8fffd5a28d10b71097b36","head":{"label":"matrix-org:anoa/as_forward_events","ref":"anoa/as_forward_events","sha":"af841cdcc54abe6985a8fffd5a28d10b71097b36","user":{"login":"matrix-org","id":8418310,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg0MTgzMTA=","avatar_url":"https://avatars2.githubusercontent.com/u/8418310?v=4","gravatar_id":"","url":"https://api.github.com/users/matrix-org","html_url":"https://github.com/matrix-org","followers_url":"https://api.github.com/users/matrix-org/followers","following_url":"https://api.github.com/users/matrix-org/following{/other_user}","gists_url":"https://api.github.com/users/matrix-org/gists{/gist_id}","starred_url":"https://api.github.com/users/matrix-org/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matrix-org/subscriptions","organizations_url":"https://api.github.com/users/matrix-org/orgs","repos_url":"https://api.github.com/users/matrix-org/repos","events_url":"https://api.github.com/users/matrix-org/events{/privacy}","received_events_url":"https://api.github.com/users/matrix-org/received_events","type":"Organization","site_admin":false},"repo":{"id":79585104,"node_id":"MDEwOlJlcG9zaXRvcnk3OTU4NTEwNA==","name":"dendrite","full_name":"matrix-org/dendrite","owner":{"login":"matrix-org","id":8418310,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg0MTgzMTA=","avatar_url":"https://avatars2.githubusercontent.com/u/8418310?v=4","gravatar_id":"","url":"https://api.github.com/users/matrix-org","html_url":"https://github.com/matrix-org","followers_url":"https://api.github.com/users/matrix-org/followers","following_url":"https://api.github.com/users/matrix-org/following{/other_user}","gists_url":"https://api.github.com/users/matrix-org/gists{/gist_id}","starred_url":"https://api.github.com/users/matrix-org/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matrix-org/subscriptions","organizations_url":"https://api.github.com/users/matrix-org/orgs","repos_url":"https://api.github.com/users/matrix-org/repos","events_url":"https://api.github.com/users/matrix-org/events{/privacy}","received_events_url":"https://api.github.com/users/matrix-org/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/matrix-org/dendrite","description":"Dendrite will be a matrix homeserver written in go","fork":false,"url":"https://api.github.com/repos/matrix-org/dendrite","forks_url":"https://api.github.com/repos/matrix-org/dendrite/forks","keys_url":"https://api.github.com/repos/matrix-org/dendrite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/matrix-org/dendrite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/matrix-org/dendrite/teams","hooks_url":"https://api.github.com/repos/matrix-org/dendrite/hooks","issue_events_url":"https://api.github.com/repos/matrix-org/dendrite/issues/events{/number}","events_url":"https://api.github.com/repos/matrix-org/dendrite/events","assignees_url":"https://api.github.com/repos/matrix-org/dendrite/assignees{/user}","branches_url":"https://api.github.com/repos/matrix-org/dendrite/branches{/branch}","tags_url":"https://api.github.com/repos/matrix-org/dendrite/tags","blobs_url":"https://api.github.com/repos/matrix-org/dendrite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/matrix-org/dendrite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/matrix-org/dendrite/git/refs{/sha}","trees_url":"https://api.github.com/repos/matrix-org/dendrite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/matrix-org/dendrite/statuses/{sha}","languages_url":"https://api.github.com/repos/matrix-org/dendrite/languages","stargazers_url":"https://api.github.com/repos/matrix-org/dendrite/stargazers","contributors_url":"https://api.github.com/repos/matrix-org/dendrite/contributors","subscribers_url":"https://api.github.com/repos/matrix-org/dendrite/subscribers","subscription_url":"https://api.github.com/repos/matrix-org/dendrite/subscription","commits_url":"https://api.github.com/repos/matrix-org/dendrite/commits{/sha}","git_commits_url":"https://api.github.com/repos/matrix-org/dendrite/git/commits{/sha}","comments_url":"https://api.github.com/repos/matrix-org/dendrite/comments{/number}","issue_comment_url":"https://api.github.com/repos/matrix-org/dendrite/issues/comments{/number}","contents_url":"https://api.github.com/repos/matrix-org/dendrite/contents/{+path}","compare_url":"https://api.github.com/repos/matrix-org/dendrite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/matrix-org/dendrite/merges","archive_url":"https://api.github.com/repos/matrix-org/dendrite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/matrix-org/dendrite/downloads","issues_url":"https://api.github.com/repos/matrix-org/dendrite/issues{/number}","pulls_url":"https://api.github.com/repos/matrix-org/dendrite/pulls{/number}","milestones_url":"https://api.github.com/repos/matrix-org/dendrite/milestones{/number}","notifications_url":"https://api.github.com/repos/matrix-org/dendrite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/matrix-org/dendrite/labels{/name}","releases_url":"https://api.github.com/repos/matrix-org/dendrite/releases{/id}","deployments_url":"https://api.github.com/repos/matrix-org/dendrite/deployments","created_at":"2017-01-20T18:22:16Z","updated_at":"2018-06-13T02:14:26Z","pushed_at":"2018-06-12T22:33:10Z","git_url":"git://github.com/matrix-org/dendrite.git","ssh_url":"git@github.com:matrix-org/dendrite.git","clone_url":"https://github.com/matrix-org/dendrite.git","svn_url":"https://github.com/matrix-org/dendrite","homepage":"","size":27930,"stargazers_count":318,"watchers_count":318,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":56,"mirror_url":null,"archived":false,"open_issues_count":86,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":56,"open_issues":86,"watchers":318,"default_branch":"master"}},"base":{"label":"matrix-org:master","ref":"master","sha":"78440083dfc8f522e55f11f0fce0f3ea4a103f0b","user":{"login":"matrix-org","id":8418310,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg0MTgzMTA=","avatar_url":"https://avatars2.githubusercontent.com/u/8418310?v=4","gravatar_id":"","url":"https://api.github.com/users/matrix-org","html_url":"https://github.com/matrix-org","followers_url":"https://api.github.com/users/matrix-org/followers","following_url":"https://api.github.com/users/matrix-org/following{/other_user}","gists_url":"https://api.github.com/users/matrix-org/gists{/gist_id}","starred_url":"https://api.github.com/users/matrix-org/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matrix-org/subscriptions","organizations_url":"https://api.github.com/users/matrix-org/orgs","repos_url":"https://api.github.com/users/matrix-org/repos","events_url":"https://api.github.com/users/matrix-org/events{/privacy}","received_events_url":"https://api.github.com/users/matrix-org/received_events","type":"Organization","site_admin":false},"repo":{"id":79585104,"node_id":"MDEwOlJlcG9zaXRvcnk3OTU4NTEwNA==","name":"dendrite","full_name":"matrix-org/dendrite","owner":{"login":"matrix-org","id":8418310,"node_id":"MDEyOk9yZ2FuaXphdGlvbjg0MTgzMTA=","avatar_url":"https://avatars2.githubusercontent.com/u/8418310?v=4","gravatar_id":"","url":"https://api.github.com/users/matrix-org","html_url":"https://github.com/matrix-org","followers_url":"https://api.github.com/users/matrix-org/followers","following_url":"https://api.github.com/users/matrix-org/following{/other_user}","gists_url":"https://api.github.com/users/matrix-org/gists{/gist_id}","starred_url":"https://api.github.com/users/matrix-org/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matrix-org/subscriptions","organizations_url":"https://api.github.com/users/matrix-org/orgs","repos_url":"https://api.github.com/users/matrix-org/repos","events_url":"https://api.github.com/users/matrix-org/events{/privacy}","received_events_url":"https://api.github.com/users/matrix-org/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/matrix-org/dendrite","description":"Dendrite will be a matrix homeserver written in go","fork":false,"url":"https://api.github.com/repos/matrix-org/dendrite","forks_url":"https://api.github.com/repos/matrix-org/dendrite/forks","keys_url":"https://api.github.com/repos/matrix-org/dendrite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/matrix-org/dendrite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/matrix-org/dendrite/teams","hooks_url":"https://api.github.com/repos/matrix-org/dendrite/hooks","issue_events_url":"https://api.github.com/repos/matrix-org/dendrite/issues/events{/number}","events_url":"https://api.github.com/repos/matrix-org/dendrite/events","assignees_url":"https://api.github.com/repos/matrix-org/dendrite/assignees{/user}","branches_url":"https://api.github.com/repos/matrix-org/dendrite/branches{/branch}","tags_url":"https://api.github.com/repos/matrix-org/dendrite/tags","blobs_url":"https://api.github.com/repos/matrix-org/dendrite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/matrix-org/dendrite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/matrix-org/dendrite/git/refs{/sha}","trees_url":"https://api.github.com/repos/matrix-org/dendrite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/matrix-org/dendrite/statuses/{sha}","languages_url":"https://api.github.com/repos/matrix-org/dendrite/languages","stargazers_url":"https://api.github.com/repos/matrix-org/dendrite/stargazers","contributors_url":"https://api.github.com/repos/matrix-org/dendrite/contributors","subscribers_url":"https://api.github.com/repos/matrix-org/dendrite/subscribers","subscription_url":"https://api.github.com/repos/matrix-org/dendrite/subscription","commits_url":"https://api.github.com/repos/matrix-org/dendrite/commits{/sha}","git_commits_url":"https://api.github.com/repos/matrix-org/dendrite/git/commits{/sha}","comments_url":"https://api.github.com/repos/matrix-org/dendrite/comments{/number}","issue_comment_url":"https://api.github.com/repos/matrix-org/dendrite/issues/comments{/number}","contents_url":"https://api.github.com/repos/matrix-org/dendrite/contents/{+path}","compare_url":"https://api.github.com/repos/matrix-org/dendrite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/matrix-org/dendrite/merges","archive_url":"https://api.github.com/repos/matrix-org/dendrite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/matrix-org/dendrite/downloads","issues_url":"https://api.github.com/repos/matrix-org/dendrite/issues{/number}","pulls_url":"https://api.github.com/repos/matrix-org/dendrite/pulls{/number}","milestones_url":"https://api.github.com/repos/matrix-org/dendrite/milestones{/number}","notifications_url":"https://api.github.com/repos/matrix-org/dendrite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/matrix-org/dendrite/labels{/name}","releases_url":"https://api.github.com/repos/matrix-org/dendrite/releases{/id}","deployments_url":"https://api.github.com/repos/matrix-org/dendrite/deployments","created_at":"2017-01-20T18:22:16Z","updated_at":"2018-06-13T02:14:26Z","pushed_at":"2018-06-12T22:33:10Z","git_url":"git://github.com/matrix-org/dendrite.git","ssh_url":"git@github.com:matrix-org/dendrite.git","clone_url":"https://github.com/matrix-org/dendrite.git","svn_url":"https://github.com/matrix-org/dendrite","homepage":"","size":27930,"stargazers_count":318,"watchers_count":318,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":56,"mirror_url":null,"archived":false,"open_issues_count":86,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":56,"open_issues":86,"watchers":318,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/477"},"html":{"href":"https://github.com/matrix-org/dendrite/pull/477"},"issue":{"href":"https://api.github.com/repos/matrix-org/dendrite/issues/477"},"comments":{"href":"https://api.github.com/repos/matrix-org/dendrite/issues/477/comments"},"review_comments":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/477/comments"},"review_comment":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/matrix-org/dendrite/pulls/477/commits"},"statuses":{"href":"https://api.github.com/repos/matrix-org/dendrite/statuses/af841cdcc54abe6985a8fffd5a28d10b71097b36"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"anoadragon453"}}