{"action":"created","issue":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/3583","repository_url":"https://api.github.com/repos/sourcegraph/sourcegraph","labels_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/3583/labels{/name}","comments_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/3583/comments","events_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/3583/events","html_url":"https://github.com/sourcegraph/sourcegraph/issues/3583","id":436958837,"node_id":"MDU6SXNzdWU0MzY5NTg4Mzc=","number":3583,"title":"Implement Redis Streams for usage statistics and audit logging","user":{"login":"dadlerj","id":5589410,"node_id":"MDQ6VXNlcjU1ODk0MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/5589410?v=4","gravatar_id":"","url":"https://api.github.com/users/dadlerj","html_url":"https://github.com/dadlerj","followers_url":"https://api.github.com/users/dadlerj/followers","following_url":"https://api.github.com/users/dadlerj/following{/other_user}","gists_url":"https://api.github.com/users/dadlerj/gists{/gist_id}","starred_url":"https://api.github.com/users/dadlerj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadlerj/subscriptions","organizations_url":"https://api.github.com/users/dadlerj/orgs","repos_url":"https://api.github.com/users/dadlerj/repos","events_url":"https://api.github.com/users/dadlerj/events{/privacy}","received_events_url":"https://api.github.com/users/dadlerj/received_events","type":"User","site_admin":false},"labels":[{"id":1125572665,"node_id":"MDU6TGFiZWwxMTI1NTcyNjY1","url":"https://api.github.com/repos/sourcegraph/sourcegraph/labels/roadmap","name":"roadmap","color":"000000","default":false},{"id":1220107372,"node_id":"MDU6TGFiZWwxMjIwMTA3Mzcy","url":"https://api.github.com/repos/sourcegraph/sourcegraph/labels/site-admin","name":"site-admin","color":"c3f9a9","default":false}],"state":"open","locked":false,"assignee":{"login":"unknwon","id":2946214,"node_id":"MDQ6VXNlcjI5NDYyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/2946214?v=4","gravatar_id":"","url":"https://api.github.com/users/unknwon","html_url":"https://github.com/unknwon","followers_url":"https://api.github.com/users/unknwon/followers","following_url":"https://api.github.com/users/unknwon/following{/other_user}","gists_url":"https://api.github.com/users/unknwon/gists{/gist_id}","starred_url":"https://api.github.com/users/unknwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unknwon/subscriptions","organizations_url":"https://api.github.com/users/unknwon/orgs","repos_url":"https://api.github.com/users/unknwon/repos","events_url":"https://api.github.com/users/unknwon/events{/privacy}","received_events_url":"https://api.github.com/users/unknwon/received_events","type":"User","site_admin":false},"assignees":[{"login":"unknwon","id":2946214,"node_id":"MDQ6VXNlcjI5NDYyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/2946214?v=4","gravatar_id":"","url":"https://api.github.com/users/unknwon","html_url":"https://github.com/unknwon","followers_url":"https://api.github.com/users/unknwon/followers","following_url":"https://api.github.com/users/unknwon/following{/other_user}","gists_url":"https://api.github.com/users/unknwon/gists{/gist_id}","starred_url":"https://api.github.com/users/unknwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unknwon/subscriptions","organizations_url":"https://api.github.com/users/unknwon/orgs","repos_url":"https://api.github.com/users/unknwon/repos","events_url":"https://api.github.com/users/unknwon/events{/privacy}","received_events_url":"https://api.github.com/users/unknwon/received_events","type":"User","site_admin":false},{"login":"dadlerj","id":5589410,"node_id":"MDQ6VXNlcjU1ODk0MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/5589410?v=4","gravatar_id":"","url":"https://api.github.com/users/dadlerj","html_url":"https://github.com/dadlerj","followers_url":"https://api.github.com/users/dadlerj/followers","following_url":"https://api.github.com/users/dadlerj/following{/other_user}","gists_url":"https://api.github.com/users/dadlerj/gists{/gist_id}","starred_url":"https://api.github.com/users/dadlerj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadlerj/subscriptions","organizations_url":"https://api.github.com/users/dadlerj/orgs","repos_url":"https://api.github.com/users/dadlerj/repos","events_url":"https://api.github.com/users/dadlerj/events{/privacy}","received_events_url":"https://api.github.com/users/dadlerj/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/milestones/25","html_url":"https://github.com/sourcegraph/sourcegraph/milestone/25","labels_url":"https://api.github.com/repos/sourcegraph/sourcegraph/milestones/25/labels","id":4456164,"node_id":"MDk6TWlsZXN0b25lNDQ1NjE2NA==","number":25,"title":"3.8","description":"","creator":{"login":"slimsag","id":3173176,"node_id":"MDQ6VXNlcjMxNzMxNzY=","avatar_url":"https://avatars2.githubusercontent.com/u/3173176?v=4","gravatar_id":"","url":"https://api.github.com/users/slimsag","html_url":"https://github.com/slimsag","followers_url":"https://api.github.com/users/slimsag/followers","following_url":"https://api.github.com/users/slimsag/following{/other_user}","gists_url":"https://api.github.com/users/slimsag/gists{/gist_id}","starred_url":"https://api.github.com/users/slimsag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slimsag/subscriptions","organizations_url":"https://api.github.com/users/slimsag/orgs","repos_url":"https://api.github.com/users/slimsag/repos","events_url":"https://api.github.com/users/slimsag/events{/privacy}","received_events_url":"https://api.github.com/users/slimsag/received_events","type":"User","site_admin":false},"open_issues":27,"closed_issues":2,"state":"open","created_at":"2019-07-01T20:36:40Z","updated_at":"2019-08-19T21:46:54Z","due_on":"2019-09-20T07:00:00Z","closed_at":null},"comments":18,"created_at":"2019-04-25T00:18:31Z","updated_at":"2019-08-19T23:00:38Z","closed_at":null,"author_association":"MEMBER","body":"**Background**\r\nWe currently provide high-level, aggregate usage statistics to site admins and users (such as a count of daily, weekly, and monthly active users) as well as some individual aggregate metrics (such as a flag for a user's latest activity).\r\n\r\nGoing forward, we'd like to be able to provide more context to admins around historical usage, both for tracking product usage (e.g. how many times a user logged in in a given week, rather than just capturing the \"latest\" time) and for audit logging (e.g. who most recently made updates to the site config, etc.).\r\n\r\n**Approach**\r\nWe won't be able to enumerate every possible metric a priori, implying that an instance would need to be able to record arbitrary event logs. Redis Streams would be an ideal way to accomplish this (per discussions with @tsenart ). \r\n\r\n**Implementation**\r\n\r\n(HIGHLY WIP)\r\n\r\n- [ ] Update to Redis 5.0+\r\n  - [ ] Update all relevant Go libraries\r\n- [ ] Write migrations (as needed for upgrades from previous Sourcegraph versions)\r\n- [ ] Refactor `usagestats` package \r\n  - [ ] Remove most user-level metrics (such as the new stage metrics)\r\n  - [ ] Begin logging whitelisted user actions via Redis Streams (authenticated users only?)\r\n  - [ ] Replace event handlers and usage stats lookups to aggregate Stream data (versus incrementing/accessing user-level data)\r\n  - [ ] If necessary on 25,000 user test instance, implement caching\r\n- [ ] Add audit logging\r\n  - [ ] Add new audit log site-admin page that aggregates account, authentication, and security-related events (specific list TBD)\r\n  - [ ] Add GraphQL layer\r\n  - [ ] Only store 7 days of history on free and OSS instances, 30 days on Enterprise, and unlimited on Unlimited\r\n- [ ] [stretch] Provide option to use external database for Unlimited instances"},"comment":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/comments/522785823","html_url":"https://github.com/sourcegraph/sourcegraph/issues/3583#issuecomment-522785823","issue_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/3583","id":522785823,"node_id":"MDEyOklzc3VlQ29tbWVudDUyMjc4NTgyMw==","user":{"login":"nicksnyder","id":754768,"node_id":"MDQ6VXNlcjc1NDc2OA==","avatar_url":"https://avatars1.githubusercontent.com/u/754768?v=4","gravatar_id":"","url":"https://api.github.com/users/nicksnyder","html_url":"https://github.com/nicksnyder","followers_url":"https://api.github.com/users/nicksnyder/followers","following_url":"https://api.github.com/users/nicksnyder/following{/other_user}","gists_url":"https://api.github.com/users/nicksnyder/gists{/gist_id}","starred_url":"https://api.github.com/users/nicksnyder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicksnyder/subscriptions","organizations_url":"https://api.github.com/users/nicksnyder/orgs","repos_url":"https://api.github.com/users/nicksnyder/repos","events_url":"https://api.github.com/users/nicksnyder/events{/privacy}","received_events_url":"https://api.github.com/users/nicksnyder/received_events","type":"User","site_admin":false},"created_at":"2019-08-19T23:00:38Z","updated_at":"2019-08-19T23:00:38Z","author_association":"MEMBER","body":"Sounds good. To be clear, I didn't mean to imply that we would skip testing/validation for PostgreSQL, just that we might be able to decide to skip testing other approaches like Clickhouse if we can validate that PostgreSQL will meet our needs."}}
{"actor":{"display_login":"nicksnyder"}}