{"action":"created","issue":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/1467","repository_url":"https://api.github.com/repos/sourcegraph/sourcegraph","labels_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/1467/labels{/name}","comments_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/1467/comments","events_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/1467/events","html_url":"https://github.com/sourcegraph/sourcegraph/issues/1467","id":391744804,"node_id":"MDU6SXNzdWUzOTE3NDQ4MDQ=","number":1467,"title":"Proposal: Repository mirroring improvements","user":{"login":"tsenart","id":67471,"node_id":"MDQ6VXNlcjY3NDcx","avatar_url":"https://avatars2.githubusercontent.com/u/67471?v=4","gravatar_id":"","url":"https://api.github.com/users/tsenart","html_url":"https://github.com/tsenart","followers_url":"https://api.github.com/users/tsenart/followers","following_url":"https://api.github.com/users/tsenart/following{/other_user}","gists_url":"https://api.github.com/users/tsenart/gists{/gist_id}","starred_url":"https://api.github.com/users/tsenart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsenart/subscriptions","organizations_url":"https://api.github.com/users/tsenart/orgs","repos_url":"https://api.github.com/users/tsenart/repos","events_url":"https://api.github.com/users/tsenart/events{/privacy}","received_events_url":"https://api.github.com/users/tsenart/received_events","type":"User","site_admin":false},"labels":[{"id":1076682412,"node_id":"MDU6TGFiZWwxMDc2NjgyNDEy","url":"https://api.github.com/repos/sourcegraph/sourcegraph/labels/repo-updater","name":"repo-updater","color":"c621a5","default":false}],"state":"open","locked":false,"assignee":{"login":"nicksnyder","id":754768,"node_id":"MDQ6VXNlcjc1NDc2OA==","avatar_url":"https://avatars1.githubusercontent.com/u/754768?v=4","gravatar_id":"","url":"https://api.github.com/users/nicksnyder","html_url":"https://github.com/nicksnyder","followers_url":"https://api.github.com/users/nicksnyder/followers","following_url":"https://api.github.com/users/nicksnyder/following{/other_user}","gists_url":"https://api.github.com/users/nicksnyder/gists{/gist_id}","starred_url":"https://api.github.com/users/nicksnyder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicksnyder/subscriptions","organizations_url":"https://api.github.com/users/nicksnyder/orgs","repos_url":"https://api.github.com/users/nicksnyder/repos","events_url":"https://api.github.com/users/nicksnyder/events{/privacy}","received_events_url":"https://api.github.com/users/nicksnyder/received_events","type":"User","site_admin":false},"assignees":[{"login":"nicksnyder","id":754768,"node_id":"MDQ6VXNlcjc1NDc2OA==","avatar_url":"https://avatars1.githubusercontent.com/u/754768?v=4","gravatar_id":"","url":"https://api.github.com/users/nicksnyder","html_url":"https://github.com/nicksnyder","followers_url":"https://api.github.com/users/nicksnyder/followers","following_url":"https://api.github.com/users/nicksnyder/following{/other_user}","gists_url":"https://api.github.com/users/nicksnyder/gists{/gist_id}","starred_url":"https://api.github.com/users/nicksnyder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicksnyder/subscriptions","organizations_url":"https://api.github.com/users/nicksnyder/orgs","repos_url":"https://api.github.com/users/nicksnyder/repos","events_url":"https://api.github.com/users/nicksnyder/events{/privacy}","received_events_url":"https://api.github.com/users/nicksnyder/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/milestones/15","html_url":"https://github.com/sourcegraph/sourcegraph/milestone/15","labels_url":"https://api.github.com/repos/sourcegraph/sourcegraph/milestones/15/labels","id":3768216,"node_id":"MDk6TWlsZXN0b25lMzc2ODIxNg==","number":15,"title":"3.1","description":"","creator":{"login":"sqs","id":1976,"node_id":"MDQ6VXNlcjE5NzY=","avatar_url":"https://avatars0.githubusercontent.com/u/1976?v=4","gravatar_id":"","url":"https://api.github.com/users/sqs","html_url":"https://github.com/sqs","followers_url":"https://api.github.com/users/sqs/followers","following_url":"https://api.github.com/users/sqs/following{/other_user}","gists_url":"https://api.github.com/users/sqs/gists{/gist_id}","starred_url":"https://api.github.com/users/sqs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sqs/subscriptions","organizations_url":"https://api.github.com/users/sqs/orgs","repos_url":"https://api.github.com/users/sqs/repos","events_url":"https://api.github.com/users/sqs/events{/privacy}","received_events_url":"https://api.github.com/users/sqs/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":0,"state":"open","created_at":"2018-10-25T10:06:08Z","updated_at":"2018-12-17T23:39:50Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2018-12-17T14:47:28Z","updated_at":"2018-12-18T15:54:34Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"## Background\r\n\r\nAs it's implemented today, repository mirroring isn't meeting user's expectations in a number of scenarios. Whenever admins or users of a configured external service do any of the below listed actions, Sourcegraph should gracefully converge the set of repositories it mirrors to what users expect following those actions, and do so with clear representation in its UI.\r\n\r\nThis is a sub-issue of #914.\r\n\r\n## Proposal\r\n\r\n1. **Removed permissions**: Removing permissions from an access token used to read previously mirrored repositories from an external service **SHOULD** result in those now inaccessible repositories to be soft-deleted from Sourcegraph, with the reason for which they were deleted made clear to the user.\r\n1. **Added permissions**: Adding extended permissions to an access token to read previously inaccessible repositories of an external service **SHOULD** result in those now accessible repositories to be automatically mirrored by Sourcegraph given they match the external service's configuration of which repositories to mirror (e.g via `github.repositoryQuery`)\r\n1. **Renaming repositories**: Renaming a mirrored repository in an external service **SHOULD** result in a graceful repository rename operation in Sourcegraph (i.e. all data is preserved, only the name changes).\r\n1. **Deleting repositories**: Deleting a mirrored repository in an external service **SHOULD** result in the repository being soft-deleted via a `deleted_at` timestamp in a database column and associated application logic. Soft-deleted repositories can be restored gracefully by clearing out their `deleted_at` column. Associated data like code discussions is preserved and restorable. Cached or derived data like git-server storage and search indexes is not preserved because it's cheap to re-fetch / re-compute.\r\n1. **Modifying repositories**: Modifying a mirrored repository in any way that it no longer matches the external service's configuration of which repositories to mirror (e.g via `gitlab.projectQuery`, `github.repositoryQuery`, `github.repos`, etc) **SHOULD** result in newly accessible repositories to be mirrored and no longer accessible repositories to be soft-deleted.\r\n1. **Modifying configuration**: Modifying an external service's configuration of which repositories to mirror (e.g via `gitlab.projectQuery`, `github.repositoryQuery`, `github.repos`, etc) in any way that it results in some previously mirrored repositories no longer being selected to be mirrored **SHOULD** result in those repositories to be soft-deleted and previously unselected repositories now being selected, to be mirrored.\r\n\r\n## Dependencies\r\n\r\nBesides the previous work done as part of [\"Handle renames and deletions of mirrored repositories\" (#914)](https://github.com/sourcegraph/sourcegraph/issues/914), this proposal requires additional UI changes for:\r\n\r\n1. Showing repositories as deleted, including the reason for which they were deleted.\r\n\r\n## Implementation\r\n\r\nThis section describes the possible implementation paths for the above proposal and some key questions that need to be answered.\r\n\r\n### Tasks\r\n\r\n- [ ] TODO\r\n\r\n### Questions\r\n\r\n**How do we handle the same repository being selected from multiple external services and/or from multiple connections in a single external service?**\r\n\r\nMultiple matches of the same repository across multiple connections of the **same** external service are de-duplicated based on that external service's stable identifier for a single repository.\r\n\r\nWe don't de-deduplicate the same repository across different external services. See https://github.com/sourcegraph/sourcegraph/issues/1467#issuecomment-447979678 for more context.\r\n\r\n**What happens if a repository is renamed?**\r\n\r\nBased on the repository's stable identifier as provided by its external service, we can detect that its name has changed and just update it in our database.\r\n\r\nCached and derived data (git-server, search index) is re-fetched and re-computed. (@nicksnyder : How can this be triggered this?)\r\n\r\nCached and derived data associated with the previous repository name is lazily deleted.\r\n\r\n**What happens to a repository when no external service configuration selects it to be mirrored any longer?**\r\n\r\nIt's soft-deleted.\r\n\r\n**What happens to data that is attached to repositories (e.g. code discussions) when a repository is deleted?**\r\n\r\nThis data is soft-deleted along with the repository and can be restored with it too.\r\n\r\n**What happens when a token is deleted from the database?**\r\n\r\nAll repositories that can't be seen anymore because the token is gone will be soft-deleted."},"comment":{"url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/comments/448269214","html_url":"https://github.com/sourcegraph/sourcegraph/issues/1467#issuecomment-448269214","issue_url":"https://api.github.com/repos/sourcegraph/sourcegraph/issues/1467","id":448269214,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0ODI2OTIxNA==","user":{"login":"nicksnyder","id":754768,"node_id":"MDQ6VXNlcjc1NDc2OA==","avatar_url":"https://avatars1.githubusercontent.com/u/754768?v=4","gravatar_id":"","url":"https://api.github.com/users/nicksnyder","html_url":"https://github.com/nicksnyder","followers_url":"https://api.github.com/users/nicksnyder/followers","following_url":"https://api.github.com/users/nicksnyder/following{/other_user}","gists_url":"https://api.github.com/users/nicksnyder/gists{/gist_id}","starred_url":"https://api.github.com/users/nicksnyder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicksnyder/subscriptions","organizations_url":"https://api.github.com/users/nicksnyder/orgs","repos_url":"https://api.github.com/users/nicksnyder/repos","events_url":"https://api.github.com/users/nicksnyder/events{/privacy}","received_events_url":"https://api.github.com/users/nicksnyder/received_events","type":"User","site_admin":false},"created_at":"2018-12-18T15:54:34Z","updated_at":"2018-12-18T15:54:34Z","author_association":"MEMBER","body":"> both search and git-server had their own eviction policy implemented\r\n\r\nTo be precise:\r\n- gitserver currently only evicts on sourcegraph.com, but we might want to do this on private instances too in the new world\r\n- I don't know what search already does off the top of my head (@keegancsmith would know but he is on vacation), but I assume we would want searcher to behave similarly as gitserver in the new world.\r\n\r\n> Showing repositories as deleted, including the reason for which they were deleted.\r\n\r\nSince the repository is going to be removed from the UI, it is unclear where we would show this reason. It is also unclear to me how we would distinguish different reasons (e.g. access token no longer has scope, repo query changed, etc.). I don't think we need to show a reason because the reason is always \"this repo is no longer returned by the code host based on your current config\".\r\n\r\n> Cached or derived data like git-server storage and search indexes is not preserved because it's cheap to re-fetch / re-compute.\r\n\r\nI would rephrase this to be:\r\n\r\n\"Cached or derived data like git-server storage and search indexes is preserved because it's cheap to re-fetch / re-compute.\"\r\n\r\n> Cached and derived data (git-server, search index) is re-fetched and re-computed. (@nicksnyder : How can this be triggered this?)\r\n\r\nWe currently model things like this as a periodic janitor job (e.g. run every 24 hours) since we don't need to proactively purge this data."}}
{"actor":{"display_login":"nicksnyder"}}