{"action":"created","comment":{"url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/comments/187943249","pull_request_review_id":119821451,"id":187943249,"diff_hunk":"@@ -0,0 +1,123 @@\n+# Pod Eviction & Replacement\n+\n+This chapter specifies the rules around evicting pods from nodes and\n+restarting or replacing them.\n+\n+## Eviction\n+\n+Eviction is the process of removing a pod that is running on a node from that node.\n+\n+This is typically the result of a drain action (`kubectl drain`) or\n+from a taint being added to a node (either automatically by Kubernetes or manually by an operator).\n+\n+## Replacement\n+\n+Replacement is the process of replacing a pod an another pod that takes over the responsibilities\n+of the original pod.\n+\n+The replacement pod has a new ID and new (read empty) persistent data.\n+\n+Note that replacing a pod is different from restarting a pod. A pod is restarted when it has been reported\n+to have termined.\n+\n+## NoExecute Tolerations\n+\n+NoExecute tolerations are used to control the behavior of Kubernetes (wrt. to a Pod) when the node\n+that the pod is running on is no longer reachable or becomes not-ready.\n+\n+See the applicable [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) for more info.\n+\n+## Rules\n+\n+The rules for eviction & replacement are specified per type of pod.\n+\n+### Image ID Pods\n+\n+The Image ID pods are starter to fetch the ArangoDB version of a specific\n+ArangoDB image and fetch the docker sha256 of that image.\n+They have no persistent state.\n+\n+- Image ID pods can always be evicted from any node\n+- Image ID pods can always be restarted on a different node.\n+  There is no need to replace an image ID pod.\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set very low (5sec)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set very low (5sec)\n+\n+### Coordinator Pods\n+\n+Coordinator pods run an ArangoDB coordinator as part of an ArangoDB cluster.\n+They have no persistent state, but do have a unique ID.\n+\n+- Coordinator pods can always be evicted from any node\n+- Coordinator pods can always be replaced with another coordinator pod with a different ID on a different node\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set low (15sec)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set low (15sec)\n+\n+### DBServer Pods\n+\n+DBServer pods run an ArangoDB dbserver as part of an ArangoDB cluster.\n+It has persistent state potentially tight to the node it runs on and it has a unique ID.\n+\n+- DBServer pods can be evicted from any node as soon as:\n+  - It has been completely drained AND\n+  - It is no longer the shard master for any shard\n+- DBServer pods can be replaced with another dbserver pod with a different ID on a different node when:\n+  - It is not the shard master for any shard OR\n+  - For every shard it is the master for, there is an in-sync follower\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set high to \"wait it out a while\" (5min)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set high to \"wait it out a while\" (5min)\n+\n+### Agent Pods\n+\n+Agent pods run an ArangoDB dbserver as part of an ArangoDB agency.\n+It has persistent state potentially tight to the node it runs on and it has a unique ID.\n+\n+- Agent pods can be evicted from any node as soon as:\n+  - It is no longer the agency leader AND\n+  - There is at least an agency leader that is responding AND\n+  - There is at least an agency follower that is responding\n+- Agent pods can be replaced with another agent pod with the same ID but whiped persistent state on a different node when:\n+  - The old pod is known to be deleted (e.g. explicit eviction)\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is not set to \"wait it out forever\"\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is not set \"wait it out forever\"\n+\n+### Single Server Pods\n+\n+Single server pods run an ArangoDB server as part of an ArangoDB single server deployment.\n+It has persistent state potentially tight to the node.\n+\n+- Single server pods cannot be evicted from any node.\n+- Single server pods cannot be replaced with another pod.\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is not set to \"wait it out forever\"\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is not set \"wait it out forever\"\n+\n+### Single Pods in Active Failover Deployment\n+\n+Single pods run an ArangoDB single server as part of an ArangoDB active failover deployment.\n+It has persistent state potentially tight to the node it runs on and it has a unique ID.\n+\n+- Single pods can be evicted from any node as soon as:\n+  - It is a follower of an active-failover deployment (Q: can we trigger this failover to another server?)\n+- Single pods can always be replaced with another single pod with a different ID on a different node.\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set high to \"wait it out a while\" (5min)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set high to \"wait it out a while\" (5min)\n+\n+### SyncMaster Pods\n+\n+SyncMaster pods run an ArangoSync as master as part of an ArangoDB DC2DC cluster.\n+They have no persistent state, but do have a unique address.\n+\n+- SyncMaster pods can always be evicted from any node\n+- SyncMaster pods can always be replaced with another syncmaster pod on a different node\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set low (15sec)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set low (15sec)\n+\n+### SyncWorker Pods\n+\n+SyncWorker pods run an ArangoSync as worker as part of an ArangoDB DC2DC cluster.\n+They have no persistent state, but do have in-memory state and a unique address.\n+\n+- SyncWorker pods can always be evicted from any node\n+- SyncWorker pods can always be replaced with another syncworker pod on a different node\n+- `node.kubernetes.io/unreachable:NoExecute` toleration time is set a bit higher to try to avoid resynchronization (1min)\n+- `node.kubernetes.io/not-ready:NoExecute` toleration time is set a bit higher to try to avoid resynchronization (1min)","path":"docs/design/pod_evication_and_replacement.md","position":124,"original_position":123,"commit_id":"ab28b5007e011d84fb6ebc0913561436784e4755","original_commit_id":"6c612d12fb4e387bef3bb0a23cb5431a35cb53f5","user":{"login":"ewoutp","id":4750892,"avatar_url":"https://avatars0.githubusercontent.com/u/4750892?v=4","gravatar_id":"","url":"https://api.github.com/users/ewoutp","html_url":"https://github.com/ewoutp","followers_url":"https://api.github.com/users/ewoutp/followers","following_url":"https://api.github.com/users/ewoutp/following{/other_user}","gists_url":"https://api.github.com/users/ewoutp/gists{/gist_id}","starred_url":"https://api.github.com/users/ewoutp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ewoutp/subscriptions","organizations_url":"https://api.github.com/users/ewoutp/orgs","repos_url":"https://api.github.com/users/ewoutp/repos","events_url":"https://api.github.com/users/ewoutp/events{/privacy}","received_events_url":"https://api.github.com/users/ewoutp/received_events","type":"User","site_admin":false},"body":"no","created_at":"2018-05-14T12:59:26Z","updated_at":"2018-05-14T12:59:26Z","html_url":"https://github.com/arangodb/kube-arangodb/pull/133#discussion_r187943249","pull_request_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/comments/187943249"},"html":{"href":"https://github.com/arangodb/kube-arangodb/pull/133#discussion_r187943249"},"pull_request":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133"}},"in_reply_to_id":187929045},"pull_request":{"url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133","id":187132154,"html_url":"https://github.com/arangodb/kube-arangodb/pull/133","diff_url":"https://github.com/arangodb/kube-arangodb/pull/133.diff","patch_url":"https://github.com/arangodb/kube-arangodb/pull/133.patch","issue_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/133","number":133,"state":"open","locked":false,"title":"Added a spec regarding the rules for eviction & replacement of pods","user":{"login":"ewoutp","id":4750892,"avatar_url":"https://avatars0.githubusercontent.com/u/4750892?v=4","gravatar_id":"","url":"https://api.github.com/users/ewoutp","html_url":"https://github.com/ewoutp","followers_url":"https://api.github.com/users/ewoutp/followers","following_url":"https://api.github.com/users/ewoutp/following{/other_user}","gists_url":"https://api.github.com/users/ewoutp/gists{/gist_id}","starred_url":"https://api.github.com/users/ewoutp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ewoutp/subscriptions","organizations_url":"https://api.github.com/users/ewoutp/orgs","repos_url":"https://api.github.com/users/ewoutp/repos","events_url":"https://api.github.com/users/ewoutp/events{/privacy}","received_events_url":"https://api.github.com/users/ewoutp/received_events","type":"User","site_admin":false},"body":null,"created_at":"2018-05-10T08:32:16Z","updated_at":"2018-05-14T12:59:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"89c5288f53e6a74ba001a57b7b5a5f358ee2d5dd","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":867239703,"url":"https://api.github.com/repos/arangodb/kube-arangodb/labels/9%20Review%20Required","name":"9 Review Required","color":"1d76db","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133/commits","review_comments_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133/comments","review_comment_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/133/comments","statuses_url":"https://api.github.com/repos/arangodb/kube-arangodb/statuses/ab28b5007e011d84fb6ebc0913561436784e4755","head":{"label":"arangodb:documentation/eviction-and-replacement-spec","ref":"documentation/eviction-and-replacement-spec","sha":"ab28b5007e011d84fb6ebc0913561436784e4755","user":{"login":"arangodb","id":5547849,"avatar_url":"https://avatars0.githubusercontent.com/u/5547849?v=4","gravatar_id":"","url":"https://api.github.com/users/arangodb","html_url":"https://github.com/arangodb","followers_url":"https://api.github.com/users/arangodb/followers","following_url":"https://api.github.com/users/arangodb/following{/other_user}","gists_url":"https://api.github.com/users/arangodb/gists{/gist_id}","starred_url":"https://api.github.com/users/arangodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arangodb/subscriptions","organizations_url":"https://api.github.com/users/arangodb/orgs","repos_url":"https://api.github.com/users/arangodb/repos","events_url":"https://api.github.com/users/arangodb/events{/privacy}","received_events_url":"https://api.github.com/users/arangodb/received_events","type":"Organization","site_admin":false},"repo":{"id":120325908,"name":"kube-arangodb","full_name":"arangodb/kube-arangodb","owner":{"login":"arangodb","id":5547849,"avatar_url":"https://avatars0.githubusercontent.com/u/5547849?v=4","gravatar_id":"","url":"https://api.github.com/users/arangodb","html_url":"https://github.com/arangodb","followers_url":"https://api.github.com/users/arangodb/followers","following_url":"https://api.github.com/users/arangodb/following{/other_user}","gists_url":"https://api.github.com/users/arangodb/gists{/gist_id}","starred_url":"https://api.github.com/users/arangodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arangodb/subscriptions","organizations_url":"https://api.github.com/users/arangodb/orgs","repos_url":"https://api.github.com/users/arangodb/repos","events_url":"https://api.github.com/users/arangodb/events{/privacy}","received_events_url":"https://api.github.com/users/arangodb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/arangodb/kube-arangodb","description":"ArangoDB Kubernetes Operator - Start ArangoDB on Kubernetes in 5min","fork":false,"url":"https://api.github.com/repos/arangodb/kube-arangodb","forks_url":"https://api.github.com/repos/arangodb/kube-arangodb/forks","keys_url":"https://api.github.com/repos/arangodb/kube-arangodb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/arangodb/kube-arangodb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/arangodb/kube-arangodb/teams","hooks_url":"https://api.github.com/repos/arangodb/kube-arangodb/hooks","issue_events_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/events{/number}","events_url":"https://api.github.com/repos/arangodb/kube-arangodb/events","assignees_url":"https://api.github.com/repos/arangodb/kube-arangodb/assignees{/user}","branches_url":"https://api.github.com/repos/arangodb/kube-arangodb/branches{/branch}","tags_url":"https://api.github.com/repos/arangodb/kube-arangodb/tags","blobs_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/refs{/sha}","trees_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/arangodb/kube-arangodb/statuses/{sha}","languages_url":"https://api.github.com/repos/arangodb/kube-arangodb/languages","stargazers_url":"https://api.github.com/repos/arangodb/kube-arangodb/stargazers","contributors_url":"https://api.github.com/repos/arangodb/kube-arangodb/contributors","subscribers_url":"https://api.github.com/repos/arangodb/kube-arangodb/subscribers","subscription_url":"https://api.github.com/repos/arangodb/kube-arangodb/subscription","commits_url":"https://api.github.com/repos/arangodb/kube-arangodb/commits{/sha}","git_commits_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/commits{/sha}","comments_url":"https://api.github.com/repos/arangodb/kube-arangodb/comments{/number}","issue_comment_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/comments{/number}","contents_url":"https://api.github.com/repos/arangodb/kube-arangodb/contents/{+path}","compare_url":"https://api.github.com/repos/arangodb/kube-arangodb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/arangodb/kube-arangodb/merges","archive_url":"https://api.github.com/repos/arangodb/kube-arangodb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/arangodb/kube-arangodb/downloads","issues_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues{/number}","pulls_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls{/number}","milestones_url":"https://api.github.com/repos/arangodb/kube-arangodb/milestones{/number}","notifications_url":"https://api.github.com/repos/arangodb/kube-arangodb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/arangodb/kube-arangodb/labels{/name}","releases_url":"https://api.github.com/repos/arangodb/kube-arangodb/releases{/id}","deployments_url":"https://api.github.com/repos/arangodb/kube-arangodb/deployments","created_at":"2018-02-05T15:50:22Z","updated_at":"2018-05-14T08:15:55Z","pushed_at":"2018-05-14T12:58:47Z","git_url":"git://github.com/arangodb/kube-arangodb.git","ssh_url":"git@github.com:arangodb/kube-arangodb.git","clone_url":"https://github.com/arangodb/kube-arangodb.git","svn_url":"https://github.com/arangodb/kube-arangodb","homepage":"https://docs.arangodb.com/devel/Manual/Tutorials/Kubernetes/","size":12692,"stargazers_count":24,"watchers_count":24,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":14,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":2,"open_issues":14,"watchers":24,"default_branch":"master"}},"base":{"label":"arangodb:master","ref":"master","sha":"0c825bece5cfab8ea644295191185d2c347d4fd4","user":{"login":"arangodb","id":5547849,"avatar_url":"https://avatars0.githubusercontent.com/u/5547849?v=4","gravatar_id":"","url":"https://api.github.com/users/arangodb","html_url":"https://github.com/arangodb","followers_url":"https://api.github.com/users/arangodb/followers","following_url":"https://api.github.com/users/arangodb/following{/other_user}","gists_url":"https://api.github.com/users/arangodb/gists{/gist_id}","starred_url":"https://api.github.com/users/arangodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arangodb/subscriptions","organizations_url":"https://api.github.com/users/arangodb/orgs","repos_url":"https://api.github.com/users/arangodb/repos","events_url":"https://api.github.com/users/arangodb/events{/privacy}","received_events_url":"https://api.github.com/users/arangodb/received_events","type":"Organization","site_admin":false},"repo":{"id":120325908,"name":"kube-arangodb","full_name":"arangodb/kube-arangodb","owner":{"login":"arangodb","id":5547849,"avatar_url":"https://avatars0.githubusercontent.com/u/5547849?v=4","gravatar_id":"","url":"https://api.github.com/users/arangodb","html_url":"https://github.com/arangodb","followers_url":"https://api.github.com/users/arangodb/followers","following_url":"https://api.github.com/users/arangodb/following{/other_user}","gists_url":"https://api.github.com/users/arangodb/gists{/gist_id}","starred_url":"https://api.github.com/users/arangodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arangodb/subscriptions","organizations_url":"https://api.github.com/users/arangodb/orgs","repos_url":"https://api.github.com/users/arangodb/repos","events_url":"https://api.github.com/users/arangodb/events{/privacy}","received_events_url":"https://api.github.com/users/arangodb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/arangodb/kube-arangodb","description":"ArangoDB Kubernetes Operator - Start ArangoDB on Kubernetes in 5min","fork":false,"url":"https://api.github.com/repos/arangodb/kube-arangodb","forks_url":"https://api.github.com/repos/arangodb/kube-arangodb/forks","keys_url":"https://api.github.com/repos/arangodb/kube-arangodb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/arangodb/kube-arangodb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/arangodb/kube-arangodb/teams","hooks_url":"https://api.github.com/repos/arangodb/kube-arangodb/hooks","issue_events_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/events{/number}","events_url":"https://api.github.com/repos/arangodb/kube-arangodb/events","assignees_url":"https://api.github.com/repos/arangodb/kube-arangodb/assignees{/user}","branches_url":"https://api.github.com/repos/arangodb/kube-arangodb/branches{/branch}","tags_url":"https://api.github.com/repos/arangodb/kube-arangodb/tags","blobs_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/refs{/sha}","trees_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/arangodb/kube-arangodb/statuses/{sha}","languages_url":"https://api.github.com/repos/arangodb/kube-arangodb/languages","stargazers_url":"https://api.github.com/repos/arangodb/kube-arangodb/stargazers","contributors_url":"https://api.github.com/repos/arangodb/kube-arangodb/contributors","subscribers_url":"https://api.github.com/repos/arangodb/kube-arangodb/subscribers","subscription_url":"https://api.github.com/repos/arangodb/kube-arangodb/subscription","commits_url":"https://api.github.com/repos/arangodb/kube-arangodb/commits{/sha}","git_commits_url":"https://api.github.com/repos/arangodb/kube-arangodb/git/commits{/sha}","comments_url":"https://api.github.com/repos/arangodb/kube-arangodb/comments{/number}","issue_comment_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues/comments{/number}","contents_url":"https://api.github.com/repos/arangodb/kube-arangodb/contents/{+path}","compare_url":"https://api.github.com/repos/arangodb/kube-arangodb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/arangodb/kube-arangodb/merges","archive_url":"https://api.github.com/repos/arangodb/kube-arangodb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/arangodb/kube-arangodb/downloads","issues_url":"https://api.github.com/repos/arangodb/kube-arangodb/issues{/number}","pulls_url":"https://api.github.com/repos/arangodb/kube-arangodb/pulls{/number}","milestones_url":"https://api.github.com/repos/arangodb/kube-arangodb/milestones{/number}","notifications_url":"https://api.github.com/repos/arangodb/kube-arangodb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/arangodb/kube-arangodb/labels{/name}","releases_url":"https://api.github.com/repos/arangodb/kube-arangodb/releases{/id}","deployments_url":"https://api.github.com/repos/arangodb/kube-arangodb/deployments","created_at":"2018-02-05T15:50:22Z","updated_at":"2018-05-14T08:15:55Z","pushed_at":"2018-05-14T12:58:47Z","git_url":"git://github.com/arangodb/kube-arangodb.git","ssh_url":"git@github.com:arangodb/kube-arangodb.git","clone_url":"https://github.com/arangodb/kube-arangodb.git","svn_url":"https://github.com/arangodb/kube-arangodb","homepage":"https://docs.arangodb.com/devel/Manual/Tutorials/Kubernetes/","size":12692,"stargazers_count":24,"watchers_count":24,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":14,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":2,"open_issues":14,"watchers":24,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133"},"html":{"href":"https://github.com/arangodb/kube-arangodb/pull/133"},"issue":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/issues/133"},"comments":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/issues/133/comments"},"review_comments":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133/comments"},"review_comment":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/pulls/133/commits"},"statuses":{"href":"https://api.github.com/repos/arangodb/kube-arangodb/statuses/ab28b5007e011d84fb6ebc0913561436784e4755"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"ewoutp"}}