{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/4965","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/4965/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/4965/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/4965/events","html_url":"https://github.com/coreos/etcd/issues/4965","id":145873534,"number":4965,"title":"Etcd plugin / middleware proposal","user":{"login":"mqliang","id":12423751,"avatar_url":"https://avatars.githubusercontent.com/u/12423751?v=3","gravatar_id":"","url":"https://api.github.com/users/mqliang","html_url":"https://github.com/mqliang","followers_url":"https://api.github.com/users/mqliang/followers","following_url":"https://api.github.com/users/mqliang/following{/other_user}","gists_url":"https://api.github.com/users/mqliang/gists{/gist_id}","starred_url":"https://api.github.com/users/mqliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mqliang/subscriptions","organizations_url":"https://api.github.com/users/mqliang/orgs","repos_url":"https://api.github.com/users/mqliang/repos","events_url":"https://api.github.com/users/mqliang/events{/privacy}","received_events_url":"https://api.github.com/users/mqliang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-04-05T03:12:24Z","updated_at":"2016-04-05T03:36:45Z","closed_at":null,"body":"### Etcd plugin / middleware proposal\r\n----\r\n\r\n#### Motivation\r\n1. As a user, I'd like etcd support criteria query and secondary indexing.\r\n\r\n    In some case, we'd like etcd support criteria query, for example: in k8s, Scheduler just want list&watch unscheduled pods(`pod.spec.nodeName=\"\"`), Kubelet \r\njust want list&watch the pods assigned to itself(`pod.spec.nodeName=\"XXX\"`). Current, k8s address this in API server: API server list&watch all pods, but will filter the results by `ListOption` and just return \"valid\" ones. Moving the filter logic to etcd brings some network and decoding benefits. \r\n\r\n2. As a user, I'd like to integrate the authentication and authorization of etcd with my own KeyStone service.\r\n    \r\n    I already have a KeyStone service, I'd like use my own KeyStone service to do the authentication and authorization of etcd: who can just read some keys, who can read and write those keys.\r\n\r\n3. As a user, I'd like to do server side data sharding, so for example I may want a federation with 3 clusters, where:\r\n    * Cluster A contains hash slots from 0 to 100.\r\n    * Cluster B contains hash slots from 100 to 200.\r\n    * Cluster B contains hash slots from 200 to 300.\r\n\r\n    Client could send request to anyone of the three clusters, cluster will serve the request if the key is in the cluster's slot, or just forward the request to another cluster in the federation.\r\n\r\n\r\n#### One possibe approache to meet above requirements\r\n\r\n1. Support third-party plugin\r\nExpose some hook functions to third-party developers. For example, for all Put(), Delete(), Get() methods, third-party developers could defines their own preXXX and postXXX methods in a plugin. For some popular foramt such json, third-party developers may provide a good and widely-use plugin to support criteria query and secondary indexing.\r\n\r\n2. Introduce middleware or something alike when serve requests.\r\nIf etcd could introduce middleware when serve requests, third-party developer could easily write their own middleware to authenticate/authorize requests, and forward requests based on some sharding rules.\r\n\r\n\r\nAs a open source project, even if we don't want support some features since we want make etcd \"simple, easy to use, focusing on reliable key-value store\", allowing third-party developer write their plugin and middleware to implement their feature they want seems not a bad thing. "},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/205626113","html_url":"https://github.com/coreos/etcd/issues/4965#issuecomment-205626113","issue_url":"https://api.github.com/repos/coreos/etcd/issues/4965","id":205626113,"user":{"login":"magicwang-cn","id":7587313,"avatar_url":"https://avatars.githubusercontent.com/u/7587313?v=3","gravatar_id":"","url":"https://api.github.com/users/magicwang-cn","html_url":"https://github.com/magicwang-cn","followers_url":"https://api.github.com/users/magicwang-cn/followers","following_url":"https://api.github.com/users/magicwang-cn/following{/other_user}","gists_url":"https://api.github.com/users/magicwang-cn/gists{/gist_id}","starred_url":"https://api.github.com/users/magicwang-cn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magicwang-cn/subscriptions","organizations_url":"https://api.github.com/users/magicwang-cn/orgs","repos_url":"https://api.github.com/users/magicwang-cn/repos","events_url":"https://api.github.com/users/magicwang-cn/events{/privacy}","received_events_url":"https://api.github.com/users/magicwang-cn/received_events","type":"User","site_admin":false},"created_at":"2016-04-05T03:36:45Z","updated_at":"2016-04-05T03:36:45Z","body":"+1. \r\n\r\nyes, with the huge load, client will return (`401`)  frequently, because etcd has a default watch window(`1000 size for all keys`), and the etcd client does not support streaming results. so filter in server side will be useful.\r\n>In some case, we'd like etcd support criteria query, for example: in k8s, Scheduler just want list&watch unscheduled pods(pod.spec.nodeName=\"\"), Kubelet \r\njust want list&watch the pods assigned to itself(pod.spec.nodeName=\"XXX\"). Current, k8s address this in API server: API server list&watch all pods, but will filter the results by ListOption and just return \"valid\" ones. Moving the filter logic to etcd brings some network and decoding benefits."}}