{"action":"created","issue":{"url":"https://api.github.com/repos/koding/koding/issues/10776","repository_url":"https://api.github.com/repos/koding/koding","labels_url":"https://api.github.com/repos/koding/koding/issues/10776/labels{/name}","comments_url":"https://api.github.com/repos/koding/koding/issues/10776/comments","events_url":"https://api.github.com/repos/koding/koding/issues/10776/events","html_url":"https://github.com/koding/koding/issues/10776","id":214498457,"node_id":"MDU6SXNzdWUyMTQ0OTg0NTc=","number":10776,"title":"kloud/provider: add Kubernetes support","user":{"login":"acbodine","id":7049554,"node_id":"MDQ6VXNlcjcwNDk1NTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/7049554?v=4","gravatar_id":"","url":"https://api.github.com/users/acbodine","html_url":"https://github.com/acbodine","followers_url":"https://api.github.com/users/acbodine/followers","following_url":"https://api.github.com/users/acbodine/following{/other_user}","gists_url":"https://api.github.com/users/acbodine/gists{/gist_id}","starred_url":"https://api.github.com/users/acbodine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acbodine/subscriptions","organizations_url":"https://api.github.com/users/acbodine/orgs","repos_url":"https://api.github.com/users/acbodine/repos","events_url":"https://api.github.com/users/acbodine/events{/privacy}","received_events_url":"https://api.github.com/users/acbodine/received_events","type":"User","site_admin":false},"labels":[{"id":581037654,"node_id":"MDU6TGFiZWw1ODEwMzc2NTQ=","url":"https://api.github.com/repos/koding/koding/labels/brave","name":"brave","color":"082756","default":false}],"state":"open","locked":false,"assignee":{"login":"rjeczalik","id":1162017,"node_id":"MDQ6VXNlcjExNjIwMTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1162017?v=4","gravatar_id":"","url":"https://api.github.com/users/rjeczalik","html_url":"https://github.com/rjeczalik","followers_url":"https://api.github.com/users/rjeczalik/followers","following_url":"https://api.github.com/users/rjeczalik/following{/other_user}","gists_url":"https://api.github.com/users/rjeczalik/gists{/gist_id}","starred_url":"https://api.github.com/users/rjeczalik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rjeczalik/subscriptions","organizations_url":"https://api.github.com/users/rjeczalik/orgs","repos_url":"https://api.github.com/users/rjeczalik/repos","events_url":"https://api.github.com/users/rjeczalik/events{/privacy}","received_events_url":"https://api.github.com/users/rjeczalik/received_events","type":"User","site_admin":false},"assignees":[{"login":"rjeczalik","id":1162017,"node_id":"MDQ6VXNlcjExNjIwMTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1162017?v=4","gravatar_id":"","url":"https://api.github.com/users/rjeczalik","html_url":"https://github.com/rjeczalik","followers_url":"https://api.github.com/users/rjeczalik/followers","following_url":"https://api.github.com/users/rjeczalik/following{/other_user}","gists_url":"https://api.github.com/users/rjeczalik/gists{/gist_id}","starred_url":"https://api.github.com/users/rjeczalik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rjeczalik/subscriptions","organizations_url":"https://api.github.com/users/rjeczalik/orgs","repos_url":"https://api.github.com/users/rjeczalik/repos","events_url":"https://api.github.com/users/rjeczalik/events{/privacy}","received_events_url":"https://api.github.com/users/rjeczalik/received_events","type":"User","site_admin":false},{"login":"acbodine","id":7049554,"node_id":"MDQ6VXNlcjcwNDk1NTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/7049554?v=4","gravatar_id":"","url":"https://api.github.com/users/acbodine","html_url":"https://github.com/acbodine","followers_url":"https://api.github.com/users/acbodine/followers","following_url":"https://api.github.com/users/acbodine/following{/other_user}","gists_url":"https://api.github.com/users/acbodine/gists{/gist_id}","starred_url":"https://api.github.com/users/acbodine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acbodine/subscriptions","organizations_url":"https://api.github.com/users/acbodine/orgs","repos_url":"https://api.github.com/users/acbodine/repos","events_url":"https://api.github.com/users/acbodine/events{/privacy}","received_events_url":"https://api.github.com/users/acbodine/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2017-03-15T19:21:26Z","updated_at":"2018-09-27T03:09:14Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"This serves as a retainer for the design and implementation of a Kubernetes provider for Koding.\r\n\r\n## Overview\r\n\r\n[Kubernetes](https://kubernetes.io/) is a container orchestration technology that has gained critical traction in the world of development and production devops centered around container-based application workloads. You can even bring over your existing `Dockerfile`s and Kubernetes will happily accept your Docker images into its workload abstractions called `pods`.\r\n\r\n<img\r\n    alt=\"Kubernetes Architecture Overview\"\r\n    width=\"600\"\r\n    src=\"https://github.com/kubernetes/kubernetes/raw/v1.5.1/docs/design/architecture.png?raw=true\"\r\n/>\r\n\r\nYou tell Kubernetes (with `kubectl`) what you want the topology of your application to look like `desired state`, usually by means of a YAML specification (deployment, replica set, job), and Kubernetes makes the `actual state` match the `desired state` in a safe and reliable way.\r\n\r\nInterested in [design docs](https://github.com/kubernetes/kubernetes/tree/v1.5.1/docs/design) for Kubernetes?\r\n\r\nAs a basic example of a deployment specification to Kubernetes for nginx:\r\n```\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  name: nginx-deployment\r\nspec:\r\n  replicas: 3\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: nginx\r\n    spec:\r\n      containers:\r\n      - name: nginx\r\n        image: nginx:1.7.9\r\n        ports:\r\n        - containerPort: 80\r\n```\r\nThis will result in Kubernetes:\r\n1. Pulling the `ngingx:1.7.9` image from [Docker Hub](https://hub.docker.com/) if it doesn't already have it\r\n2. Creating and running 3 containers from that image all listening on their own port 80s\r\n3. Creating a service endpoint in the Kubernetes proxy that all 3 of the nginx containers serve.\r\n\r\nThe service endpoint should persist through application updates (rolling), so it is ideal for configuration/environment variables for consumers. This would be analogous to the public endpoint that current Koding stacks serve once provisioned based on their specific application port(s).\r\n\r\n## Expected Behavior\r\n\r\nWhat do we want/expect from Kubernetes?\r\n\r\nAs a Koding user I want to be able to configure the following for my Kubernetes Pod Stack:\r\n- Images\r\n- Scale\r\n- Storage\r\n- Network\r\n- Quotas\r\n\r\n> NOTE: Modifications to deployments running on Kubernetes that are done outside of Koding's knowledge, shouldn't break Koding's implementation. Have some sort of discovery process, the Kubernetes API should be able to make this a simple task.\r\n\r\nOnce my Pod Stack has been provisioned on the existing Kubernetes cluster, that I specify at stack creation/initialization, I should be able to access my Pod Stack from a Kubernetes service endpoint.\r\n\r\nI also want the ability to:\r\n- Exec into any container that Koding is aware of in the Kubernetes cluster\r\n- Scale in/out any Pod Stack\r\n\r\n## Current Behavior\r\n\r\nAs a close reference point, we are doing a small comparison with the existing Marathon provider for Koding. Currently the Marathon provider focuses on single container applications, given this restriction the implementation injects Koding's agent `klient` into the container via a startup script before running the user's application inside the container.\r\n\r\nThis isn’t great in terms of scale, and wrapping the application with an entrypoint script to install klient at container runtime is slow and hacky. Thus, we haven’t ruled out modifying `klient`.\r\n\r\nWhile single container applications work great for the basic use cases, it might not be the most realistic view of how users expect to consume hosted container orchestration services. Perhaps a more realistic view is at least an application that has more than one process (multiple container images) and each of those processes needs to be scaled out. The number of `klient` processes that would be in play with the current approach, grows very rapidly.\r\n\r\n## Possible Solution\r\n\r\nIf we take a step back, and let's say we rely on Kubernetes to do it's job, trust it even. All the Koding provider needs to do at that point is call the right Kubernetes API operations with the correct data structures representing the `desired state`. Every API operation from Kubernetes results in a [record of intent](https://kubernetes.io/docs/concepts/abstractions/overview/) response. Meaning Kubernetes will make the `actual state` match the `desired state` as soon as possible.\r\n\r\nA possible solution that we have discussed briefly is to have a dedicated `klient` container per deployment that the provider deploys to Kubernetes cluster. There are certain guarantees that Kubernetes provides that makes this an appealing solution per deployment: \r\n- Co-located processes\r\n- Shared storage (volumes)\r\n- Shared network\r\n- Shared namespaces\r\n- Everyone fails or succeeds together (no half-functioning deployment)\r\n\r\nIn short, we are guaranteed that:\r\n- if our deployment is up, our corresponding `klient` container is also up\r\n- if our deployment is down for any reason, our corresponding `klient` container is also down\r\n\r\nThis would remove the need to inject an entrypoint script into each container image, and we would gain back any built-in Kubernetes features that the current approach would mask such as: healthchecking.\r\n\r\nPossibly we should look into syncing Koding credentials to the provided Kubernetes cluster to leverage the benefits of Kubernetes providing credentials to user containers.\r\n\r\nKubernetes community maintains a Golang [client](https://github.com/kubernetes/client-go) so we can programmatically invoke the Kubernetes API to set `desired state`.\r\n\r\nDo we still want to use Terraform under the covers here?\r\n> It might not make sense to. Terraform is about provisioning cloud resources, and in our proposed > solution we believe the Kubernetes provider shouldn't be involved with managing the Kubernetes\r\n> cluster (GCE, Bluemix Container Service, On-Prem), the provider should only\r\n> create/update/destroy deployments/pods via the Kubernetes API. Terraform also assumes it is\r\n> authoritative over cloud resources. If someone were to use `kubectl` outside of Koding\r\n> there would likely be a difference in state records.\r\n"},"comment":{"url":"https://api.github.com/repos/koding/koding/issues/comments/424944166","html_url":"https://github.com/koding/koding/issues/10776#issuecomment-424944166","issue_url":"https://api.github.com/repos/koding/koding/issues/10776","id":424944166,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDk0NDE2Ng==","user":{"login":"ChrisCates","id":1561091,"node_id":"MDQ6VXNlcjE1NjEwOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/1561091?v=4","gravatar_id":"","url":"https://api.github.com/users/ChrisCates","html_url":"https://github.com/ChrisCates","followers_url":"https://api.github.com/users/ChrisCates/followers","following_url":"https://api.github.com/users/ChrisCates/following{/other_user}","gists_url":"https://api.github.com/users/ChrisCates/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisCates/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisCates/subscriptions","organizations_url":"https://api.github.com/users/ChrisCates/orgs","repos_url":"https://api.github.com/users/ChrisCates/repos","events_url":"https://api.github.com/users/ChrisCates/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisCates/received_events","type":"User","site_admin":false},"created_at":"2018-09-27T03:09:14Z","updated_at":"2018-09-27T03:09:14Z","author_association":"NONE","body":"@LemonAndroid, I agree... For book keeping purposes, I created #11505 to clarify any existing issues between Docker and Kuburnetes.\r\n\r\nFurthermore. this bounty has been sitting on BountySource.com for several years...\r\nThe Koding Maintainers have been active for almost 2 months as well...\r\n\r\nMy suggestion if you're looking for bounties is to check out @gitcoin and gitcoin.co.\r\nThey are much more active."}}
{"actor":{"display_login":"ChrisCates"}}