{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/AKS/issues/521","repository_url":"https://api.github.com/repos/Azure/AKS","labels_url":"https://api.github.com/repos/Azure/AKS/issues/521/labels{/name}","comments_url":"https://api.github.com/repos/Azure/AKS/issues/521/comments","events_url":"https://api.github.com/repos/Azure/AKS/issues/521/events","html_url":"https://github.com/Azure/AKS/issues/521","id":340200342,"node_id":"MDU6SXNzdWUzNDAyMDAzNDI=","number":521,"title":"helm install breaks nondeterministically","user":{"login":"markwaldkat","id":35069986,"node_id":"MDQ6VXNlcjM1MDY5OTg2","avatar_url":"https://avatars1.githubusercontent.com/u/35069986?v=4","gravatar_id":"","url":"https://api.github.com/users/markwaldkat","html_url":"https://github.com/markwaldkat","followers_url":"https://api.github.com/users/markwaldkat/followers","following_url":"https://api.github.com/users/markwaldkat/following{/other_user}","gists_url":"https://api.github.com/users/markwaldkat/gists{/gist_id}","starred_url":"https://api.github.com/users/markwaldkat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markwaldkat/subscriptions","organizations_url":"https://api.github.com/users/markwaldkat/orgs","repos_url":"https://api.github.com/users/markwaldkat/repos","events_url":"https://api.github.com/users/markwaldkat/events{/privacy}","received_events_url":"https://api.github.com/users/markwaldkat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-11T11:23:53Z","updated_at":"2018-07-12T07:35:21Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nI created a AKS cluster like this:\r\n\r\n```\r\n  az aks create \\\r\n  --name ${CONTAINER_SERVICE} \\\r\n  --resource-group ${RESOURCE_GROUP} \\\r\n  --client-secret ${CLIENT_SECRET} \\\r\n  --node-count 4 \\\r\n  --service-principal ${SERVICE_PRINCIPAL} \\\r\n  --location westeurope \\\r\n  --node-vm-size Standard_D4s_v3 \\\r\n  --kubernetes-version \"1.9.6\" \r\n  --enable-rbac\r\n```\r\n\r\nbut when I install our companies software via multiple \"helm install ...\" calls at least one fails always and it seems it's nondeterministic which one.\r\n\r\nError messages look like this:\r\n\r\n- Error: release excited-scorpion failed: the server could not find the requested resource (get pods)\r\n- Error: release quieting-saola failed: the server could not find the requested resource (get replicasets.extensions)\r\n- message: 'Deployment creation failed: the server could not find the requested resource                                                                                                                                                                                       \r\n    (post deployments.apps)'                                                                                                                                                                                                                                                   \r\n\r\nI found some interesting logs for a specific case:\r\nError: release dapper-aardvark failed: the server could not find the requested resource (post deployments.extensions)\r\n\r\nTiller logs:\r\nThese are the correspondent tiller logs:\r\n\r\nKubectl -n kube-system logs tiller-deploy-7b7b795779-gthts\r\n[tiller] 2018/07/10 11:33:53 preparing install for\r\n[storage] 2018/07/10 11:33:53 getting release \"dapper-aardvark.v1\"\r\n[tiller] 2018/07/10 11:33:53 rendering xxx chart using values\r\n[tiller] 2018/07/10 11:33:53 performing install for\r\n[tiller] 2018/07/10 11:33:53 executing 0 pre-install hooks for dapper-aardvark\r\n[tiller] 2018/07/10 11:33:53 hooks complete for pre-install dapper-aardvark\r\n[storage] 2018/07/10 11:33:53 getting release history for \"\"\r\n[storage] 2018/07/10 11:33:53 creating release \"dapper-aardvark.v1\"\r\n[kube] 2018/07/10 11:33:54 building resources from manifest\r\n[kube] 2018/07/10 11:33:54 creating 2 resource(s)\r\n[tiller] 2018/07/10 11:33:54 warning: Release \"dapper-aardvark\" failed: the server could not find the requested resource (post deployments.extensions)\r\n[storage] 2018/07/10 11:33:54 updating release \"dapper-aardvark.v1\"\r\n[tiller] 2018/07/10 11:33:54 failed install perform step: release dapper-aardvark failed: the server could not find the requested resource (post deployments.extensions)\r\n\r\nI actually can see all calls to the API server as they’re going through the pod azureproxy.\r\nkubectl -n kube-system logs azureproxy-5fc9fc6976-f4tq4\r\nand I see this interesting line:\r\n10.244.3.2 - - [10/Jul/2018:11:33:54 +0000] \"POST /apis/extensions/v1beta1/namespaces/<mynamespace>/deployments HTTP/1.1\" 404 19 \"-\" \"tiller/v0.0.0 (linux/amd64) kubernetes/$Format\"\r\n\r\nNote the HTTP 404.\r\n\r\nA lot of these calls have been done successfully before:\r\n10.244.3.2 - - [10/Jul/2018:10:38:02 +0000] \"POST /apis/extensions/v1beta1/namespaces/<mynamespace>/deployments HTTP/1.1\" 201 3293 \"-\" \"tiller/v0.0.0 (linux/amd64) kubernetes/$Format\"\r\n\r\nNot sure if this is related:\r\nkubectl -n kube-system get event|grep NetworkNotReady\r\n42m         42m          3         azureproxy-5fc9fc6976-f4tq4.153ffd86dab1dbbd             Pod                   Warning   NetworkNotReady         kubelet, aks-nodepool1-26240651-0   network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]\r\n\r\nThe VMs have the tag acsengineVersion : v0.19.0-aks.\r\nI remember not having this problem in earlier versions, such as  v0.16.0.\r\n\r\nThis is definitely reproducable on every new AKS cluster I create and this doesn't happen on other clouds such as GKE and AWS.\r\nAll nodes are ready btw.\r\n\r\nVersions I used:\r\nkubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.6\", GitCommit:\"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\", GitTreeState:\"clean\", BuildDate:\"2018-03-21T15:21:50Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.6\", GitCommit:\"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\", GitTreeState:\"clean\", BuildDate:\"2018-03-21T15:13:31Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\nhelm version\r\nClient: &version.Version{SemVer:\"v2.9.1\", GitCommit:\"20adb27c7c5868466912eebdf6664e7390ebe710\", GitTreeState:\"clean\"}\r\nServer: &version.Version{SemVer:\"v2.9.1\", GitCommit:\"20adb27c7c5868466912eebdf6664e7390ebe710\", GitTreeState:\"clean\"}\r\n\r\naz --version\r\nazure-cli (2.0.41)\r\n\r\nBest,\r\nMark\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Azure/AKS/issues/comments/404418228","html_url":"https://github.com/Azure/AKS/issues/521#issuecomment-404418228","issue_url":"https://api.github.com/repos/Azure/AKS/issues/521","id":404418228,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDQxODIyOA==","user":{"login":"mwaldkat","id":25075807,"node_id":"MDQ6VXNlcjI1MDc1ODA3","avatar_url":"https://avatars3.githubusercontent.com/u/25075807?v=4","gravatar_id":"","url":"https://api.github.com/users/mwaldkat","html_url":"https://github.com/mwaldkat","followers_url":"https://api.github.com/users/mwaldkat/followers","following_url":"https://api.github.com/users/mwaldkat/following{/other_user}","gists_url":"https://api.github.com/users/mwaldkat/gists{/gist_id}","starred_url":"https://api.github.com/users/mwaldkat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwaldkat/subscriptions","organizations_url":"https://api.github.com/users/mwaldkat/orgs","repos_url":"https://api.github.com/users/mwaldkat/repos","events_url":"https://api.github.com/users/mwaldkat/events{/privacy}","received_events_url":"https://api.github.com/users/mwaldkat/received_events","type":"User","site_admin":false},"created_at":"2018-07-12T07:35:21Z","updated_at":"2018-07-12T07:35:21Z","author_association":"NONE","body":"I removed the namespace for this bug report because of privacy reasons and added a placeholder that is swallowed by the rendering on this page because I didn't properly escape it.\r\nIt is /apis/extensions/v1beta1/namespaces/\\<some_actual_working_namespace\\>/deployments.\r\nThanks, but this is not the problem.\r\n"}}
{"actor":{"display_login":"mwaldkat"}}