{"action":"created","issue":{"url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/350","repository_url":"https://api.github.com/repos/nginxinc/kubernetes-ingress","labels_url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/350/labels{/name}","comments_url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/350/comments","events_url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/350/events","html_url":"https://github.com/nginxinc/kubernetes-ingress/issues/350","id":354503010,"node_id":"MDU6SXNzdWUzNTQ1MDMwMTA=","number":350,"title":"Error retrieving default-server-secret","user":{"login":"mkryshak","id":36676421,"node_id":"MDQ6VXNlcjM2Njc2NDIx","avatar_url":"https://avatars2.githubusercontent.com/u/36676421?v=4","gravatar_id":"","url":"https://api.github.com/users/mkryshak","html_url":"https://github.com/mkryshak","followers_url":"https://api.github.com/users/mkryshak/followers","following_url":"https://api.github.com/users/mkryshak/following{/other_user}","gists_url":"https://api.github.com/users/mkryshak/gists{/gist_id}","starred_url":"https://api.github.com/users/mkryshak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkryshak/subscriptions","organizations_url":"https://api.github.com/users/mkryshak/orgs","repos_url":"https://api.github.com/users/mkryshak/repos","events_url":"https://api.github.com/users/mkryshak/events{/privacy}","received_events_url":"https://api.github.com/users/mkryshak/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-27T22:35:00Z","updated_at":"2018-08-28T09:15:20Z","closed_at":null,"author_association":"NONE","body":"**Description**\r\nNGINX/NGINX+ ingress controller is unable to start because the default-server-secret is unable to be retrieved.\r\n\r\n**How to Reproduce**\r\n1.  Deploy the following yaml file (with redactions):\r\n\r\n```apiVersion: v1\r\nkind: ServiceAccount\r\nmetadata:\r\n  name: nginx-ingress\r\n  namespace: default\r\n---\r\nkind: ClusterRole\r\napiVersion: rbac.authorization.k8s.io/v1beta1\r\nmetadata:\r\n  name: nginx-ingress-rbac\r\nrules:\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - services\r\n  - endpoints\r\n  verbs:\r\n  - get\r\n  - list\r\n  - watch\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - secrets\r\n  verbs:\r\n  - get\r\n  - list\r\n  - watch\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - configmaps\r\n  verbs:\r\n  - get\r\n  - list\r\n  - watch\r\n  - update\r\n  - create\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - pods\r\n  verbs:\r\n  - list\r\n- apiGroups:\r\n  - \"\"\r\n  resources:\r\n  - events\r\n  verbs:\r\n  - create\r\n  - patch\r\n- apiGroups:\r\n  - extensions\r\n  resources:\r\n  - ingresses\r\n  verbs:\r\n  - list\r\n  - watch\r\n---\r\nkind: ClusterRoleBinding\r\napiVersion: rbac.authorization.k8s.io/v1beta1\r\nmetadata:\r\n  name: nginx-ingress\r\nsubjects:\r\n- kind: ServiceAccount\r\n  name: nginx-ingress\r\n  namespace: default\r\nroleRef:\r\n  kind: ClusterRole\r\n  name: nginx-ingress-rbac\r\n  apiGroup: rbac.authorization.k8s.io\r\n---\r\napiVersion: v1\r\nkind: Secret\r\nmetadata:\r\n  name: default-server-secret\r\n  namespace: default\r\ntype: Opaque\r\ndata:\r\n  tls.crt: <redacted>\r\n  tls.key: <redacted>\r\n---\r\nkind: ConfigMap\r\napiVersion: v1\r\nmetadata:\r\n  name: httpcontroller-configmap\r\n  namespace: default\r\ndata:\r\n  set-real-ip-from: \"0.0.0.0/0\"\r\n  log-format: '$proxy_protocol_addr - $remote_user [$time_local] \"$request_method $scheme://$host$request_uri $server_protocol\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time'\r\n---\r\napiVersion: extensions/v1beta1\r\nkind: DaemonSet\r\nmetadata:\r\n  name: httpcontroller-ic\r\n  namespace: default\r\nspec:\r\n  updateStrategy:\r\n    type: RollingUpdate\r\n    rollingUpdate:\r\n      maxUnavailable: 1\r\n  selector:\r\n    matchLabels:\r\n      app: httpcontroller-ic\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: httpcontroller-ic\r\n      annotations:\r\n        prometheus.io/scrape: \"true\"\r\n        prometheus.io/port: \"9113\"\r\n    spec:\r\n      serviceAccountName: nginx-ingress\r\n      containers:\r\n      - image: artifactory.nevint.com:5004/<redacted>/nginxplus:1.3.0\r\n        name: nginx-plus-ingress\r\n        ports:\r\n        - name: http\r\n          containerPort: 80\r\n          hostPort: 80\r\n        - name: https\r\n          containerPort: 443\r\n          hostPort: 443\r\n        env:\r\n        - name: POD_NAMESPACE\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.namespace\r\n        - name: POD_NAME\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.name\r\n        args:\r\n          - -nginx-plus\r\n          - -nginx-configmaps=$(POD_NAMESPACE)/httpcontroller-configmap\r\n          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret\r\n          - -ingress-class=httpcontroller\r\n          - -use-ingress-class-only\r\n      - image: artifactory.nevint.com:5004/<redacted>/nginx-prometheus-exporter:0.1.0\r\n        name: nginx-prometheus-exporter\r\n        ports:\r\n        - name: prometheus\r\n          containerPort: 9113\r\n        args:\r\n          - -web.listen-address\r\n          - :9113\r\n          - -nginx.plus\r\n          - -nginx.scrape-uri\r\n          - http://127.0.0.1:8080/api\r\n```\r\n\r\n2. Dockerfile for N+\r\n```\r\nFROM debian:stretch-slim\r\n \r\nLABEL maintainer=\"NGINX Docker Maintainers <docker-maint@nginx.com>\"\r\n \r\nENV NGINX_PLUS_VERSION 15-2~stretch\r\n \r\n### added for debugging tools\r\n \r\nRUN DEBIAN_FRONTEND=noninteractive apt-get -yq update && DEBIAN_FRONTEND=noninteractive apt-get upgrade -yqq -o Dpkg::Options::=\"--force-confold\"\r\n \r\nRUN DEBIAN_FRONTEND=noninteractive apt-get install -y apt-transport-https lsb-release ca-certificates\r\n \r\nRUN DEBIAN_FRONTEND=noninteractive apt-get install -y curl wget dnsutils netcat net-tools vim socat telnet\r\n \r\n####\r\n \r\n# Download certificate and key from the customer portal (https://cs.nginx.com)\r\n# and copy to the build context\r\nCOPY nginx-repo.crt /etc/ssl/nginx/\r\nCOPY nginx-repo.key /etc/ssl/nginx/\r\n \r\n# Make sure the certificate and key have correct permissions\r\nRUN chmod 644 /etc/ssl/nginx/*\r\n \r\n# Install NGINX Plus\r\nRUN set -x \\\r\n  && apt-get update \\\r\n  && apt-get install --no-install-recommends --no-install-suggests -y apt-transport-https ca-certificates gnupg1 \\\r\n  && \\\r\n  NGINX_GPGKEY=573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62; \\\r\n  found=''; \\\r\n  for server in \\\r\n    ha.pool.sks-keyservers.net \\\r\n    hkp://keyserver.ubuntu.com:80 \\\r\n    hkp://p80.pool.sks-keyservers.net:80 \\\r\n    pgp.mit.edu \\\r\n  ; do \\\r\n    echo \"Fetching GPG key $NGINX_GPGKEY from $server\"; \\\r\n    apt-key adv --keyserver \"$server\" --keyserver-options timeout=10 --recv-keys \"$NGINX_GPGKEY\" && found=yes && break; \\\r\n  done; \\\r\n  test -z \"$found\" && echo >&2 \"error: failed to fetch GPG key $NGINX_GPGKEY\" && exit 1; \\\r\n  echo \"Acquire::https::plus-pkgs.nginx.com::Verify-Peer \\\"true\\\";\" >> /etc/apt/apt.conf.d/90nginx \\\r\n  && echo \"Acquire::https::plus-pkgs.nginx.com::Verify-Host \\\"true\\\";\" >> /etc/apt/apt.conf.d/90nginx \\\r\n  && echo \"Acquire::https::plus-pkgs.nginx.com::SslCert     \\\"/etc/ssl/nginx/nginx-repo.crt\\\";\" >> /etc/apt/apt.conf.d/90nginx \\\r\n  && echo \"Acquire::https::plus-pkgs.nginx.com::SslKey      \\\"/etc/ssl/nginx/nginx-repo.key\\\";\" >> /etc/apt/apt.conf.d/90nginx \\\r\n  && printf \"deb https://plus-pkgs.nginx.com/debian stretch nginx-plus\\n\" > /etc/apt/sources.list.d/nginx-plus.list \\\r\n  && apt-get update && apt-get install -y nginx-plus=${NGINX_PLUS_VERSION} \\\r\n  && apt-get remove --purge --auto-remove -y gnupg1 \\\r\n  && rm -rf /var/lib/apt/lists/* \\\r\n  && rm -rf /etc/ssl/nginx \\\r\n  && rm /etc/apt/apt.conf.d/90nginx /etc/apt/sources.list.d/nginx-plus.list\r\n \r\n \r\n# forward nginx access and error logs to stdout and stderr of the ingress\r\n# controller process\r\nRUN ln -sf /proc/1/fd/1 /var/log/nginx/access.log \\\r\n                && ln -sf /proc/1/fd/2 /var/log/nginx/error.log\r\n \r\n \r\nEXPOSE 80 443\r\n \r\nCOPY nginx-ingress nginx/templates/nginx-plus.ingress.tmpl nginx/templates/nginx-plus.tmpl /\r\n \r\nRUN rm /etc/nginx/conf.d/* \\\r\n  && mkdir -p /etc/nginx/secrets\r\n \r\n# Uncomment the line below if you would like to add the default.pem to the image\r\n# and use it as a certificate and key for the default server\r\n# ADD default.pem /etc/nginx/secrets/default\r\n \r\nENTRYPOINT [\"/nginx-ingress\"]\r\n```\r\n\r\n**Expected Behavior**\r\nThe ingress controller should start and not go into a \"CrashLoopBackoff\"\r\n\r\n**Environment**\r\n* Version of Ingress Controller: 1.3.0 (open source and plus)\r\n* Version of Kubernetes: 1.9.4\r\n* Platform: Bare Metal\r\n* NGINX Plus\r\n\r\n**Additional Context**\r\nOpen Source Log: \r\n```\r\nkubectl -n default logs -f httpcontroller-reg-2g82n\r\nI0827 13:33:13.653094       1 main.go:92] Starting NGINX Ingress controller Version=1.3.0 GitCommit=b8acd634\r\nF0827 13:33:23.703371       1 main.go:147] Error when getting default/default-server-secret: Get https://10.255.0.1:443/api/v1/namespaces/default/secrets/default-server-secret: net/http: TLS handshake timeout\r\n```\r\nPlus Log:\r\n```\r\nI0814 16:42:00.385735       1 main.go:75] Starting NGINX Ingress controller Version=0.0.1 GitCommit=1144d00\r\nF0814 16:42:10.545015       1 main.go:118] Error when getting default/default-server-secret: Get https://10.255.0.1:443/api/v1/namespaces/default/secrets/default-server-secret: net/http: TLS handshake timeout\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/comments/416511326","html_url":"https://github.com/nginxinc/kubernetes-ingress/issues/350#issuecomment-416511326","issue_url":"https://api.github.com/repos/nginxinc/kubernetes-ingress/issues/350","id":416511326,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjUxMTMyNg==","user":{"login":"isaachawley","id":29185,"node_id":"MDQ6VXNlcjI5MTg1","avatar_url":"https://avatars2.githubusercontent.com/u/29185?v=4","gravatar_id":"","url":"https://api.github.com/users/isaachawley","html_url":"https://github.com/isaachawley","followers_url":"https://api.github.com/users/isaachawley/followers","following_url":"https://api.github.com/users/isaachawley/following{/other_user}","gists_url":"https://api.github.com/users/isaachawley/gists{/gist_id}","starred_url":"https://api.github.com/users/isaachawley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isaachawley/subscriptions","organizations_url":"https://api.github.com/users/isaachawley/orgs","repos_url":"https://api.github.com/users/isaachawley/repos","events_url":"https://api.github.com/users/isaachawley/events{/privacy}","received_events_url":"https://api.github.com/users/isaachawley/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T09:15:20Z","updated_at":"2018-08-28T09:15:20Z","author_association":"CONTRIBUTOR","body":"I believe that error is coming from [this line in main.go](https://github.com/nginxinc/kubernetes-ingress/blob/master/nginx-controller/main.go#L160) which happens to be the first call the IC makes to the kubernetes API.\r\n\r\nThe IC needs to be able to connect to the kubernetes API in order to work.\r\n\r\nIt appears that you're running a custom install of kubernetes. My guess is that it's slightly misconfigured:\r\n- Ensure the Kubernetes service cluster CIDR [is correct](https://github.com/mesosphere/dcos-kubernetes-quickstart/issues/58#issuecomment-362549761)\r\n- Ensure the [service account is setup properly](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod)\r\n- Check [this issue](https://github.com/kubernetes/kubernetes/issues/22351#issuecomment-257930764) maybe the etcd token has become invalid.\r\n\r\nAnyway this issue is happening because the IC can't access the K8s API - ensure that the API is accessible from the pod and the problem should be solved. "}}
{"actor":{"display_login":"isaachawley"}}