{"action":"opened","issue":{"url":"https://api.github.com/repos/heptio/contour/issues/499","repository_url":"https://api.github.com/repos/heptio/contour","labels_url":"https://api.github.com/repos/heptio/contour/issues/499/labels{/name}","comments_url":"https://api.github.com/repos/heptio/contour/issues/499/comments","events_url":"https://api.github.com/repos/heptio/contour/issues/499/events","html_url":"https://github.com/heptio/contour/issues/499","id":337632522,"node_id":"MDU6SXNzdWUzMzc2MzI1MjI=","number":499,"title":"Observed container memory while adding|removing ingress","user":{"login":"mattalberts","id":6054865,"node_id":"MDQ6VXNlcjYwNTQ4NjU=","avatar_url":"https://avatars3.githubusercontent.com/u/6054865?v=4","gravatar_id":"","url":"https://api.github.com/users/mattalberts","html_url":"https://github.com/mattalberts","followers_url":"https://api.github.com/users/mattalberts/followers","following_url":"https://api.github.com/users/mattalberts/following{/other_user}","gists_url":"https://api.github.com/users/mattalberts/gists{/gist_id}","starred_url":"https://api.github.com/users/mattalberts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattalberts/subscriptions","organizations_url":"https://api.github.com/users/mattalberts/orgs","repos_url":"https://api.github.com/users/mattalberts/repos","events_url":"https://api.github.com/users/mattalberts/events{/privacy}","received_events_url":"https://api.github.com/users/mattalberts/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-02T19:17:51Z","updated_at":"2018-07-02T19:17:51Z","closed_at":null,"author_association":"NONE","body":"## Envoy Memory Investigation\r\nOdd container memory observations do not affect the contour container.  As seen in the graphs, contour memory changes are dwarfed by envoy memory changes.  I wanted to start the question/discussion with the contour project to see if similar observations have been seen. \r\n\r\nI found two possibly related issues in the envoy project\r\n  1. https://github.com/envoyproxy/envoy/issues/3592\r\n  1. https://github.com/envoyproxy/envoy/issues/3508\r\n\r\nThe likelihood of a relation is low, but I've included links to the issues for the sake of completion.\r\n\r\nI have yet to find the root cause, but it feels like handling around config (either a leak or purposeful check-point while merging changes).\r\n\r\n@davecheney \r\n - Any ideas?\r\n - Have you seen anything similar?\r\n - Would you rather I reach out directly to envoy?\r\n\r\n### Summary\r\nThe goal is to observe container memory while adding and removing ingress definitions.\r\n  * nothing other than the ingress definitions are changing.\r\n  * there is also no traffic to the ingresses.\r\n\r\n### Launch Context\r\nTo help eliminate possible causes for memory growth, both hot restart and envoy metrics scraping have been disabled.\r\n```yaml\r\n---\r\napiVersion: v1\r\nkind: ServiceAccount\r\nmetadata:\r\n  labels:\r\n    app: contour-ingress\r\n  name: contour-ingress\r\n  namespace: ingress-system\r\n---\r\napiVersion: v1\r\nkind: Service\r\nmetadata:\r\n  labels:\r\n    app: contour-ingress\r\n    component: perf\r\n  name: contour-ingress-perf\r\n  namespace: ingress-system\r\n  annotations:\r\n    # This annotation puts the AWS ELB into \"TCP\" mode so that it does not\r\n    # do HTTP negotiation for HTTPS connections at the ELB edge.\r\n    # The downside of this is the remote IP address of all connections will\r\n    # appear to be the internal address of the ELB. See docs/proxy-proto.md\r\n    # for information about enabling the PROXY protocol on the ELB to recover\r\n    # the original remote IP address.\r\n    #service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp\r\n    # https://github.com/heptio/contour/blob/v0.5.0/docs/annotations.md\r\n    #contour.heptio.com/max-connections: \"1024\"\r\n    #contour.heptio.com/max-pending-requests: \"1024\"\r\n    #contour.heptio.com/max-requests: \"1024\"\r\n    #contour.heptio.com/max-retries: \"1024\"\r\n    #contour.heptio.com/upstream-protocol.h2: \"80,443\"\r\nspec:\r\n  ports:\r\n    - port: 80\r\n      name: http\r\n      protocol: TCP\r\n      targetPort: 8080\r\n    - port: 443\r\n      name: https\r\n      protocol: TCP\r\n      targetPort: 8443\r\n  selector:\r\n    app: contour-ingress\r\n    component: perf\r\n  type: LoadBalancer\r\n---\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  labels:\r\n    app: contour-ingress\r\n    component: perf\r\n  name: contour-ingress-perf\r\n  namespace: ingress-system\r\nspec:\r\n  selector:\r\n    matchLabels:\r\n      app: contour-ingress\r\n      component: perf\r\n  replicas: 1\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: contour-ingress\r\n        component: perf\r\n      annotations:\r\n        prometheus.io/scrape: \"false\"\r\n        prometheus.io/port: \"9001\"\r\n        prometheus.io/path: \"/stats\"\r\n        prometheus.io/format: \"prometheus\"\r\n    spec:\r\n      dnsPolicy: ClusterFirst\r\n      serviceAccountName: contour-ingress\r\n      terminationGracePeriodSeconds: 30\r\n      initContainers:\r\n        - name: envoy-init\r\n          image: gcr.io/heptio-images/contour:v0.5.0\r\n          imagePullPolicy: IfNotPresent\r\n          command: [\"contour\"]\r\n          args:\r\n            - bootstrap\r\n            - /config/contour.yaml\r\n            - --admin-address=0.0.0.0\r\n          volumeMounts:\r\n            - name: contour-config\r\n              mountPath: /config\r\n\r\n      containers:\r\n        - name: contour\r\n          image: gcr.io/heptio-images/contour:v0.5.0\r\n          imagePullPolicy: IfNotPresent\r\n          command: [\"contour\"]\r\n          args:\r\n            - serve\r\n            - --incluster\r\n            - --log-level=warn\r\n            - --ingress-class-name=contour-perf\r\n            - --default-tls-secret=$(POD_NAMESPACE)/contour-ingress\r\n          env:\r\n            - name: POD_NAME\r\n              valueFrom:\r\n                fieldRef:\r\n                  fieldPath: metadata.name\r\n            - name: POD_NAMESPACE\r\n              valueFrom:\r\n                fieldRef:\r\n                  fieldPath: metadata.namespace\r\n        - name: envoy\r\n          # Select the debug image when inspecting envoy operations\r\n          # image: docker.io/envoyproxy/envoy-alpine-debug:v1.6.0\r\n          # image: docker.io/envoyproxy/envoy-alpine:v1.6.0\r\n          image: docker.io/envoyproxy/envoy-alpine@sha256:cb8d30a0aec15a55574b10825aca021b379f837296eae55dec3a4d0c006e1d3a\r\n          imagePullPolicy: IfNotPresent\r\n          ports:\r\n            - containerPort: 8080\r\n              name: http\r\n            - containerPort: 8443\r\n              name: https\r\n          command: [\"envoy\"]\r\n          args:\r\n            - -c /config/contour.yaml\r\n            - --service-cluster cluster0\r\n            - --service-node node0\r\n            - --disable-hot-restart\r\n            - --v2-config-only\r\n            # Set to info|debug when debugging\r\n            - --log-level warning\r\n          volumeMounts:\r\n            - name: contour-config\r\n              mountPath: /config\r\n\r\n      volumes:\r\n        - name: contour-config\r\n          emptyDir:\r\n            medium: \"Memory\"\r\n            sizeLimit: 8Mi\r\n      affinity:\r\n        podAntiAffinity:\r\n          preferredDuringSchedulingIgnoredDuringExecution:\r\n          - weight: 100\r\n            podAffinityTerm:\r\n              labelSelector:\r\n                matchLabels:\r\n                  app: contour-ingress\r\n                  component: perf\r\n              topologyKey: kubernetes.io/hostname\r\n---\r\n```\r\n\r\n### Observation ~ Adding Ingresses\r\nLet's observe container memory while adding ingress definitions.\r\n\r\nThough envoy specific metrics are currently unavailable (the scrape is disabled); container memory metrics were mirrored by envoy heap allocation prior to disabling.\r\n\r\n```bash\r\n>$ kubectl -n ingress-system top pods -l'app=contour-ingress,component=perf'\r\nNAME                                          CPU(cores)   MEMORY(bytes)   \r\ncontour-ingress-perf-64d76ff7ff-sv7z9         121m         94Mi            \r\n```\r\n<img width=\"1384\" alt=\"envoy_mem_state_a_pod_start_0_ing\" src=\"https://user-images.githubusercontent.com/6054865/42181884-4da622e0-7df1-11e8-8ff1-96a022601676.png\">\r\n\r\nThe tool ```clone-http-ingress.sh``` is bash script that adds ingress definitions in blocks\r\n - n - total number of ingress definitions\r\n - b - block size\r\n\r\nThe command below will add a total of 5000 definitions in blocks of 100.\r\n```bash\r\n>$ ./clone-http-ingress.sh -b 100 -n 5000\r\nREPORT\r\ntotal=5000\r\nblock_size=100\r\ndelay enabled=0 delay=0.000000\r\n\r\nBLOCK    MEMORY   DURATION\r\n00000000 91Mi     17.20000000\r\n00000001 91Mi     15.18000000\r\n00000002 91Mi     19.02000000\r\n00000003 1628Mi   21.68000000\r\n00000004 1628Mi   19.76000000\r\n00000005 1628Mi   20.33000000\r\n00000006 3550Mi   19.08000000\r\n00000007 3550Mi   23.22000000\r\n00000008 3550Mi   24.30000000\r\n00000009 6123Mi   25.92000000\r\n00000010 6123Mi   24.22000000\r\n00000011 8075Mi   28.61000000\r\n00000012 8075Mi   24.51000000\r\n00000013 10248Mi  24.16000000\r\n00000014 10248Mi  30.01000000\r\n00000015 12601Mi  27.30000000\r\n00000016 12601Mi  27.13000000\r\n00000017 12601Mi  29.85000000\r\n00000018 14741Mi  31.88000000\r\n00000019 16793Mi  32.22000000\r\n00000020 16793Mi  34.20000000\r\n00000021 18388Mi  34.93000000\r\n00000022 18388Mi  38.84000000\r\n00000023 20918Mi  40.79000000\r\n00000024 22845Mi  41.11000000\r\n00000025 22845Mi  40.71000000\r\n00000026 24758Mi  43.63000000\r\n00000027 27056Mi  45.06000000\r\n00000028 28994Mi  50.16000000\r\n00000029 30965Mi  46.90000000\r\n00000030 30965Mi  47.86000000\r\n00000031 33168Mi  47.39000000\r\n00000032 35386Mi  48.52000000\r\n00000033 37886Mi  51.76000000\r\n00000034 39765Mi  58.24000000\r\n00000035 42228Mi  56.12000000\r\n00000036 44600Mi  51.11000000\r\n00000037 46879Mi  54.84000000\r\n00000038 46879Mi  53.24000000\r\n00000039 49202Mi  51.93000000\r\n00000040 51236Mi  55.18000000\r\n00000041 53300Mi  53.77000000\r\n00000042 54895Mi  55.32000000\r\n00000043 56731Mi  57.56000000\r\n00000044 58954Mi  58.81000000\r\n00000045 60353Mi  61.28000000\r\n00000046 62116Mi  57.53000000\r\n00000047 63968Mi  61.20000000\r\n00000048 65922Mi  61.22000000\r\n00000049 67487Mi  61.11000000\r\n```\r\n> * envoy admin page is unresponsive (\"/\" or any \"/command\")\r\n> * when enabled, scraping fails (admin /stats page is also unresponsive)\r\n> * memory climbs linearly during addition\r\n> * memory never decreases, stuck at ~64Gi\r\n\r\n<img width=\"1387\" alt=\"envoy_mem_state_b_fill_with_n_ing\" src=\"https://user-images.githubusercontent.com/6054865/42181898-597da7dc-7df1-11e8-9b02-dbb906af5760.png\">\r\n\r\n\r\n```bash\r\n>$ POD_NAME=$(kubectl get pods -n ingress-system -l \"app=contour-ingress,component=perf\" -o jsonpath=\"{.items[0].metadata.name}\"); echo $POD_NAME\r\n>$ kubectl -n ingress-system delete pod $POD_NAME\r\n```\r\n\r\n```bash\r\n>$ kubectl -n ingress-system top pods -l'app=contour-ingress,component=perf'\r\nNAME                                    CPU(cores)   MEMORY(bytes)   \r\ncontour-ingress-perf-64d76ff7ff-kt5hk   284m         1391Mi \r\n```\r\n> * memory after reboot is <2Gi\r\n> * all 5000 ingress definitions still exist\r\n\r\n<img width=\"1392\" alt=\"envoy_mem_state_d_restart_no_change\" src=\"https://user-images.githubusercontent.com/6054865/42181929-6eb04362-7df1-11e8-9b10-e2ed0806d201.png\">\r\n\r\n\r\n### Observation ~ Deleting Ingresses\r\nContinuing from the conditions above, let's observe memory while cleaning up the cluster.\r\n\r\n```bash\r\n>$ kubectl -n ingress-system top pods -l'app=contour-ingress,component=perf'\r\nNAME                                    CPU(cores)   MEMORY(bytes)   \r\ncontour-ingress-perf-64d76ff7ff-kt5hk   259m         1228Mi  \r\n```\r\n> * all 5000 ingress definitions still exist\r\n\r\n```bash\r\n>$ kubectl delete ing --all\r\ningress.extensions \"httpbin-contour-0\" deleted\r\ningress.extensions \"httpbin-contour-1\" deleted\r\n...\r\n```\r\n> * the command takes a while, but eventually completes\r\n> * memory climbs linearly during deletion (~1Gi to ~8Gi)\r\n\r\n```bash\r\n>$ kubectl -n ingress-system top pods -l'app=contour-ingress,component=perf'\r\nNAME                                    CPU(cores)   MEMORY(bytes)   \r\ncontour-ingress-perf-64d76ff7ff-kt5hk   272m         7853Mi   \r\n```\r\n> * memory never decrease, stuck at ~8Gi\r\n\r\n<img width=\"1393\" alt=\"envoy_mem_state_e_delete_ing\" src=\"https://user-images.githubusercontent.com/6054865/42181940-77a20cda-7df1-11e8-9079-0ffe2b7b1930.png\">\r\n"}}
{"actor":{"display_login":"mattalberts"}}