{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/1602","repository_url":"https://api.github.com/repos/coreos/prometheus-operator","labels_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/1602/labels{/name}","comments_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/1602/comments","events_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/1602/events","html_url":"https://github.com/coreos/prometheus-operator/issues/1602","id":339881119,"node_id":"MDU6SXNzdWUzMzk4ODExMTk=","number":1602,"title":"update to v0.22.0: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.","user":{"login":"r0bj","id":1297747,"node_id":"MDQ6VXNlcjEyOTc3NDc=","avatar_url":"https://avatars0.githubusercontent.com/u/1297747?v=4","gravatar_id":"","url":"https://api.github.com/users/r0bj","html_url":"https://github.com/r0bj","followers_url":"https://api.github.com/users/r0bj/followers","following_url":"https://api.github.com/users/r0bj/following{/other_user}","gists_url":"https://api.github.com/users/r0bj/gists{/gist_id}","starred_url":"https://api.github.com/users/r0bj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/r0bj/subscriptions","organizations_url":"https://api.github.com/users/r0bj/orgs","repos_url":"https://api.github.com/users/r0bj/repos","events_url":"https://api.github.com/users/r0bj/events{/privacy}","received_events_url":"https://api.github.com/users/r0bj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-07-10T14:57:33Z","updated_at":"2018-07-12T11:19:05Z","closed_at":null,"author_association":"NONE","body":"**What did you do?**\r\n\r\nUpdate prometheus-operator version from 0.21.0 to 0.22.0.\r\n\r\n**What did you expect to see?**\r\n\r\nprometheus-operator works and doesn't log errors\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\nprometheus-operator log errors after update:\r\n```\r\nts=2018-07-10T14:28:52.946172806Z caller=main.go:115 msg=\"Starting Prometheus Operator version '0.22.0'.\"\r\nlevel=info ts=2018-07-10T14:28:53.06458421Z caller=operator.go:293 component=prometheusoperator msg=\"connection established\" cluster-version=v1.10.5\r\nlevel=info ts=2018-07-10T14:28:53.065586765Z caller=operator.go:172 component=alertmanageroperator msg=\"connection established\" cluster-version=v1.10.5\r\nlevel=info ts=2018-07-10T14:28:53.651032595Z caller=operator.go:560 component=alertmanageroperator msg=\"CRD updated\" crd=Alertmanager\r\nlevel=info ts=2018-07-10T14:28:53.799728545Z caller=operator.go:1140 component=prometheusoperator msg=\"CRD updated\" crd=Prometheus\r\nlevel=info ts=2018-07-10T14:28:53.860727161Z caller=operator.go:1140 component=prometheusoperator msg=\"CRD updated\" crd=ServiceMonitor\r\nlevel=info ts=2018-07-10T14:28:53.881357556Z caller=operator.go:1140 component=prometheusoperator msg=\"CRD updated\" crd=PrometheusRule\r\nlevel=info ts=2018-07-10T14:28:56.671310977Z caller=operator.go:186 component=alertmanageroperator msg=\"CRD API endpoints ready\"\r\nlevel=info ts=2018-07-10T14:28:56.749566071Z caller=operator.go:396 component=alertmanageroperator msg=\"sync alertmanager\" key=monitoring/main\r\nE0710 14:28:56.846066       1 operator.go:272] Sync \"monitoring/main\" failed: updating statefulset failed: StatefulSet.apps \"alertmanager-main\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.\r\nlevel=info ts=2018-07-10T14:28:56.948586107Z caller=operator.go:396 component=alertmanageroperator msg=\"sync alertmanager\" key=monitoring/main\r\nE0710 14:28:56.971821       1 operator.go:272] Sync \"monitoring/main\" failed: updating statefulset failed: StatefulSet.apps \"alertmanager-main\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.\r\nlevel=info ts=2018-07-10T14:28:56.971895007Z caller=operator.go:396 component=alertmanageroperator msg=\"sync alertmanager\" key=monitoring/main\r\nE0710 14:28:56.985815       1 operator.go:272] Sync \"monitoring/main\" failed: updating statefulset failed: StatefulSet.apps \"alertmanager-main\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.\r\n```\r\n[...]\r\n```\r\nE0710 14:29:04.774183       1 operator.go:646] Sync \"monitoring/k8s\" failed: updating statefulset failed: StatefulSet.apps \"prometheus-k8s\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.\r\n```\r\n\r\n**Environment**\r\n\r\n* Kubernetes version information:\r\n\r\n```\r\n# kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.5\", GitCommit:\"32ac1c9073b132b8ba18aa830f46b77dcceb0723\", GitTreeState:\"clean\", BuildDate:\"2018-06-21T11:46:00Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.5\", GitCommit:\"32ac1c9073b132b8ba18aa830f46b77dcceb0723\", GitTreeState:\"clean\", BuildDate:\"2018-06-21T11:34:22Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n* Kubernetes cluster kind: \r\n\r\nK8s cluster build in-house.\r\n\r\n* Manifests:\r\n\r\n```\r\napiVersion: apps/v1\r\nkind: Deployment\r\nmetadata:\r\n  labels:\r\n    k8s-app: prometheus-operator\r\n  name: prometheus-operator\r\n  namespace: monitoring\r\nspec:\r\n  replicas: 1\r\n  selector:\r\n    matchLabels:\r\n      k8s-app: prometheus-operator\r\n  template:\r\n    metadata:\r\n      labels:\r\n        k8s-app: prometheus-operator\r\n    spec:\r\n      serviceAccountName: prometheus-operator\r\n      securityContext:\r\n        runAsNonRoot: true\r\n        runAsUser: 65534\r\n      containers:\r\n      - name: prometheus-operator\r\n        image: quay.io/coreos/prometheus-operator:v0.21.0\r\n        args:\r\n        - --kubelet-service=kube-system/kubelet\r\n        - --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1\r\n        - --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.21.0\r\n        ports:\r\n        - containerPort: 8080\r\n          name: http\r\n        resources:\r\n          limits:\r\n            cpu: 200m\r\n            memory: 100Mi\r\n          requests:\r\n            cpu: 100m\r\n            memory: 50Mi\r\n```\r\n```\r\napiVersion: monitoring.coreos.com/v1\r\nkind: Alertmanager\r\nmetadata:\r\n  name: main\r\n  labels:\r\n    alertmanager: main\r\n  namespace: monitoring\r\nspec:\r\n  replicas: 3\r\n  version: v0.15.0\r\n  externalUrl: http://alertmanager.example.com\r\n  storage:\r\n    volumeClaimTemplate:\r\n      metadata:\r\n        name: alertmanager-data\r\n      spec:\r\n        accessModes:\r\n        - ReadWriteOnce\r\n        resources:\r\n          requests:\r\n            storage: 100Mi\r\n```\r\n```\r\napiVersion: monitoring.coreos.com/v1\r\nkind: Prometheus\r\nmetadata:\r\n  name: k8s\r\n  labels:\r\n    app: prometheus\r\n    prometheus: k8s\r\n  namespace: monitoring\r\nspec:\r\n  replicas: 2\r\n  version: v2.3.1\r\n  externalUrl: http://prometheus.example.com\r\n  externalLabels:\r\n    prometheus_cluster: prometheus-k8s\r\n  serviceAccountName: prometheus-k8s\r\n  serviceMonitorSelector:\r\n    matchExpressions:\r\n    - key: k8s-app\r\n      operator: Exists\r\n  ruleSelector:\r\n    matchLabels:\r\n      role: alert-rules\r\n      prometheus: k8s\r\n  resources:\r\n    limits:\r\n      memory: 12Gi\r\n    requests:\r\n      memory: 12Gi\r\n  alerting:\r\n    alertmanagers:\r\n    - namespace: monitoring\r\n      name: alertmanager-main\r\n      port: web\r\n  secrets:\r\n  - etcd-certs\r\n  affinity:\r\n    podAntiAffinity:\r\n      requiredDuringSchedulingIgnoredDuringExecution:\r\n      - labelSelector:\r\n          matchExpressions:\r\n          - key: prometheus\r\n            operator: In\r\n            values:\r\n            - k8s\r\n        topologyKey: kubernetes.io/hostname\r\n  retention: 3d\r\n  storage:\r\n    volumeClaimTemplate:\r\n      metadata:\r\n        name: prometheus-k8s-data\r\n      spec:\r\n        storageClassName: local-disks\r\n        accessModes:\r\n        - ReadWriteOnce\r\n        resources:\r\n          requests:\r\n            storage: 5Gi\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/comments/404478727","html_url":"https://github.com/coreos/prometheus-operator/issues/1602#issuecomment-404478727","issue_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/1602","id":404478727,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDQ3ODcyNw==","user":{"login":"r0bj","id":1297747,"node_id":"MDQ6VXNlcjEyOTc3NDc=","avatar_url":"https://avatars0.githubusercontent.com/u/1297747?v=4","gravatar_id":"","url":"https://api.github.com/users/r0bj","html_url":"https://github.com/r0bj","followers_url":"https://api.github.com/users/r0bj/followers","following_url":"https://api.github.com/users/r0bj/following{/other_user}","gists_url":"https://api.github.com/users/r0bj/gists{/gist_id}","starred_url":"https://api.github.com/users/r0bj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/r0bj/subscriptions","organizations_url":"https://api.github.com/users/r0bj/orgs","repos_url":"https://api.github.com/users/r0bj/repos","events_url":"https://api.github.com/users/r0bj/events{/privacy}","received_events_url":"https://api.github.com/users/r0bj/received_events","type":"User","site_admin":false},"created_at":"2018-07-12T11:19:05Z","updated_at":"2018-07-12T11:19:05Z","author_association":"NONE","body":"Thanks @rmartinez3, after removing `metadata: name: prometheus-k8s-data` section, prometheus-operator upgrade works as expected."}}
{"actor":{"display_login":"r0bj"}}