{"action":"created","issue":{"url":"https://api.github.com/repos/containous/traefik/issues/905","repository_url":"https://api.github.com/repos/containous/traefik","labels_url":"https://api.github.com/repos/containous/traefik/issues/905/labels{/name}","comments_url":"https://api.github.com/repos/containous/traefik/issues/905/comments","events_url":"https://api.github.com/repos/containous/traefik/issues/905/events","html_url":"https://github.com/containous/traefik/issues/905","id":191871192,"number":905,"title":"[ACME] Too many currently pending authorizations","user":{"login":"Mika56","id":600347,"avatar_url":"https://avatars.githubusercontent.com/u/600347?v=3","gravatar_id":"","url":"https://api.github.com/users/Mika56","html_url":"https://github.com/Mika56","followers_url":"https://api.github.com/users/Mika56/followers","following_url":"https://api.github.com/users/Mika56/following{/other_user}","gists_url":"https://api.github.com/users/Mika56/gists{/gist_id}","starred_url":"https://api.github.com/users/Mika56/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mika56/subscriptions","organizations_url":"https://api.github.com/users/Mika56/orgs","repos_url":"https://api.github.com/users/Mika56/repos","events_url":"https://api.github.com/users/Mika56/events{/privacy}","received_events_url":"https://api.github.com/users/Mika56/received_events","type":"User","site_admin":false},"labels":[{"id":291414578,"url":"https://api.github.com/repos/containous/traefik/labels/investigation-needed","name":"investigation-needed","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/containous/traefik/milestones/2","html_url":"https://github.com/containous/traefik/milestones/1.1","labels_url":"https://api.github.com/repos/containous/traefik/milestones/2/labels","id":1803140,"number":2,"title":"1.1","description":"- Add official constraint support — documented, and for `docker`, `consul`, etc… providers\r\n- Add a flag to disable default exposed service using `docker` provider\r\n- More metrics endpoint(s)\r\n- Enable K/V store for traefik configuration","creator":{"login":"emilevauge","id":6207234,"avatar_url":"https://avatars.githubusercontent.com/u/6207234?v=3","gravatar_id":"","url":"https://api.github.com/users/emilevauge","html_url":"https://github.com/emilevauge","followers_url":"https://api.github.com/users/emilevauge/followers","following_url":"https://api.github.com/users/emilevauge/following{/other_user}","gists_url":"https://api.github.com/users/emilevauge/gists{/gist_id}","starred_url":"https://api.github.com/users/emilevauge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emilevauge/subscriptions","organizations_url":"https://api.github.com/users/emilevauge/orgs","repos_url":"https://api.github.com/users/emilevauge/repos","events_url":"https://api.github.com/users/emilevauge/events{/privacy}","received_events_url":"https://api.github.com/users/emilevauge/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":80,"state":"open","created_at":"2016-06-01T19:41:13Z","updated_at":"2016-12-02T15:04:54Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2016-11-27T16:46:38Z","updated_at":"2016-12-02T15:05:01Z","closed_at":null,"body":"Hi,\r\n\r\nI'm migrating my infrastructure to Docker. I was running an Apache server on one virtual machine, with only a few websites protected with TLS.\r\nI'm now running Traefik in a Docker Swarm cluster composed of three hosts, and want to go full TLS (Let's Encrypt).\r\n\r\nAt first it worked correctly, but now it seems to be behaving oddly.\r\nMy Traefik gets its configuration from Consul, and stores the ACME certificates there too. `acme/onDemand` used to be set to `true`, but I've now set it to `false`.\r\nEven with `onDemand` set to `false`, my log file keeps growing, with these messages:\r\n\r\n```\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"map[portus.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"Error getting ACME certificates [portus.something1.com] : Cannot obtain certificates map[portus.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]+v\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"map[jira.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. jira.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"Error getting ACME certificates [jira.something1.com jira.something2.com] : Cannot obtain certificates map[jira.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. jira.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]+v\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"map[confluence.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. confluence.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"Error getting ACME certificates [confluence.something1.com confluence.something2.com] : Cannot obtain certificates map[confluence.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. confluence.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]+v\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"map[stash.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. stash.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. bitbucket.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. bitbucket.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"Error getting ACME certificates [stash.something1.com bitbucket.something1.com stash.something2.com bitbucket.something2.com] : Cannot obtain certificates map[stash.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. bitbucket.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. bitbucket.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. stash.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]+v\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"map[stats.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. stats.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]\"\r\ntime=\"2016-11-27T16:37:40Z\" level=error msg=\"Error getting ACME certificates [stats.something2.com stats.something1.com] : Cannot obtain certificates map[stats.something1.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations. stats.something2.com:acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.]+v\"\r\n```\r\n\r\nSo, obviously, Traefik is trying to get new certificates, but too much authorizations are pending.\r\nI've opened Consul and read my `acme/account/object` value, which is where things gets odd:\r\n\r\n```json\r\n{\r\n    \"Email\":\"...\",\r\n    \"Registration\":{\r\n        \"body\":{\r\n            \"resource\":\"reg\",\r\n            \"id\":0000,\r\n            \"key\":{\r\n            \"kty\":\"RSA\",\r\n            \"n\":\"...\",\r\n            \"e\":\"AQAB\"\r\n            },\r\n            \"contact\":[\r\n                \"mailto:...\"\r\n            ],\r\n            \"agreement\":\"https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf\"\r\n        },\r\n        \"uri\":\"https://acme-v01.api.letsencrypt.org/acme/reg/0000\",\r\n        \"new_authzr_uri\":\"https://acme-v01.api.letsencrypt.org/acme/new-authz\",\r\n        \"terms_of_service\":\"https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf\"\r\n    },\r\n    \"PrivateKey\":\"...\",\r\n    \"DomainsCertificate\":{\r\n        \"Certs\":[\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stash.something2.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stash.something2.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stash.something2.com\",\r\n                    \"SANs\":[\r\n                        \"bitbucket.something2.com\"\r\n                    ]\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stash.something2.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"confluence.something2.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"confluence.something2.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"jira.something2.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"jira.something2.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"confluence.something2.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"confluence.something2.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"prtg.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"prtg.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"prtg.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"prtg.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"prtg.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"prtg.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"prtg.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"prtg.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stats.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stats.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stats.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stats.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stats.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stats.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"stats.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"stats.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"glados.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"glados.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"glados.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"glados.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"glados.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"glados.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            },\r\n            {\r\n                \"Domains\":{\r\n                    \"Main\":\"glados.something1.com\",\r\n                    \"SANs\":null\r\n                },\r\n                \"Certificate\":{\r\n                    \"Domain\":\"glados.something1.com\",\r\n                    \"CertURL\":\"https://acme-v01.api.letsencrypt.org/acme/cert/...\",\r\n                    \"CertStableURL\":\"\",\r\n                    \"PrivateKey\":\"...\",\r\n                    \"Certificate\":\"...\"\r\n                }\r\n            }\r\n        ]\r\n    },\r\n    \"ChallengeCerts\":{\r\n        \"bitbucket.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"confluence.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"jira.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"prtg.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"stash.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"stash.something1.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        },\r\n        \"stats.something2.com\":{\r\n            \"Certificate\":\"...\",\r\n            \"PrivateKey\":\"...\"\r\n        }\r\n    }\r\n}\r\n```\r\n\r\nIt seems to me that I have multiple times the same certificates under `DomainsCertificate`.\r\nHowever, some of the domains that are multiple times works, while some others don't.\r\n\r\nI've also extracted one of the non-working certificate, and analysed it. It seems to me everything should work properly?\r\n\r\n```\r\n➜  traefik_tls openssl x509 -in stats.cert -text -noout\r\nCertificate:\r\n    Data:\r\n        Version: 3 (0x2)\r\n        Serial Number:\r\n            (...)\r\n    Signature Algorithm: sha256WithRSAEncryption\r\n        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3\r\n        Validity\r\n            Not Before: Nov 23 10:33:00 2016 GMT\r\n            Not After : Feb 21 10:33:00 2017 GMT\r\n        Subject: CN=stats.something1.com\r\n        Subject Public Key Info:\r\n            Public Key Algorithm: rsaEncryption\r\n                Public-Key: (4096 bit)\r\n                Modulus:\r\n                Exponent: 65537 (0x10001)\r\n        X509v3 extensions:\r\n            X509v3 Key Usage: critical\r\n                Digital Signature, Key Encipherment\r\n            X509v3 Extended Key Usage:\r\n                TLS Web Server Authentication, TLS Web Client Authentication\r\n            X509v3 Basic Constraints: critical\r\n                CA:FALSE\r\n            X509v3 Subject Key Identifier:\r\n                75:D9:8C:ED:71:C5:D5:A1:93:29:A2:C3:A4:9A:AA:27:85:BA:68:0C\r\n            X509v3 Authority Key Identifier:\r\n                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\r\n\r\n            Authority Information Access:\r\n                OCSP - URI:http://ocsp.int-x3.letsencrypt.org/\r\n                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/\r\n\r\n            X509v3 Subject Alternative Name:\r\n                DNS:stats.something1.com\r\n            X509v3 Certificate Policies:\r\n                Policy: 2.23.140.1.2.1\r\n                Policy: 1.3.6.1.4.1.44947.1.1.1\r\n                  CPS: http://cps.letsencrypt.org\r\n                  User Notice:\r\n                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\r\n\r\n    Signature Algorithm: sha256WithRSAEncryption\r\n         (...)\r\n```\r\n\r\nSo, my questions are:\r\n* How can I stop Traefik from requesting certificates against Let's Encrypt? From what I read, I need to stop sending authorizations requests for a week before I can send requests again\r\n* Is it normal that I have multiple times the same certs under `DomainsCertificate` (at least the same name, I haven't compared the certificates themselves)?\r\n* Can you add additional checks so Traefik does not try to generate certificates every minutes when it is rate limited?\r\n* Do I need to clean my `acme/account/object` value to remove incorrect certificates?"},"comment":{"url":"https://api.github.com/repos/containous/traefik/issues/comments/264474413","html_url":"https://github.com/containous/traefik/issues/905#issuecomment-264474413","issue_url":"https://api.github.com/repos/containous/traefik/issues/905","id":264474413,"user":{"login":"emilevauge","id":6207234,"avatar_url":"https://avatars.githubusercontent.com/u/6207234?v=3","gravatar_id":"","url":"https://api.github.com/users/emilevauge","html_url":"https://github.com/emilevauge","followers_url":"https://api.github.com/users/emilevauge/followers","following_url":"https://api.github.com/users/emilevauge/following{/other_user}","gists_url":"https://api.github.com/users/emilevauge/gists{/gist_id}","starred_url":"https://api.github.com/users/emilevauge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emilevauge/subscriptions","organizations_url":"https://api.github.com/users/emilevauge/orgs","repos_url":"https://api.github.com/users/emilevauge/repos","events_url":"https://api.github.com/users/emilevauge/events{/privacy}","received_events_url":"https://api.github.com/users/emilevauge/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T15:05:01Z","updated_at":"2016-12-02T15:05:01Z","body":"@Mika56 \r\nIt seems you are hitting multiple diffenret issues here.\r\n\r\n**1/ `JWS has invalid anti-replay nonce`**\r\nThis issue seems to be due to https://github.com/xenolf/lego/issues/318#issuecomment-263339163.\r\n\r\n**2/ Duplicate certificates**\r\nWere you using an older version of Traefik (v1.0.x) before?\r\nIt seems an older version were sometimes generating duplicate certificates.\r\nAs a workaround, I will clean up certificates list at startup in v1.1.2.\r\n"}}
{"actor":{"display_login":"emilevauge"}}