{"action":"created","issue":{"url":"https://api.github.com/repos/scality/S3/issues/84","repository_url":"https://api.github.com/repos/scality/S3","labels_url":"https://api.github.com/repos/scality/S3/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/scality/S3/issues/84/comments","events_url":"https://api.github.com/repos/scality/S3/issues/84/events","html_url":"https://github.com/scality/S3/issues/84","id":162659693,"number":84,"title":"Specifying a port in s3cmd makes v4 fail","user":{"login":"DavidPineauScality","id":14309735,"avatar_url":"https://avatars.githubusercontent.com/u/14309735?v=3","gravatar_id":"","url":"https://api.github.com/users/DavidPineauScality","html_url":"https://github.com/DavidPineauScality","followers_url":"https://api.github.com/users/DavidPineauScality/followers","following_url":"https://api.github.com/users/DavidPineauScality/following{/other_user}","gists_url":"https://api.github.com/users/DavidPineauScality/gists{/gist_id}","starred_url":"https://api.github.com/users/DavidPineauScality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DavidPineauScality/subscriptions","organizations_url":"https://api.github.com/users/DavidPineauScality/orgs","repos_url":"https://api.github.com/users/DavidPineauScality/repos","events_url":"https://api.github.com/users/DavidPineauScality/events{/privacy}","received_events_url":"https://api.github.com/users/DavidPineauScality/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scality/S3/labels/question","name":"question","color":"cc317c"}],"state":"closed","locked":false,"assignee":{"login":"GiorgioRegni","id":109447,"avatar_url":"https://avatars.githubusercontent.com/u/109447?v=3","gravatar_id":"","url":"https://api.github.com/users/GiorgioRegni","html_url":"https://github.com/GiorgioRegni","followers_url":"https://api.github.com/users/GiorgioRegni/followers","following_url":"https://api.github.com/users/GiorgioRegni/following{/other_user}","gists_url":"https://api.github.com/users/GiorgioRegni/gists{/gist_id}","starred_url":"https://api.github.com/users/GiorgioRegni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GiorgioRegni/subscriptions","organizations_url":"https://api.github.com/users/GiorgioRegni/orgs","repos_url":"https://api.github.com/users/GiorgioRegni/repos","events_url":"https://api.github.com/users/GiorgioRegni/events{/privacy}","received_events_url":"https://api.github.com/users/GiorgioRegni/received_events","type":"User","site_admin":false},"assignees":[{"login":"GiorgioRegni","id":109447,"avatar_url":"https://avatars.githubusercontent.com/u/109447?v=3","gravatar_id":"","url":"https://api.github.com/users/GiorgioRegni","html_url":"https://github.com/GiorgioRegni","followers_url":"https://api.github.com/users/GiorgioRegni/followers","following_url":"https://api.github.com/users/GiorgioRegni/following{/other_user}","gists_url":"https://api.github.com/users/GiorgioRegni/gists{/gist_id}","starred_url":"https://api.github.com/users/GiorgioRegni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GiorgioRegni/subscriptions","organizations_url":"https://api.github.com/users/GiorgioRegni/orgs","repos_url":"https://api.github.com/users/GiorgioRegni/repos","events_url":"https://api.github.com/users/GiorgioRegni/events{/privacy}","received_events_url":"https://api.github.com/users/GiorgioRegni/received_events","type":"User","site_admin":false},{"login":"vrancurel","id":383894,"avatar_url":"https://avatars.githubusercontent.com/u/383894?v=3","gravatar_id":"","url":"https://api.github.com/users/vrancurel","html_url":"https://github.com/vrancurel","followers_url":"https://api.github.com/users/vrancurel/followers","following_url":"https://api.github.com/users/vrancurel/following{/other_user}","gists_url":"https://api.github.com/users/vrancurel/gists{/gist_id}","starred_url":"https://api.github.com/users/vrancurel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vrancurel/subscriptions","organizations_url":"https://api.github.com/users/vrancurel/orgs","repos_url":"https://api.github.com/users/vrancurel/repos","events_url":"https://api.github.com/users/vrancurel/events{/privacy}","received_events_url":"https://api.github.com/users/vrancurel/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2016-06-28T11:06:39Z","updated_at":"2016-07-07T16:23:44Z","closed_at":"2016-07-07T16:23:44Z","body":"Here is the s3cmd command and its output:\r\n```sh\r\n# s3cmd --debug -c s3cfg.mem-v4 mb s3://testbucket\r\nDEBUG: s3cmd version 1.6.1\r\nDEBUG: ConfigParser: Reading file 's3cfg.mem-v4'\r\nDEBUG: ConfigParser: access_key->ac...7_chars...1\r\nDEBUG: ConfigParser: secret_key->ve...11_chars...1\r\nDEBUG: ConfigParser: host_base->{{HOST}}:80\r\nDEBUG: ConfigParser: host_bucket->%(bucket)s.{{HOST}}:80\r\nDEBUG: ConfigParser: signature_v2->False\r\nDEBUG: ConfigParser: use_https->False\r\nDEBUG: ConfigParser: website_endpoint->http://192.168.1.101:80/answertoall\r\nDEBUG: Updating Config.Config cache_file -> \r\nDEBUG: Updating Config.Config follow_symlinks -> False\r\nDEBUG: Updating Config.Config verbosity -> 10\r\nDEBUG: Unicodising 'mb' using UTF-8\r\nDEBUG: Unicodising 's3://testbucket' using UTF-8\r\nDEBUG: Command: mb\r\nDEBUG: CreateRequest: resource[uri]=/\r\nDEBUG: Using signature v4\r\nDEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80\r\nDEBUG: canonical_headers = host:testbucket.{{HOST}}:80\r\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nx-amz-date:20160628T105414Z\r\n\r\nDEBUG: Canonical Request:\r\nPUT\r\n/\r\n\r\nhost:testbucket.{{HOST}}:80\r\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nx-amz-date:20160628T105414Z\r\n\r\nhost;x-amz-content-sha256;x-amz-date\r\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n----------------------\r\nDEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}\r\nDEBUG: Processing request, please wait...\r\nDEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80\r\nDEBUG: ConnMan.get(): creating new connection: http://testbucket.{{HOST}}:80\r\nDEBUG: non-proxied HTTPConnection(testbucket.{{HOST}}:80)\r\nDEBUG: format_uri(): /\r\nDEBUG: Sending request method_string='PUT', uri='/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}, body=(0 bytes)\r\nDEBUG: Response: {'status': 403, 'headers': {'x-amz-id-2': '781591391d87c1f6a3d3', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'x-amz-request-id': '781591391d87c1f6a3d3', 'date': 'Tue, 28 Jun 2016 10:54:15 GMT', 'content-type': 'application/xml'}, 'reason': 'Forbidden', 'data': '<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message><Resource></Resource><RequestId>781591391d87c1f6a3d3</RequestId></Error>'}\r\nDEBUG: ConnMan.put(): connection put back to pool (http://testbucket.{{HOST}}:80#1)\r\nDEBUG: S3Error: 403 (Forbidden)\r\nDEBUG: HttpHeader: x-amz-id-2: 781591391d87c1f6a3d3\r\nDEBUG: HttpHeader: server: AmazonS3\r\nDEBUG: HttpHeader: transfer-encoding: chunked\r\nDEBUG: HttpHeader: connection: keep-alive\r\nDEBUG: HttpHeader: x-amz-request-id: 781591391d87c1f6a3d3\r\nDEBUG: HttpHeader: date: Tue, 28 Jun 2016 10:54:15 GMT\r\nDEBUG: HttpHeader: content-type: application/xml\r\nDEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'\r\nDEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided.'\r\nDEBUG: ErrorXML: Resource: None\r\nDEBUG: ErrorXML: RequestId: '781591391d87c1f6a3d3'\r\nERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided.\r\n```\r\n\r\n\r\nAnd here is the log from our Kibana instance (First, the CanonicalRequest, and second the constructed stringToSign):\r\n```\r\n{\r\n  \"_index\": \"logstash-scality-2016.06.28\",\r\n  \"_type\": \"log\",\r\n  \"_id\": \"AVWWpUvFjIh-3to3jiRi\",\r\n  \"_score\": null,\r\n  \"_source\": {\r\n    \"name\": \"S3\",\r\n    \"bucketName\": \"testbucket\",\r\n    \"canonicalReqResult\": \"PUT\\n/\\n\\nhost:testbucket.{{HOST}}\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20160628T105414Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\r\n    \"req_id\": \"781591391d87c1f6a3d3\",\r\n    \"level\": \"debug\",\r\n    \"message\": \"constructed canonicalRequest\",\r\n    \"hostname\": \"connect05.scality.riakpoc.local\",\r\n    \"pid\": 85,\r\n    \"@version\": \"1\",\r\n    \"@timestamp\": \"2016-06-28T10:54:15.022Z\",\r\n    \"fields\": {},\r\n    \"beat\": {},\r\n    \"type\": \"log\",\r\n    \"count\": 1\r\n  },\r\n  \"fields\": {\r\n    \"@timestamp\": [\r\n      1467111255022\r\n    ]\r\n  },\r\n  \"highlight\": {\r\n    \"req_id\": [\r\n      \"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@\"\r\n    ],\r\n    \"req_id.raw\": [\r\n      \"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@\"\r\n    ]\r\n  },\r\n  \"sort\": [\r\n    1467111255022\r\n  ]\r\n}\r\n{\r\n  \"_index\": \"logstash-scality-2016.06.28\",\r\n  \"_type\": \"log\",\r\n  \"_id\": \"AVWWpUvFjIh-3to3jiRl\",\r\n  \"_score\": null,\r\n  \"_source\": {\r\n    \"name\": \"S3\",\r\n    \"bucketName\": \"testbucket\",\r\n    \"stringToSign\": \"AWS4-HMAC-SHA256\\n20160628T105414Z\\n20160628/US/s3/aws4_request\\n1dff0237362e0f8b069cba1748f99ccca9221cc54a6d37187a268071d38e65e4\",\r\n    \"req_id\": \"781591391d87c1f6a3d3\",\r\n    \"level\": \"trace\",\r\n    \"message\": \"constructed stringToSign\",\r\n    \"hostname\": \"connect05.scality.riakpoc.local\",\r\n    \"pid\": 85,\r\n    \"@version\": \"1\",\r\n    \"@timestamp\": \"2016-06-28T10:54:15.022Z\",\r\n    \"type\": \"log\",\r\n    \"count\": 1,\r\n    \"fields\": {},\r\n    \"beat\": {}\r\n  },\r\n  \"fields\": {\r\n    \"@timestamp\": [\r\n      1467111255022\r\n    ]\r\n  },\r\n  \"highlight\": {\r\n    \"req_id\": [\r\n      \"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@\"\r\n    ],\r\n    \"req_id.raw\": [\r\n      \"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@\"\r\n    ]\r\n  },\r\n  \"sort\": [\r\n    1467111255022\r\n  ]\r\n}\r\n```\r\n\r\n\r\nAs you can see, comparing the two canonical Requests:\r\n(from s3cmd)\r\n```\r\nPUT\r\n/\r\n\r\nhost:testbucket.{{HOST}}:80\r\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nx-amz-date:20160628T105414Z\r\n\r\nhost;x-amz-content-sha256;x-amz-date\r\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n```\r\n\r\n(from S3)\r\n```\r\nPUT\r\n/\r\n\r\nhost:testbucket.{{HOST}}\r\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\nx-amz-date:20160628T105414Z\r\n\r\nhost;x-amz-content-sha256;x-amz-date\r\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\r\n```\r\n\r\nWe can see that the only difference is that s3cmd includes the target's port into the canonical request, while the S3 server computes it without a port. In this setup here, we're using HAProxy to load-balance the requests only multiple S3 servers, so it makes sense to avoid using the port inside S3; but as s3cmd is actually using it, it may make the v4 auth fail for no real valid reason.\r\n\r\nI do not think it is for us to fix, but we should at least document it properly.\r\n\r\n@vrancurel @GiorgioRegni what is your take on this ?"},"comment":{"url":"https://api.github.com/repos/scality/S3/issues/comments/231131562","html_url":"https://github.com/scality/S3/issues/84#issuecomment-231131562","issue_url":"https://api.github.com/repos/scality/S3/issues/84","id":231131562,"user":{"login":"LaurenSpiegel","id":7368115,"avatar_url":"https://avatars.githubusercontent.com/u/7368115?v=3","gravatar_id":"","url":"https://api.github.com/users/LaurenSpiegel","html_url":"https://github.com/LaurenSpiegel","followers_url":"https://api.github.com/users/LaurenSpiegel/followers","following_url":"https://api.github.com/users/LaurenSpiegel/following{/other_user}","gists_url":"https://api.github.com/users/LaurenSpiegel/gists{/gist_id}","starred_url":"https://api.github.com/users/LaurenSpiegel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LaurenSpiegel/subscriptions","organizations_url":"https://api.github.com/users/LaurenSpiegel/orgs","repos_url":"https://api.github.com/users/LaurenSpiegel/repos","events_url":"https://api.github.com/users/LaurenSpiegel/events{/privacy}","received_events_url":"https://api.github.com/users/LaurenSpiegel/received_events","type":"User","site_admin":false},"created_at":"2016-07-07T16:23:44Z","updated_at":"2016-07-07T16:23:44Z","body":"Closing because this is a s3cmd issue.  Cannot specify port 80 with s3cmd or else the signature is wrong."}}
{"actor":{"display_login":"LaurenSpiegel"}}