{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-cli/issues/2288","repository_url":"https://api.github.com/repos/aws/aws-cli","labels_url":"https://api.github.com/repos/aws/aws-cli/issues/2288/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-cli/issues/2288/comments","events_url":"https://api.github.com/repos/aws/aws-cli/issues/2288/events","html_url":"https://github.com/aws/aws-cli/issues/2288","id":190487448,"number":2288,"title":"UnicodeEncodeError when uploading document to Amazon CloudSearch","user":{"login":"iron59","id":4161244,"avatar_url":"https://avatars.githubusercontent.com/u/4161244?v=3","gravatar_id":"","url":"https://api.github.com/users/iron59","html_url":"https://github.com/iron59","followers_url":"https://api.github.com/users/iron59/followers","following_url":"https://api.github.com/users/iron59/following{/other_user}","gists_url":"https://api.github.com/users/iron59/gists{/gist_id}","starred_url":"https://api.github.com/users/iron59/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iron59/subscriptions","organizations_url":"https://api.github.com/users/iron59/orgs","repos_url":"https://api.github.com/users/iron59/repos","events_url":"https://api.github.com/users/iron59/events{/privacy}","received_events_url":"https://api.github.com/users/iron59/received_events","type":"User","site_admin":false},"labels":[{"id":297058759,"url":"https://api.github.com/repos/aws/aws-cli/labels/api-question","name":"api-question","color":"cc317c","default":false},{"id":19841172,"url":"https://api.github.com/repos/aws/aws-cli/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-19T09:47:06Z","updated_at":"2016-12-02T06:39:19Z","closed_at":null,"body":"Good day. I'm uploading documents to my search endpoint but one of my json files have a unicode character and it seems to be causing the CLI to throw a UnicodeEncodeError. Here's the debug log of the command:\r\n\r\n> $ aws cloudsearchdomain --endpoint-url http://doc-twitch-data-search-7qmgdcs43ob54lkmjtet7xnj54.us-west-2.cloudsearch.amazonaws.com upload-documents --content-type application/json --documents game_json_data12000.json --debug\r\n> 2016-11-19 09:40:06,095 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.19 Python/2.7.12 Linux/4.4.19-29.55.amzn1.x86_64 botocore/1.4.76\r\n> 2016-11-19 09:40:06,095 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['cloudsearchdomain', '--endpoint-url', 'http://doc-twitch-data-search-7qmgdcs43ob54lkmjtet7xnj54.us-west-2.cloudsearch.amazonaws.com', 'upload-documents', '--content-type', 'application/json', '--documents', 'game_json_data12000.json', '--debug']\r\n> 2016-11-19 09:40:06,096 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fc6afabc8c0>\r\n> 2016-11-19 09:40:06,096 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fc6b0376578>\r\n> 2016-11-19 09:40:06,096 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.\r\n> 2016-11-19 09:40:06,097 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/site-packages/botocore/data/cloudsearchdomain/2013-01-01/service-2.json\r\n> 2016-11-19 09:40:06,099 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.cloudsearchdomain: calling handler <function register_retries_for_service at 0x7fc6b0c611b8>\r\n> 2016-11-19 09:40:06,100 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: cloudsearchdomain\r\n> 2016-11-19 09:40:06,100 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cloudsearchdomain: calling handler <function add_waiters at 0x7fc6afac1c80>\r\n> 2016-11-19 09:40:06,103 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'documents', <awscli.arguments.CLIArgument object at 0x7fc6af7c5d10>), (u'content-type', <awscli.arguments.CLIArgument object at 0x7fc6af7c5d50>)])\r\n> 2016-11-19 09:40:06,103 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.cloudsearchdomain.upload-documents: calling handler <function add_streaming_output_arg at 0x7fc6afabced8>\r\n> 2016-11-19 09:40:06,103 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.cloudsearchdomain.upload-documents: calling handler <function add_cli_input_json at 0x7fc6b037c488>\r\n> 2016-11-19 09:40:06,104 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.cloudsearchdomain.upload-documents: calling handler <function unify_paging_params at 0x7fc6afb43aa0>\r\n> 2016-11-19 09:40:06,106 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.cloudsearchdomain.upload-documents: calling handler <function add_generate_skeleton at 0x7fc6afb36050>\r\n> 2016-11-19 09:40:06,106 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.cloudsearchdomain.upload-documents: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fc6af7c5d90>>\r\n> 2016-11-19 09:40:06,106 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.cloudsearchdomain.upload-documents: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc6af7c5e10>>\r\n> 2016-11-19 09:40:06,107 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.cloudsearchdomain.upload-documents.documents: calling handler <function uri_param at 0x7fc6b0411e60>\r\n> 2016-11-19 09:40:06,107 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.cloudsearchdomain.upload-documents: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc6b03d56d0>\r\n> 2016-11-19 09:40:06,107 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'game_json_data12000.json' for parameter \"documents\": <open file u'game_json_data12000.json', mode 'rb' at 0x7fc6b0430f60>\r\n> 2016-11-19 09:40:06,107 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.cloudsearchdomain.upload-documents.content-type: calling handler <function uri_param at 0x7fc6b0411e60>\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.cloudsearchdomain.upload-documents: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc6b03d56d0>\r\n> 2016-11-19 09:40:06,108 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'application/json' for parameter \"content_type\": u'application/json'\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.cloudsearchdomain.upload-documents.cli-input-json: calling handler <function uri_param at 0x7fc6b0411e60>\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.cloudsearchdomain.upload-documents.generate-cli-skeleton: calling handler <function uri_param at 0x7fc6b0411e60>\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event calling-command.cloudsearchdomain.upload-documents: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fc6af7c5e10>>\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event calling-command.cloudsearchdomain.upload-documents: calling handler <function validate_endpoint_url at 0x7fc6afbddd70>\r\n> 2016-11-19 09:40:06,108 - MainThread - botocore.hooks - DEBUG - Event calling-command.cloudsearchdomain.upload-documents: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fc6af7c5d90>>\r\n> 2016-11-19 09:40:06,109 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env\r\n> 2016-11-19 09:40:06,109 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role\r\n> 2016-11-19 09:40:06,109 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file\r\n> 2016-11-19 09:40:06,109 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials\r\n> 2016-11-19 09:40:06,110 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/site-packages/botocore/data/endpoints.json\r\n> 2016-11-19 09:40:06,121 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: cloudsearchdomain\r\n> 2016-11-19 09:40:06,121 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.cloudsearchdomain: calling handler <function add_generate_presigned_url at 0x7fc6b0cb1668>\r\n> 2016-11-19 09:40:06,122 - MainThread - botocore.regions - DEBUG - Creating a regex based endpoint for cloudsearchdomain, us-west-2\r\n> 2016-11-19 09:40:06,122 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None\r\n> 2016-11-19 09:40:06,125 - MainThread - botocore.endpoint - DEBUG - Setting cloudsearchdomain timeout as (60, 60)\r\n> 2016-11-19 09:40:06,125 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=UploadDocuments) (verify_ssl=True) with params: {'body': <open file u'game_json_data12000.json', mode 'rb' at 0x7fc6b0430f60>, 'url': u'http://doc-twitch-data-search-7qmgdcs43ob54lkmjtet7xnj54.us-west-2.cloudsearch.amazonaws.com/2013-01-01/documents/batch?format=sdk', 'headers': {u'Content-Type': 'application/json', 'User-Agent': 'aws-cli/1.11.19 Python/2.7.12 Linux/4.4.19-29.55.amzn1.x86_64 botocore/1.4.76'}, 'context': {'client_region': 'us-west-2', 'has_streaming_input': True, 'client_config': <botocore.config.Config object at 0x7fc6af6cffd0>}, 'query_string': {}, 'url_path': u'/2013-01-01/documents/batch?format=sdk', 'method': u'POST'}\r\n> 2016-11-19 09:40:06,126 - MainThread - botocore.hooks - DEBUG - Event request-created.cloudsearchdomain.UploadDocuments: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fc6af6cfed0>>\r\n> 2016-11-19 09:40:06,128 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.\r\n> 2016-11-19 09:40:06,128 - MainThread - botocore.auth - DEBUG - CanonicalRequest:\r\n> POST\r\n> /2013-01-01/documents/batch\r\n> format=sdk\r\n> content-type:application/json\r\n> host:doc-twitch-data-search-7qmgdcs43ob54lkmjtet7xnj54.us-west-2.cloudsearch.amazonaws.com\r\n> x-amz-date:20161119T094006Z\r\n> content-type;host;x-amz-date\r\n> 091c18fe39f0e77147b0db6d5ff15d449f3af7a602f011a4b737e031e35d7c91\r\n> 2016-11-19 09:40:06,128 - MainThread - botocore.auth - DEBUG - StringToSign:\r\n> AWS4-HMAC-SHA256\r\n> 20161119T094006Z\r\n> 20161119/us-west-2/cloudsearch/aws4_request\r\n> d3c709b467e346745444a4d483c81416c909231bd1ec4e75eedba5281decd0a6\r\n> 2016-11-19 09:40:06,128 - MainThread - botocore.auth - DEBUG - Signature:\r\n> 12f9a0960d6c0009af59dd2af930edd43113ec6a6de40dd74f5c1f1076ddadab\r\n> 2016-11-19 09:40:06,130 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>\r\n> 2016-11-19 09:40:06,131 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): doc-twitch-data-search-7qmgdcs43ob54lkmjtet7xnj54.us-west-2.cloudsearch.amazonaws.com\r\n> 2016-11-19 09:40:07,331 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - \"POST /2013-01-01/documents/batch?format=sdk HTTP/1.1\" 400 647\r\n> 2016-11-19 09:40:07,332 - MainThread - botocore.parsers - DEBUG - Response headers: {'date': 'Sat, 19 Nov 2016 09:40:06 GMT', 'x-amzn-requestid': '274a7328-ae3c-11e6-9888-a1ae03fdeb68', 'content-length': '647', 'content-type': 'application/json; charset=utf-8', 'connection': 'keep-alive'}\r\n> 2016-11-19 09:40:07,332 - MainThread - botocore.parsers - DEBUG - Response body:\r\n> {\"status\": \"error\", \"errors\": [{\"message\": \"[Deprecated: Use the outer message field] lexical error: invalid character inside string.\\n \\t\"name\": \"Pok\\u00e9mon Seek & Find\\t\", \\t\\t\\t\"giantbomb_id\": \"47767\"\\n (right here) ------^\\n (near operation with index 566; document_id 488105)\"}], \"adds\": 0, \"__type\": \"#DocumentServiceException\", \"message\": \"{ [\"lexical error: invalid character inside string.\\n \\t\"name\": \"Pok\\u00e9mon Seek & Find\\t\", \\t\\t\\t\"giantbomb_id\": \"47767\"\\n (right here) ------^\\n (near operation with index 566; document_id 488105)\"] }\", \"deletes\": 0}\r\n> 2016-11-19 09:40:07,332 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudsearchdomain.UploadDocuments: calling handler <botocore.retryhandler.RetryHandler object at 0x7fc6af7c5590>\r\n> 2016-11-19 09:40:07,332 - MainThread - botocore.retryhandler - DEBUG - No retry needed.\r\n> 2016-11-19 09:40:07,332 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()\r\n> Traceback (most recent call last):\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 186, in main\r\n> return command_table[parsed_args.command](remaining, parsed_args)\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 381, in call\r\n> return command_table[parsed_args.operation](remaining, parsed_globals)\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 551, in call\r\n> call_parameters, parsed_globals)\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 675, in invoke\r\n> **parameters)\r\n> File \"/usr/local/lib/python2.7/site-packages/botocore/client.py\", line 251, in _api_call\r\n> return self._make_api_call(operation_name, kwargs)\r\n> File \"/usr/local/lib/python2.7/site-packages/botocore/client.py\", line 537, in _make_api_call\r\n> raise ClientError(parsed_response, operation_name)\r\n> ClientError: An error occurred (DocumentServiceException) when calling the UploadDocuments operation: { [\"lexical error: invalid character inside string.\r\n> \"name\": \"Pok\\xe9mon Seek & Find \", \"giantbomb_id\": \"47767\"\r\n> (right here) ------^\r\n> (near operation with index 566; document_id 488105)\"] }\r\n> 2016-11-19 09:40:07,333 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255\r\n> \r\n> Traceback (most recent call last):\r\n> File \"/usr/bin/aws\", line 27, in \r\n> sys.exit(main())\r\n> File \"/usr/bin/aws\", line 23, in main\r\n> return awscli.clidriver.main()\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 50, in main\r\n> return driver.main()\r\n> File \"/usr/local/lib/python2.7/site-packages/awscli/clidriver.py\", line 212, in main\r\n> sys.stderr.write(\"%s\\n\" % e)\r\n> UnicodeEncodeError: 'ascii' codec can't encode character u'\\xe9' in position 164: ordinal not in range(128)\r\n\r\nFor what it's worth, here's my locale:\r\n\r\n> $ locale\r\nLANG=en_US.UTF-8\r\nLC_CTYPE=\"en_US.UTF-8\"\r\nLC_NUMERIC=\"en_US.UTF-8\"\r\nLC_TIME=\"en_US.UTF-8\"\r\nLC_COLLATE=\"en_US.UTF-8\"\r\nLC_MONETARY=\"en_US.UTF-8\"\r\nLC_MESSAGES=\"en_US.UTF-8\"\r\nLC_PAPER=\"en_US.UTF-8\"\r\nLC_NAME=\"en_US.UTF-8\"\r\nLC_ADDRESS=\"en_US.UTF-8\"\r\nLC_TELEPHONE=\"en_US.UTF-8\"\r\nLC_MEASUREMENT=\"en_US.UTF-8\"\r\nLC_IDENTIFICATION=\"en_US.UTF-8\"\r\nLC_ALL=\r\n\r\nAny suggestions on how to fix this? Thank you very much."},"comment":{"url":"https://api.github.com/repos/aws/aws-cli/issues/comments/264385021","html_url":"https://github.com/aws/aws-cli/issues/2288#issuecomment-264385021","issue_url":"https://api.github.com/repos/aws/aws-cli/issues/2288","id":264385021,"user":{"login":"iron59","id":4161244,"avatar_url":"https://avatars.githubusercontent.com/u/4161244?v=3","gravatar_id":"","url":"https://api.github.com/users/iron59","html_url":"https://github.com/iron59","followers_url":"https://api.github.com/users/iron59/followers","following_url":"https://api.github.com/users/iron59/following{/other_user}","gists_url":"https://api.github.com/users/iron59/gists{/gist_id}","starred_url":"https://api.github.com/users/iron59/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iron59/subscriptions","organizations_url":"https://api.github.com/users/iron59/orgs","repos_url":"https://api.github.com/users/iron59/repos","events_url":"https://api.github.com/users/iron59/events{/privacy}","received_events_url":"https://api.github.com/users/iron59/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T06:39:19Z","updated_at":"2016-12-02T06:39:19Z","body":"Any progress on this? Thank you. :)"}}
{"actor":{"display_login":"iron59"}}