{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapyd/issues/165","repository_url":"https://api.github.com/repos/scrapy/scrapyd","labels_url":"https://api.github.com/repos/scrapy/scrapyd/issues/165/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapyd/issues/165/comments","events_url":"https://api.github.com/repos/scrapy/scrapyd/issues/165/events","html_url":"https://github.com/scrapy/scrapyd/pull/165","id":168071397,"number":165,"title":"PY3 Fix decoding of spider list","user":{"login":"redapple","id":886296,"avatar_url":"https://avatars.githubusercontent.com/u/886296?v=3","gravatar_id":"","url":"https://api.github.com/users/redapple","html_url":"https://github.com/redapple","followers_url":"https://api.github.com/users/redapple/followers","following_url":"https://api.github.com/users/redapple/following{/other_user}","gists_url":"https://api.github.com/users/redapple/gists{/gist_id}","starred_url":"https://api.github.com/users/redapple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redapple/subscriptions","organizations_url":"https://api.github.com/users/redapple/orgs","repos_url":"https://api.github.com/users/redapple/repos","events_url":"https://api.github.com/users/redapple/events{/privacy}","received_events_url":"https://api.github.com/users/redapple/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scrapy/scrapyd/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-07-28T11:14:31Z","updated_at":"2016-07-28T18:17:39Z","closed_at":"2016-07-28T18:04:40Z","pull_request":{"url":"https://api.github.com/repos/scrapy/scrapyd/pulls/165","html_url":"https://github.com/scrapy/scrapyd/pull/165","diff_url":"https://github.com/scrapy/scrapyd/pull/165.diff","patch_url":"https://github.com/scrapy/scrapyd/pull/165.patch"},"body":"In Python 3, interpreting output of subprocess call for `scrapyd runner list` (which calls `scrapy list`) as bytes fails JSON encoding tests, spider names being passed as bytes too:\r\n\r\n```\r\nUtilsCache.__setitem__(key=mybot, value={'': [b'spider1', b'spider2']})\r\n...\r\nTraceback (most recent call last):\r\n  File \"/home/paul/src/scrapyd/scrapyd/tests/test_utils.py\", line 66, in test_get_spider_list\r\n    spiders = get_spider_list('mybot', pythonpath=get_pythonpath_scrapyd())\r\n  File \"/home/paul/src/scrapyd/scrapyd/utils.py\", line 127, in get_spider_list\r\n    get_spider_list.cache[project] = project_cache\r\n  File \"/home/paul/src/scrapyd/scrapyd/utils.py\", line 52, in __setitem__\r\n    self.cache_manager[key] = value\r\n  File \"/home/paul/src/scrapyd/scrapyd/sqlite.py\", line 38, in __setitem__\r\n    key, value = self.encode(key), self.encode(value)\r\n  File \"/home/paul/src/scrapyd/scrapyd/sqlite.py\", line 97, in encode\r\n    return sqlite3.Binary(to_bytes(json.dumps(obj)))\r\n  File \"/usr/lib/python3.5/json/__init__.py\", line 230, in dumps\r\n    return _default_encoder.encode(obj)\r\n  File \"/usr/lib/python3.5/json/encoder.py\", line 198, in encode\r\n    chunks = self.iterencode(o, _one_shot=True)\r\n  File \"/usr/lib/python3.5/json/encoder.py\", line 256, in iterencode\r\n    return _iterencode(o, 0)\r\n  File \"/usr/lib/python3.5/json/encoder.py\", line 179, in default\r\n    raise TypeError(repr(o) + \" is not JSON serializable\")\r\nbuiltins.TypeError: b'spider1' is not JSON serializable\r\n\r\nscrapyd.tests.test_utils.GetSpiderListTest.test_get_spider_list\r\n```\r\n\r\nThis PR forces the [subprocess output](https://github.com/scrapy/scrapyd/blob/master/scrapyd/utils.py#L112) environement IO encoding to UTF-8 (using [`PYTHONIOENCODING`](https://docs.python.org/2/using/cmdline.html#envvar-PYTHONIOENCODING) in order to work with Unicode spider names.\r\nThis is needed for Python 2, otherwise, ASCII is used by the subprocess.\r\n"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapyd/issues/comments/235980113","html_url":"https://github.com/scrapy/scrapyd/pull/165#issuecomment-235980113","issue_url":"https://api.github.com/repos/scrapy/scrapyd/issues/165","id":235980113,"user":{"login":"redapple","id":886296,"avatar_url":"https://avatars.githubusercontent.com/u/886296?v=3","gravatar_id":"","url":"https://api.github.com/users/redapple","html_url":"https://github.com/redapple","followers_url":"https://api.github.com/users/redapple/followers","following_url":"https://api.github.com/users/redapple/following{/other_user}","gists_url":"https://api.github.com/users/redapple/gists{/gist_id}","starred_url":"https://api.github.com/users/redapple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redapple/subscriptions","organizations_url":"https://api.github.com/users/redapple/orgs","repos_url":"https://api.github.com/users/redapple/repos","events_url":"https://api.github.com/users/redapple/events{/privacy}","received_events_url":"https://api.github.com/users/redapple/received_events","type":"User","site_admin":false},"created_at":"2016-07-28T18:17:38Z","updated_at":"2016-07-28T18:17:38Z","body":"I think the coverage report failure are due to coverage==4.2 (on Travis)\r\nMy local virtualenv has 4.1 and works.\r\nI'll try a few things"}}
{"actor":{"display_login":"redapple"}}