{"action":"created","comment":{"url":"https://api.github.com/repos/pydata/pandas/pulls/comments/71813155","id":71813155,"diff_hunk":"@@ -1134,90 +1132,80 @@ cdef class TextReader:\n                              bint user_dtype,\n                              kh_str_t *na_hashset,\n                              object na_flist):\n-        if dtype[1] == 'i' or dtype[1] == 'u':\n-            result, na_count = _try_int64(self.parser, i, start, end,\n-                                          na_filter, na_hashset)\n+        if is_integer_dtype(dtype):\n+            result, na_count = _try_int64(self.parser, i, start, end, na_filter,\n+                                          na_hashset)\n             if user_dtype and na_count is not None:\n                 if na_count > 0:\n                     raise ValueError(\"Integer column has NA values in \"\n-                                    \"column {column}\".format(column=i))\n+                                     \"column {column}\".format(column=i))\n \n-            if result is not None and dtype[1:] != 'i8':\n+            if result is not None and dtype != 'int64':\n                 result = result.astype(dtype)\n \n             return result, na_count\n \n-        elif dtype[1] == 'f':\n+        elif is_float_dtype(dtype):\n             result, na_count = _try_double(self.parser, i, start, end,\n                                            na_filter, na_hashset, na_flist)\n \n-            if result is not None and dtype[1:] != 'f8':\n+            if result is not None and dtype != 'float64':\n                 result = result.astype(dtype)\n             return result, na_count\n \n-        elif dtype[1] == 'b':\n+        elif is_bool_dtype(dtype):\n             result, na_count = _try_bool_flex(self.parser, i, start, end,\n                                               na_filter, na_hashset,\n                                               self.true_set, self.false_set)\n             return result, na_count\n-        elif dtype[1] == 'c':\n-            raise NotImplementedError(\"the dtype %s is not supported for parsing\" % dtype)","path":"pandas/parser.pyx","position":161,"original_position":161,"commit_id":"822a42389ff387ac2603b069470c17f5480516ed","original_commit_id":"bd4e5155d2c4cff50994e24e4accf2348d443b2e","user":{"login":"chris-b1","id":1924092,"avatar_url":"https://avatars.githubusercontent.com/u/1924092?v=3","gravatar_id":"","url":"https://api.github.com/users/chris-b1","html_url":"https://github.com/chris-b1","followers_url":"https://api.github.com/users/chris-b1/followers","following_url":"https://api.github.com/users/chris-b1/following{/other_user}","gists_url":"https://api.github.com/users/chris-b1/gists{/gist_id}","starred_url":"https://api.github.com/users/chris-b1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chris-b1/subscriptions","organizations_url":"https://api.github.com/users/chris-b1/orgs","repos_url":"https://api.github.com/users/chris-b1/repos","events_url":"https://api.github.com/users/chris-b1/events{/privacy}","received_events_url":"https://api.github.com/users/chris-b1/received_events","type":"User","site_admin":false},"body":"This wasn't tested, so I switched it to `TypeError` and added a test.","created_at":"2016-07-22T00:56:02Z","updated_at":"2016-07-22T00:56:02Z","html_url":"https://github.com/pydata/pandas/pull/13406#discussion_r71813155","pull_request_url":"https://api.github.com/repos/pydata/pandas/pulls/13406","_links":{"self":{"href":"https://api.github.com/repos/pydata/pandas/pulls/comments/71813155"},"html":{"href":"https://github.com/pydata/pandas/pull/13406#discussion_r71813155"},"pull_request":{"href":"https://api.github.com/repos/pydata/pandas/pulls/13406"}}},"pull_request":{"url":"https://api.github.com/repos/pydata/pandas/pulls/13406","id":73137590,"html_url":"https://github.com/pydata/pandas/pull/13406","diff_url":"https://github.com/pydata/pandas/pull/13406.diff","patch_url":"https://github.com/pydata/pandas/pull/13406.patch","issue_url":"https://api.github.com/repos/pydata/pandas/issues/13406","number":13406,"state":"open","locked":false,"title":"ENH: parse categoricals in read_csv","user":{"login":"chris-b1","id":1924092,"avatar_url":"https://avatars.githubusercontent.com/u/1924092?v=3","gravatar_id":"","url":"https://api.github.com/users/chris-b1","html_url":"https://github.com/chris-b1","followers_url":"https://api.github.com/users/chris-b1/followers","following_url":"https://api.github.com/users/chris-b1/following{/other_user}","gists_url":"https://api.github.com/users/chris-b1/gists{/gist_id}","starred_url":"https://api.github.com/users/chris-b1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chris-b1/subscriptions","organizations_url":"https://api.github.com/users/chris-b1/orgs","repos_url":"https://api.github.com/users/chris-b1/repos","events_url":"https://api.github.com/users/chris-b1/events{/privacy}","received_events_url":"https://api.github.com/users/chris-b1/received_events","type":"User","site_admin":false},"body":"Closes #10153 (at least partly)\r\n\r\nAdds the ability to directly parse a `Categorical` through the `dtype` parameter to `read_csv`.  Currently just uses whatever is there as the categories, a possible enhancement would be to allow and enforce user-specified categories, through not quite sure what the api would be.\r\n\r\nThis only parses string categories - originally I had an impl that did type inference on the categories, but it added a lot of complication without much benefit, now the recommendation in the docs is to convert after parsing.\r\n\r\nHere's an example timing.  For reasonably sparse data, a slightly worse than 2x speedup is what I'm typically seeing, along with much better memory usage.\r\n\r\n```python\r\ngroup1 = ['aaaaa', 'bbbbb', 'cccccc', 'ddddddd', 'eeeeeeee']\r\n\r\ndf = pd.DataFrame({'a': np.random.choice(group1, 10000000).astype('object'),\r\n                   'b': np.random.choice(group1, 10000000).astype('object'),\r\n                   'c': np.random.choice(group1, 10000000).astype('object')})\r\ndf.to_csv('strings.csv', index=False)\r\n\r\n\r\nIn [14]: %timeit pd.read_csv('strings.csv').apply(pd.Categorical)\r\n1 loops, best of 3: 6.66 s per loop\r\n\r\nIn [13]: %timeit pd.read_csv('strings.csv', dtype='category')\r\n1 loop, best of 3: 3.68 s per loop\r\n```","created_at":"2016-06-08T23:31:14Z","updated_at":"2016-07-22T00:56:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a8136831fc94902af97cf40790b00aa309768af2","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/pydata/pandas/pulls/13406/commits","review_comments_url":"https://api.github.com/repos/pydata/pandas/pulls/13406/comments","review_comment_url":"https://api.github.com/repos/pydata/pandas/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13406/comments","statuses_url":"https://api.github.com/repos/pydata/pandas/statuses/822a42389ff387ac2603b069470c17f5480516ed","head":{"label":"chris-b1:categorical-parse","ref":"categorical-parse","sha":"822a42389ff387ac2603b069470c17f5480516ed","user":{"login":"chris-b1","id":1924092,"avatar_url":"https://avatars.githubusercontent.com/u/1924092?v=3","gravatar_id":"","url":"https://api.github.com/users/chris-b1","html_url":"https://github.com/chris-b1","followers_url":"https://api.github.com/users/chris-b1/followers","following_url":"https://api.github.com/users/chris-b1/following{/other_user}","gists_url":"https://api.github.com/users/chris-b1/gists{/gist_id}","starred_url":"https://api.github.com/users/chris-b1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chris-b1/subscriptions","organizations_url":"https://api.github.com/users/chris-b1/orgs","repos_url":"https://api.github.com/users/chris-b1/repos","events_url":"https://api.github.com/users/chris-b1/events{/privacy}","received_events_url":"https://api.github.com/users/chris-b1/received_events","type":"User","site_admin":false},"repo":{"id":21056537,"name":"pandas","full_name":"chris-b1/pandas","owner":{"login":"chris-b1","id":1924092,"avatar_url":"https://avatars.githubusercontent.com/u/1924092?v=3","gravatar_id":"","url":"https://api.github.com/users/chris-b1","html_url":"https://github.com/chris-b1","followers_url":"https://api.github.com/users/chris-b1/followers","following_url":"https://api.github.com/users/chris-b1/following{/other_user}","gists_url":"https://api.github.com/users/chris-b1/gists{/gist_id}","starred_url":"https://api.github.com/users/chris-b1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chris-b1/subscriptions","organizations_url":"https://api.github.com/users/chris-b1/orgs","repos_url":"https://api.github.com/users/chris-b1/repos","events_url":"https://api.github.com/users/chris-b1/events{/privacy}","received_events_url":"https://api.github.com/users/chris-b1/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/chris-b1/pandas","description":"Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more","fork":true,"url":"https://api.github.com/repos/chris-b1/pandas","forks_url":"https://api.github.com/repos/chris-b1/pandas/forks","keys_url":"https://api.github.com/repos/chris-b1/pandas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chris-b1/pandas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chris-b1/pandas/teams","hooks_url":"https://api.github.com/repos/chris-b1/pandas/hooks","issue_events_url":"https://api.github.com/repos/chris-b1/pandas/issues/events{/number}","events_url":"https://api.github.com/repos/chris-b1/pandas/events","assignees_url":"https://api.github.com/repos/chris-b1/pandas/assignees{/user}","branches_url":"https://api.github.com/repos/chris-b1/pandas/branches{/branch}","tags_url":"https://api.github.com/repos/chris-b1/pandas/tags","blobs_url":"https://api.github.com/repos/chris-b1/pandas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chris-b1/pandas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chris-b1/pandas/git/refs{/sha}","trees_url":"https://api.github.com/repos/chris-b1/pandas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chris-b1/pandas/statuses/{sha}","languages_url":"https://api.github.com/repos/chris-b1/pandas/languages","stargazers_url":"https://api.github.com/repos/chris-b1/pandas/stargazers","contributors_url":"https://api.github.com/repos/chris-b1/pandas/contributors","subscribers_url":"https://api.github.com/repos/chris-b1/pandas/subscribers","subscription_url":"https://api.github.com/repos/chris-b1/pandas/subscription","commits_url":"https://api.github.com/repos/chris-b1/pandas/commits{/sha}","git_commits_url":"https://api.github.com/repos/chris-b1/pandas/git/commits{/sha}","comments_url":"https://api.github.com/repos/chris-b1/pandas/comments{/number}","issue_comment_url":"https://api.github.com/repos/chris-b1/pandas/issues/comments{/number}","contents_url":"https://api.github.com/repos/chris-b1/pandas/contents/{+path}","compare_url":"https://api.github.com/repos/chris-b1/pandas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chris-b1/pandas/merges","archive_url":"https://api.github.com/repos/chris-b1/pandas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chris-b1/pandas/downloads","issues_url":"https://api.github.com/repos/chris-b1/pandas/issues{/number}","pulls_url":"https://api.github.com/repos/chris-b1/pandas/pulls{/number}","milestones_url":"https://api.github.com/repos/chris-b1/pandas/milestones{/number}","notifications_url":"https://api.github.com/repos/chris-b1/pandas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chris-b1/pandas/labels{/name}","releases_url":"https://api.github.com/repos/chris-b1/pandas/releases{/id}","deployments_url":"https://api.github.com/repos/chris-b1/pandas/deployments","created_at":"2014-06-21T00:24:42Z","updated_at":"2016-01-15T01:02:30Z","pushed_at":"2016-07-22T00:55:25Z","git_url":"git://github.com/chris-b1/pandas.git","ssh_url":"git@github.com:chris-b1/pandas.git","clone_url":"https://github.com/chris-b1/pandas.git","svn_url":"https://github.com/chris-b1/pandas","homepage":"http://pandas.pydata.org","size":82615,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"pydata:master","ref":"master","sha":"bb6b5e54edaf046389e8cce28e7cd27ee87f5fcc","user":{"login":"pydata","id":1284191,"avatar_url":"https://avatars.githubusercontent.com/u/1284191?v=3","gravatar_id":"","url":"https://api.github.com/users/pydata","html_url":"https://github.com/pydata","followers_url":"https://api.github.com/users/pydata/followers","following_url":"https://api.github.com/users/pydata/following{/other_user}","gists_url":"https://api.github.com/users/pydata/gists{/gist_id}","starred_url":"https://api.github.com/users/pydata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pydata/subscriptions","organizations_url":"https://api.github.com/users/pydata/orgs","repos_url":"https://api.github.com/users/pydata/repos","events_url":"https://api.github.com/users/pydata/events{/privacy}","received_events_url":"https://api.github.com/users/pydata/received_events","type":"Organization","site_admin":false},"repo":{"id":858127,"name":"pandas","full_name":"pydata/pandas","owner":{"login":"pydata","id":1284191,"avatar_url":"https://avatars.githubusercontent.com/u/1284191?v=3","gravatar_id":"","url":"https://api.github.com/users/pydata","html_url":"https://github.com/pydata","followers_url":"https://api.github.com/users/pydata/followers","following_url":"https://api.github.com/users/pydata/following{/other_user}","gists_url":"https://api.github.com/users/pydata/gists{/gist_id}","starred_url":"https://api.github.com/users/pydata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pydata/subscriptions","organizations_url":"https://api.github.com/users/pydata/orgs","repos_url":"https://api.github.com/users/pydata/repos","events_url":"https://api.github.com/users/pydata/events{/privacy}","received_events_url":"https://api.github.com/users/pydata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pydata/pandas","description":"Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more","fork":false,"url":"https://api.github.com/repos/pydata/pandas","forks_url":"https://api.github.com/repos/pydata/pandas/forks","keys_url":"https://api.github.com/repos/pydata/pandas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pydata/pandas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pydata/pandas/teams","hooks_url":"https://api.github.com/repos/pydata/pandas/hooks","issue_events_url":"https://api.github.com/repos/pydata/pandas/issues/events{/number}","events_url":"https://api.github.com/repos/pydata/pandas/events","assignees_url":"https://api.github.com/repos/pydata/pandas/assignees{/user}","branches_url":"https://api.github.com/repos/pydata/pandas/branches{/branch}","tags_url":"https://api.github.com/repos/pydata/pandas/tags","blobs_url":"https://api.github.com/repos/pydata/pandas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pydata/pandas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pydata/pandas/git/refs{/sha}","trees_url":"https://api.github.com/repos/pydata/pandas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pydata/pandas/statuses/{sha}","languages_url":"https://api.github.com/repos/pydata/pandas/languages","stargazers_url":"https://api.github.com/repos/pydata/pandas/stargazers","contributors_url":"https://api.github.com/repos/pydata/pandas/contributors","subscribers_url":"https://api.github.com/repos/pydata/pandas/subscribers","subscription_url":"https://api.github.com/repos/pydata/pandas/subscription","commits_url":"https://api.github.com/repos/pydata/pandas/commits{/sha}","git_commits_url":"https://api.github.com/repos/pydata/pandas/git/commits{/sha}","comments_url":"https://api.github.com/repos/pydata/pandas/comments{/number}","issue_comment_url":"https://api.github.com/repos/pydata/pandas/issues/comments{/number}","contents_url":"https://api.github.com/repos/pydata/pandas/contents/{+path}","compare_url":"https://api.github.com/repos/pydata/pandas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pydata/pandas/merges","archive_url":"https://api.github.com/repos/pydata/pandas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pydata/pandas/downloads","issues_url":"https://api.github.com/repos/pydata/pandas/issues{/number}","pulls_url":"https://api.github.com/repos/pydata/pandas/pulls{/number}","milestones_url":"https://api.github.com/repos/pydata/pandas/milestones{/number}","notifications_url":"https://api.github.com/repos/pydata/pandas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pydata/pandas/labels{/name}","releases_url":"https://api.github.com/repos/pydata/pandas/releases{/id}","deployments_url":"https://api.github.com/repos/pydata/pandas/deployments","created_at":"2010-08-24T01:37:33Z","updated_at":"2016-07-21T21:52:57Z","pushed_at":"2016-07-22T00:55:27Z","git_url":"git://github.com/pydata/pandas.git","ssh_url":"git@github.com:pydata/pandas.git","clone_url":"https://github.com/pydata/pandas.git","svn_url":"https://github.com/pydata/pandas","homepage":"http://pandas.pydata.org","size":153585,"stargazers_count":6689,"watchers_count":6689,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2766,"mirror_url":null,"open_issues_count":1814,"forks":2766,"open_issues":1814,"watchers":6689,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pydata/pandas/pulls/13406"},"html":{"href":"https://github.com/pydata/pandas/pull/13406"},"issue":{"href":"https://api.github.com/repos/pydata/pandas/issues/13406"},"comments":{"href":"https://api.github.com/repos/pydata/pandas/issues/13406/comments"},"review_comments":{"href":"https://api.github.com/repos/pydata/pandas/pulls/13406/comments"},"review_comment":{"href":"https://api.github.com/repos/pydata/pandas/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pydata/pandas/pulls/13406/commits"},"statuses":{"href":"https://api.github.com/repos/pydata/pandas/statuses/822a42389ff387ac2603b069470c17f5480516ed"}}}}
{"actor":{"display_login":"chris-b1"}}