{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/12646","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/12646/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/12646/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/12646/events","html_url":"https://github.com/pydata/pandas/issues/12646","id":141351869,"number":12646,"title":"add pandas.read_text for complex text/log parsing","user":{"login":"AbsoluteMSTR","id":537223,"avatar_url":"https://avatars.githubusercontent.com/u/537223?v=3","gravatar_id":"","url":"https://api.github.com/users/AbsoluteMSTR","html_url":"https://github.com/AbsoluteMSTR","followers_url":"https://api.github.com/users/AbsoluteMSTR/followers","following_url":"https://api.github.com/users/AbsoluteMSTR/following{/other_user}","gists_url":"https://api.github.com/users/AbsoluteMSTR/gists{/gist_id}","starred_url":"https://api.github.com/users/AbsoluteMSTR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbsoluteMSTR/subscriptions","organizations_url":"https://api.github.com/users/AbsoluteMSTR/orgs","repos_url":"https://api.github.com/users/AbsoluteMSTR/repos","events_url":"https://api.github.com/users/AbsoluteMSTR/events{/privacy}","received_events_url":"https://api.github.com/users/AbsoluteMSTR/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/CSV","name":"CSV","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-03-16T17:52:07Z","updated_at":"2016-03-16T19:42:56Z","closed_at":null,"body":"Current method is using read_table/read_csv and placing logic in sep field is a bruteforce way to process log files, but isn't clean as you will get two padded columns of NaN at the start and end and does not handle parameter of regex for column names.\r\n\r\n#### Code Sample, a copy-pastable example if possible\r\n\r\ndf = pd.read_table(\"file.log\",\r\n              sep=r'^(?P<datetime>[\\d\\-]+? [\\d:]+?) (?P<level>.+?) \\[(?P<name>.+?)\\] \\((?P<filename>.+?):(?P<lineno>\\d+?)\\) \\- (?P<message>.*?)$',\r\n             )\r\n\r\ndf currently would produce...\r\n\r\n | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7\r\n-------|-------|-------|-------|-------|-------|-------|-------|-------\r\n0 | NaN | 2015-01-01 00:00:00 | INFO | main | main.java | 10 | Hello World | NaN\r\n1 | NaN | 2015-01-01 00:00:00 | ERROR | main | main.java | 11 | Ouch! | NaN\r\n#### Expected Output\r\n\r\n |datetime | level | name | filename | lineno | message\r\n-------|------|-------|-------|-------|-------|-------\r\n0|2015-01-01 00:00:00 | INFO | main | main.java | 10 | Hello World\r\n1|2015-01-01 00:00:00 | ERROR | main | main.java | 11 | Ouch!\r\n\r\n#### output of ``pd.show_versions()``\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.5.1.final.0\r\npython-bits: 64\r\nOS: Windows\r\nOS-release: 7\r\nmachine: AMD64\r\nprocessor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: None\r\n\r\npandas: 0.18.0\r\nnose: 1.3.7\r\npip: 8.1.0\r\nsetuptools: 20.2.2\r\nCython: 0.23.4\r\nnumpy: 1.10.4\r\nscipy: 0.17.0\r\nstatsmodels: 0.6.1\r\nxarray: 0.7.1\r\nIPython: 4.1.1\r\nsphinx: 1.3.1\r\npatsy: 0.4.0\r\ndateutil: 2.5.0\r\npytz: 2016.1\r\nblosc: None\r\nbottleneck: 1.0.0\r\ntables: 3.2.2\r\nnumexpr: 2.5\r\nmatplotlib: 1.5.1\r\nopenpyxl: 2.3.3\r\nxlrd: 0.9.4\r\nxlwt: 1.0.0\r\nxlsxwriter: 0.8.2\r\nlxml: 3.5.0\r\nbs4: 4.4.1\r\nhtml5lib: 0.9999999\r\nhttplib2: 0.9.2\r\napiclient: None\r\nsqlalchemy: 1.0.12\r\npymysql: 0.7.1.None\r\npsycopg2: None\r\njinja2: 2.8\r\nboto: 2.39.0"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/197510478","html_url":"https://github.com/pydata/pandas/issues/12646#issuecomment-197510478","issue_url":"https://api.github.com/repos/pydata/pandas/issues/12646","id":197510478,"user":{"login":"AbsoluteMSTR","id":537223,"avatar_url":"https://avatars.githubusercontent.com/u/537223?v=3","gravatar_id":"","url":"https://api.github.com/users/AbsoluteMSTR","html_url":"https://github.com/AbsoluteMSTR","followers_url":"https://api.github.com/users/AbsoluteMSTR/followers","following_url":"https://api.github.com/users/AbsoluteMSTR/following{/other_user}","gists_url":"https://api.github.com/users/AbsoluteMSTR/gists{/gist_id}","starred_url":"https://api.github.com/users/AbsoluteMSTR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbsoluteMSTR/subscriptions","organizations_url":"https://api.github.com/users/AbsoluteMSTR/orgs","repos_url":"https://api.github.com/users/AbsoluteMSTR/repos","events_url":"https://api.github.com/users/AbsoluteMSTR/events{/privacy}","received_events_url":"https://api.github.com/users/AbsoluteMSTR/received_events","type":"User","site_admin":false},"created_at":"2016-03-16T19:42:56Z","updated_at":"2016-03-16T19:42:56Z","body":"@TomAugspurger, It's entirely possible to utilize a generator as an alternative to read_csv, read_table, but the question becomes if pandas should be able to provide api support toward complex text files like a log file.\r\n\r\n@jreback a generic example can be an access.log file generated by Apache HTTPd\r\n```\r\n127.0.0.1 - - [13/Sep/2013:01:38:09 -0400] \"GET / HTTP/1.1\" 200 - \"-\" \"-\"\r\n127.0.0.1 - - [13/Sep/2013:01:38:09 -0400] \"GET /testing HTTP/1.1\" 200 - \"-\" \"-\"\r\n127.0.0.1 - - [13/Sep/2013:01:38:10 -0400] \"GET /index.html HTTP/1.1\" 200 - \"-\" \"-\"\r\n```"}}