{"action":"opened","issue":{"url":"https://api.github.com/repos/JuliaData/CSV.jl/issues/45","repository_url":"https://api.github.com/repos/JuliaData/CSV.jl","labels_url":"https://api.github.com/repos/JuliaData/CSV.jl/issues/45/labels{/name}","comments_url":"https://api.github.com/repos/JuliaData/CSV.jl/issues/45/comments","events_url":"https://api.github.com/repos/JuliaData/CSV.jl/issues/45/events","html_url":"https://github.com/JuliaData/CSV.jl/issues/45","id":183946593,"number":45,"title":"low level API: how to skip field, test for EOL, EOF","user":{"login":"tpapp","id":84122,"avatar_url":"https://avatars.githubusercontent.com/u/84122?v=3","gravatar_id":"","url":"https://api.github.com/users/tpapp","html_url":"https://github.com/tpapp","followers_url":"https://api.github.com/users/tpapp/followers","following_url":"https://api.github.com/users/tpapp/following{/other_user}","gists_url":"https://api.github.com/users/tpapp/gists{/gist_id}","starred_url":"https://api.github.com/users/tpapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tpapp/subscriptions","organizations_url":"https://api.github.com/users/tpapp/orgs","repos_url":"https://api.github.com/users/tpapp/repos","events_url":"https://api.github.com/users/tpapp/events{/privacy}","received_events_url":"https://api.github.com/users/tpapp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-19T12:22:59Z","updated_at":"2016-10-19T12:22:59Z","closed_at":null,"body":"I am processing a CSV file by rows. The file is huge, only certain rows are kept according to a filter function, and constructing a `DataFrame` would be prohibitively expensive and also unnecessary. So I would prefer to read into a tuple/vector, from which I would construct a composite type. Some fields need to be skipped.\r\n\r\nCurrently using something like this:\r\n```julia\r\n\"Read a line of given `types` from `source`. A `Void` type skips the field.\"\r\nfunction parseline(source::CSV.Source, types; row=0)\r\n    values = []\r\n    for (col,T) in enumerate(types)\r\n        if T == Void\r\n            # FIXME: could we just skip?\r\n            CSV.parsefield(source, Nullable{String}, row, col)\r\n        else\r\n            push!(values, CSV.parsefield(source, T, row, col))\r\n        end\r\n    end\r\n    # FIXME: test for end of line, raise error if not reached\r\n    values\r\nend\r\n```\r\nThen I would so something like (pseudocode)\r\n```julia\r\na = []\r\nwhile !eof(source)\r\n    v = MyType(parseline(source, (Int, Nullable{Date}, Void, Int, Date, Void))...)\r\n    if keep_this_row(v)\r\n        push!(a, v)\r\n    end\r\nend\r\n```\r\n\r\nQuestions:\r\n1. does `eof(s::CSV.Source)=eof(s.io)` make sense or should I use something else?\r\n2. how to check for the end of the line after `parsefield`?\r\n3. can I skip fields without parsing?\r\n\r\nOr should I just use `readsplitline` and post-process that? But that way I can't make use of the missing value framework of `CSV`."}}
{"actor":{"display_login":"tpapp"}}