{"action":"created","issue":{"url":"https://api.github.com/repos/tidyverse/readr/issues/524","repository_url":"https://api.github.com/repos/tidyverse/readr","labels_url":"https://api.github.com/repos/tidyverse/readr/issues/524/labels{/name}","comments_url":"https://api.github.com/repos/tidyverse/readr/issues/524/comments","events_url":"https://api.github.com/repos/tidyverse/readr/issues/524/events","html_url":"https://github.com/tidyverse/readr/issues/524","id":178127393,"number":524,"title":"Second chunk has 0 observations when reading in data in chunks with connections","user":{"login":"Ballings","id":7026744,"avatar_url":"https://avatars.githubusercontent.com/u/7026744?v=3","gravatar_id":"","url":"https://api.github.com/users/Ballings","html_url":"https://github.com/Ballings","followers_url":"https://api.github.com/users/Ballings/followers","following_url":"https://api.github.com/users/Ballings/following{/other_user}","gists_url":"https://api.github.com/users/Ballings/gists{/gist_id}","starred_url":"https://api.github.com/users/Ballings/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ballings/subscriptions","organizations_url":"https://api.github.com/users/Ballings/orgs","repos_url":"https://api.github.com/users/Ballings/repos","events_url":"https://api.github.com/users/Ballings/events{/privacy}","received_events_url":"https://api.github.com/users/Ballings/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-20T17:43:53Z","updated_at":"2016-12-22T19:24:34Z","closed_at":null,"body":"# The problem is that when I read in data through a connection in batches, the second\n# batch will not have a header anymore, but I do want to have the column names that were\n# obtained in the first batch (for processing). So in the first batch I set col_names=TRUE, I\n# extract the column names of the result and store them in a character vector called colnames,\n# and use col_names=colnames in the second batch. The second batch then comes out empty.\n# A similar procedure does work with read.table.\n# Am I missing anything?\n# MAKE SOME TOY DATA AND SET read_delim PARAMETERS\n\nn <- 1000\nwrite.table(data.frame(a=1:n,b=1:n,c=runif(n),response=sample(c(0,1),n,TRUE)),\n            file=\"test.csv\",\n            row.names=FALSE, sep=\",\")\n\ncolClasses <- cols(a = col_integer(),\n                   b=col_integer(),\n                   c=col_double(),\n                   response=col_factor(levels=c(0,1)))\nbatchsize=100\nsep= \",\"\nquote=\"\\\"\"\n# PROCESS DATA IN CHUNKS\n\ncloseAllConnections()\nfileconnection <- file(paste0(getwd(),\"/test.csv\"))\nopen(fileconnection, \"rb\")\n# READ IN FIRST BATCH:\n\ndata <- read_delim(file=fileconnection,\n                   delim=sep,\n                   col_names=TRUE,\n                   col_types=colClasses,\n                   n_max=batchsize)\ndata <- data.frame(data)\nstr(data)\n# > str(data)\n# 'data.frame':  10000 obs. of  4 variables:\n# $ a       : int  1 2 3 4 5 6 7 8 9 10 ...\n# $ b       : int  1 2 3 4 5 6 7 8 9 10 ...\n# $ c       : num  0.288 0.833 0.509 0.107 0.865 ...\n# $ response: Factor w/ 2 levels \"0\",\"1\": 1 2 2 1 2 2 1 1 1 1 ...\n# GET COLNAMES TO USE IN SECOND BATCH\n\ncolnames <- colnames(data)\n# DO SOME PROCESSING\n# REMOVE FROM MEMORY\n# READ IN SECOND BATCH:\n\ndata <- read_delim(file=fileconnection,\n                   delim=sep,\n                   col_names=colnames,\n                   col_types=colClasses,\n                   n_max=batchsize)\ndata <- data.frame(data)\nstr(data)\n# 'data.frame':  0 obs. of  4 variables:\n# $ a       : int\n# $ b       : int\n# $ c       : num\n# $ response: Factor w/ 2 levels \"0\",\"1\":\n# DO SOME PROCESSING\n# REMOVE FROM MEMORY\n# READ IN THIRD BATCH\n# ....\n\ncloseAllConnections()\n"},"comment":{"url":"https://api.github.com/repos/tidyverse/readr/issues/comments/268871732","html_url":"https://github.com/tidyverse/readr/issues/524#issuecomment-268871732","issue_url":"https://api.github.com/repos/tidyverse/readr/issues/524","id":268871732,"user":{"login":"hadley","id":4196,"avatar_url":"https://avatars.githubusercontent.com/u/4196?v=3","gravatar_id":"","url":"https://api.github.com/users/hadley","html_url":"https://github.com/hadley","followers_url":"https://api.github.com/users/hadley/followers","following_url":"https://api.github.com/users/hadley/following{/other_user}","gists_url":"https://api.github.com/users/hadley/gists{/gist_id}","starred_url":"https://api.github.com/users/hadley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hadley/subscriptions","organizations_url":"https://api.github.com/users/hadley/orgs","repos_url":"https://api.github.com/users/hadley/repos","events_url":"https://api.github.com/users/hadley/events{/privacy}","received_events_url":"https://api.github.com/users/hadley/received_events","type":"User","site_admin":false},"created_at":"2016-12-22T19:24:34Z","updated_at":"2016-12-22T19:24:34Z","body":"Please correct your markdown"}}
{"actor":{"display_login":"hadley"}}