{"action":"created","issue":{"url":"https://api.github.com/repos/tidyverse/readr/issues/534","repository_url":"https://api.github.com/repos/tidyverse/readr","labels_url":"https://api.github.com/repos/tidyverse/readr/issues/534/labels{/name}","comments_url":"https://api.github.com/repos/tidyverse/readr/issues/534/comments","events_url":"https://api.github.com/repos/tidyverse/readr/issues/534/events","html_url":"https://github.com/tidyverse/readr/issues/534","id":183507797,"number":534,"title":"support overlapping fwf columns","user":{"login":"gergness","id":2104579,"avatar_url":"https://avatars.githubusercontent.com/u/2104579?v=3","gravatar_id":"","url":"https://api.github.com/users/gergness","html_url":"https://github.com/gergness","followers_url":"https://api.github.com/users/gergness/followers","following_url":"https://api.github.com/users/gergness/following{/other_user}","gists_url":"https://api.github.com/users/gergness/gists{/gist_id}","starred_url":"https://api.github.com/users/gergness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gergness/subscriptions","organizations_url":"https://api.github.com/users/gergness/orgs","repos_url":"https://api.github.com/users/gergness/repos","events_url":"https://api.github.com/users/gergness/events{/privacy}","received_events_url":"https://api.github.com/users/gergness/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-17T19:40:01Z","updated_at":"2016-10-24T14:35:44Z","closed_at":null,"body":"I sometimes receive fixed with files with specifications that have overlapping columns. readr's current behavior made it difficult to figure out what was going wrong. \r\n\r\nI wish that I was able to read in the data with the original specifications and get the 2 variables as normal, but if that's not possible, perhaps fwf_positions could check for overlaps and throw a better error?\r\n\r\n```r\r\nlibrary(readr)\r\nread_fwf(\"2015a\\n2016b\", fwf_positions(c(1, 3, 5), c(4, 4, 5)))\r\n#> Warning: 4 parsing failures.\r\n#> row col  expected    actual\r\n#>   1  X2 2 chars   1        \r\n#>   1  -- 3 columns 2 columns\r\n#>   2  X2 2 chars   1        \r\n#>   2  -- 3 columns 2 columns\r\n#> \r\n#> # A tibble: 2 × 3\r\n#>      X1    X2    X3\r\n#>   <int> <chr> <chr>\r\n#> 1  2015     a  <NA>\r\n#> 2  2016     b  <NA>\r\n```"},"comment":{"url":"https://api.github.com/repos/tidyverse/readr/issues/comments/255758418","html_url":"https://github.com/tidyverse/readr/issues/534#issuecomment-255758418","issue_url":"https://api.github.com/repos/tidyverse/readr/issues/534","id":255758418,"user":{"login":"gergness","id":2104579,"avatar_url":"https://avatars.githubusercontent.com/u/2104579?v=3","gravatar_id":"","url":"https://api.github.com/users/gergness","html_url":"https://github.com/gergness","followers_url":"https://api.github.com/users/gergness/followers","following_url":"https://api.github.com/users/gergness/following{/other_user}","gists_url":"https://api.github.com/users/gergness/gists{/gist_id}","starred_url":"https://api.github.com/users/gergness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gergness/subscriptions","organizations_url":"https://api.github.com/users/gergness/orgs","repos_url":"https://api.github.com/users/gergness/repos","events_url":"https://api.github.com/users/gergness/events{/privacy}","received_events_url":"https://api.github.com/users/gergness/received_events","type":"User","site_admin":false},"created_at":"2016-10-24T14:35:44Z","updated_at":"2016-10-24T14:35:44Z","body":"In case it is helpful @lucasmation, here is my work around. It depends on dplyr, stringr and tidyr.\r\n\r\nThere are 2 limitations that are okay for my use, but might not for all (but you should be able to adapt the code if needed): \r\n1) It requires that all columns be read in as character.\r\n2) It requires that the overlapping pattern be such that there are no variables that are subsets of more than one variable.\r\n\r\n```r\r\nread_fwf_overlap <- function(file, col_positions, col_types = NULL, ...) {\r\n  if (!is.null(col_types)) {\r\n    warning(\"read_fwf_overlap does not support col_types, columns will be read in as character\", \r\n         call. = FALSE)\r\n  }\r\n  fixed_specs <- fix_for_fwf_overlap(col_positions)\r\n  \r\n  out <- readr::read_fwf(file, fixed_specs$vinfo, col_types = cols(.default = col_character()), ...)\r\n  out <- dplyr::mutate_(out, .dots = fixed_specs$mdots)\r\n  out <- out[, col_positions$col_names] # Rearranges to original order\r\n  out\r\n}\r\n\r\n\r\n# readr doesn't support overlapping fixed with columns. This\r\n# function takes a fwf specificaiton with overlaps, and returns\r\n# a non-overlapping set and another text object that can\r\n# be passed to a mutate_ call on the loaded dataset to\r\n# create the overlapping variables.\r\n# Only works if the overlapping columns fit within exactly\r\n# one other column.\r\nfix_for_fwf_overlap <- function(fwf_positions) {\r\n  vinfo <- dplyr::as_data_frame(fwf_positions)\r\n  \r\n  # Check to see if there are overlapping columns. The arrange\r\n  # sorts the data so that the widest column with a given start\r\n  # is first, this way the other columns can be created\r\n  # from it.\r\n  vinfo <- dplyr::mutate_(vinfo, width = ~end - begin + 1)\r\n  vinfo <- dplyr::arrange_(vinfo, ~begin, ~desc(width))\r\n  vinfo <- dplyr::mutate_(vinfo, maxend = ~cummax(dplyr::lag(end, default = 0)),\r\n                          is_overlapping = ~begin < maxend,\r\n                          is_overlapper = ~end > dplyr::lead(begin, default = Inf))\r\n  \r\n  # Get dataframe of non-overlapping columns\r\n  nonoverlaps <- vinfo\r\n  nonoverlaps <- dplyr::filter_(nonoverlaps, ~!is_overlapping)\r\n  nonoverlaps <- dplyr::select_(nonoverlaps, ~col_names, ~begin, ~end)\r\n  \r\n  # Make mdots to pass into a mutate_ function that will take substrings of\r\n  # The widest columns to make the sub-columns.\r\n  ## Start by filling some of the overlapper information down to the overlapping\r\n  overlaps <- vinfo\r\n  overlaps <- dplyr::filter_(overlaps, ~is_overlapper | is_overlapping)\r\n  if (nrow(overlaps) > 1) {\r\n    overlaps <- dplyr::mutate_(overlaps, overlapper_name = ~ifelse(is_overlapper, col_names, NA),\r\n                               overlapper_begin = ~ifelse(is_overlapper, begin, NA),\r\n                               overlapper_end = ~ifelse(is_overlapper, end, NA))\r\n    overlaps <- tidyr::fill_(overlaps, c(\"overlapper_name\", \"overlapper_begin\", \"overlapper_end\"))\r\n    \r\n    ## Check if there are variables that are partially overlapped, if so, then just throw\r\n    ## an error.\r\n    if (any(overlaps$end > overlaps$overlapper_end)) stop(\"Columns overlap with more than one othe variable\", \r\n                                                          call. = FALSE)\r\n    \r\n    ## Make a text representation of the substring calls we'll want to make\r\n    ## on the dataframe after it is loaded.\r\n    overlaps <- dplyr::mutate_(overlaps, rel_begin = ~begin - overlapper_begin + 1,\r\n                               rel_end = ~end - overlapper_begin + 1)\r\n    overlaps <- dplyr::filter_(overlaps, ~is_overlapping)\r\n    \r\n    mdots <- sprintf(\"stringr::str_sub(%s, %s, %s)\", overlaps$overlapper_name, \r\n                     overlaps$rel_begin, overlaps$rel_end)\r\n    \r\n    names(mdots) <- overlaps$col_names\r\n  } else {\r\n    # If no overlapping, don't worry about mdots.\r\n    mdots <- NULL\r\n  }\r\n  # Return list of the nonoverlaps, plus the dots to be passed into mdots\r\n  list(vinfo = nonoverlaps, mdots = mdots)\r\n}\r\n```\r\n\r\n```r\r\n'123456789abcdefg\\n123456789abcdefg' %>% \r\n  read_fwf_overlap(fwf_positions(start=c(1,3,3,8,10 ), \r\n                                 end=  c(2,4,7,9,13 ) )\r\n  ) \r\n\r\n#> # A tibble: 2 × 5\r\n#>      X1    X2    X3    X4    X5\r\n#>   <chr> <chr> <chr> <chr> <chr>\r\n#> 1    12   345 34567    89  abcd\r\n#> 2    12   345 34567    89  abcd\r\n\r\n```\r\n"}}
{"actor":{"display_login":"gergness"}}