{"action":"created","issue":{"url":"https://api.github.com/repos/Rdatatable/data.table/issues/1595","repository_url":"https://api.github.com/repos/Rdatatable/data.table","labels_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1595/labels{/name}","comments_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1595/comments","events_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1595/events","html_url":"https://github.com/Rdatatable/data.table/issues/1595","id":141619772,"number":1595,"title":"[Request] lagging of lists with shift()","user":{"login":"enfascination","id":960624,"avatar_url":"https://avatars.githubusercontent.com/u/960624?v=3","gravatar_id":"","url":"https://api.github.com/users/enfascination","html_url":"https://github.com/enfascination","followers_url":"https://api.github.com/users/enfascination/followers","following_url":"https://api.github.com/users/enfascination/following{/other_user}","gists_url":"https://api.github.com/users/enfascination/gists{/gist_id}","starred_url":"https://api.github.com/users/enfascination/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enfascination/subscriptions","organizations_url":"https://api.github.com/users/enfascination/orgs","repos_url":"https://api.github.com/users/enfascination/repos","events_url":"https://api.github.com/users/enfascination/events{/privacy}","received_events_url":"https://api.github.com/users/enfascination/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Rdatatable/data.table/labels/Enhancement","name":"Enhancement","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":{"login":"arunsrinivasan","id":3102045,"avatar_url":"https://avatars.githubusercontent.com/u/3102045?v=3","gravatar_id":"","url":"https://api.github.com/users/arunsrinivasan","html_url":"https://github.com/arunsrinivasan","followers_url":"https://api.github.com/users/arunsrinivasan/followers","following_url":"https://api.github.com/users/arunsrinivasan/following{/other_user}","gists_url":"https://api.github.com/users/arunsrinivasan/gists{/gist_id}","starred_url":"https://api.github.com/users/arunsrinivasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arunsrinivasan/subscriptions","organizations_url":"https://api.github.com/users/arunsrinivasan/orgs","repos_url":"https://api.github.com/users/arunsrinivasan/repos","events_url":"https://api.github.com/users/arunsrinivasan/events{/privacy}","received_events_url":"https://api.github.com/users/arunsrinivasan/received_events","type":"User","site_admin":false},"assignees":[{"login":"arunsrinivasan","id":3102045,"avatar_url":"https://avatars.githubusercontent.com/u/3102045?v=3","gravatar_id":"","url":"https://api.github.com/users/arunsrinivasan","html_url":"https://github.com/arunsrinivasan","followers_url":"https://api.github.com/users/arunsrinivasan/followers","following_url":"https://api.github.com/users/arunsrinivasan/following{/other_user}","gists_url":"https://api.github.com/users/arunsrinivasan/gists{/gist_id}","starred_url":"https://api.github.com/users/arunsrinivasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arunsrinivasan/subscriptions","organizations_url":"https://api.github.com/users/arunsrinivasan/orgs","repos_url":"https://api.github.com/users/arunsrinivasan/repos","events_url":"https://api.github.com/users/arunsrinivasan/events{/privacy}","received_events_url":"https://api.github.com/users/arunsrinivasan/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/Rdatatable/data.table/milestones/3","html_url":"https://github.com/Rdatatable/data.table/milestones/v1.9.8","labels_url":"https://api.github.com/repos/Rdatatable/data.table/milestones/3/labels","id":812342,"number":3,"title":"v1.9.8","description":"","creator":{"login":"arunsrinivasan","id":3102045,"avatar_url":"https://avatars.githubusercontent.com/u/3102045?v=3","gravatar_id":"","url":"https://api.github.com/users/arunsrinivasan","html_url":"https://github.com/arunsrinivasan","followers_url":"https://api.github.com/users/arunsrinivasan/followers","following_url":"https://api.github.com/users/arunsrinivasan/following{/other_user}","gists_url":"https://api.github.com/users/arunsrinivasan/gists{/gist_id}","starred_url":"https://api.github.com/users/arunsrinivasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arunsrinivasan/subscriptions","organizations_url":"https://api.github.com/users/arunsrinivasan/orgs","repos_url":"https://api.github.com/users/arunsrinivasan/repos","events_url":"https://api.github.com/users/arunsrinivasan/events{/privacy}","received_events_url":"https://api.github.com/users/arunsrinivasan/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":171,"state":"open","created_at":"2014-10-03T08:24:10Z","updated_at":"2016-09-30T00:13:32Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2016-03-17T15:38:06Z","updated_at":"2016-10-01T15:07:31Z","closed_at":"2016-08-11T17:55:25Z","body":"The way `shift()` works on lists, it can't be used to created lagged columns of type list the same way that it can create lagged columns of other types.  Example:\r\n\r\n```\r\ndt <- data.table(mtcars)[,.(gear, carb, cyl)]\r\n###  Make col of lists\r\ndt[,carbList:=list(list(unique(carb))), by=.(cyl, gear)]\r\n###  Now I want to lag/lead col of lists\r\ndt[,.(carb, carbLag=shift(carb)\r\n    , carbList, carbListLag=shift(carbList, type=\"lead\")), by=cyl] \r\n\r\n    cyl carb carbLag carbList carbListLag\r\n...\r\n19:   8    2      NA     2,4,3      4, 3,NA\r\n20:   8    4       2     2,4,3      4, 3,NA\r\n21:   8    3       4     2,4,3      4, 3,NA\r\n```\r\nfrom [SO](http://stackoverflow.com/questions/36040542/lagged-lists-in-data-table-r)\r\n\r\nIt would be useful if `shift()` treated lists the same.  The current behavior is technically documented — `?shift` says how it works on lists — but the docs describe that behavior outside the context of `data.table`, and the implication for within-`data.table` behavior isn't clear into you stumble into it.  In the context of `data.table`, where it functions as a really nice way to lag/lead columns by group, `shift`'s  behavior with lists seems inconsistent with how it works on other types.\r\n\r\nGeneral use case: Being able to compare a set to itself over time\r\nExample specific use case: Say I have a table of all of the people who have come to my birthday each year, one row per name per year. If I wanted to see how many people came this year that didn't come last year, and how many came last year that didn't come this year, and the people who come every year, and all the people who have ever come, I'd be able to do it all at once with something like this pseudocode:\r\n\r\n```\r\nbday[,.(friendL=list(friend)),by=year]\r\n        [order(year),{friendLlag:=shift(friendL); \r\n                             newfriends=length(setdiff(friendL, friendLlag)); \r\n                             badfriends=length(setdiff(friendLlag, friendL)); \r\n                             goodfriends=length(intersect(friendL, friendLlag)); \r\n                             allfriends=length(union(friendL, friendLlag)); \r\n                             list(allthosenumbers)\r\n         }]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/Rdatatable/data.table/issues/comments/250917177","html_url":"https://github.com/Rdatatable/data.table/issues/1595#issuecomment-250917177","issue_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1595","id":250917177,"user":{"login":"statquant","id":3612473,"avatar_url":"https://avatars.githubusercontent.com/u/3612473?v=3","gravatar_id":"","url":"https://api.github.com/users/statquant","html_url":"https://github.com/statquant","followers_url":"https://api.github.com/users/statquant/followers","following_url":"https://api.github.com/users/statquant/following{/other_user}","gists_url":"https://api.github.com/users/statquant/gists{/gist_id}","starred_url":"https://api.github.com/users/statquant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/statquant/subscriptions","organizations_url":"https://api.github.com/users/statquant/orgs","repos_url":"https://api.github.com/users/statquant/repos","events_url":"https://api.github.com/users/statquant/events{/privacy}","received_events_url":"https://api.github.com/users/statquant/received_events","type":"User","site_admin":false},"created_at":"2016-10-01T15:07:31Z","updated_at":"2016-10-01T15:07:31Z","body":"Actually... does this work ?\r\n```\r\ndt=data.table(x=c(1,1,2,2,3), y=list(1:2,3:4,5:6,7:8,9:10))\r\ndt[, z := .(y[shift(.I)]), by=x]\r\n   x     y    z\r\n1: 1   1,2 NULL\r\n2: 1   3,4  1,2\r\n3: 2   5,6 NULL\r\n4: 2   7,8 NULL <=== why NULL ??\r\n5: 3  9,10 NULL\r\n```\r\n"}}
{"actor":{"display_login":"statquant"}}