{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/pinot/issues/161","repository_url":"https://api.github.com/repos/linkedin/pinot","labels_url":"https://api.github.com/repos/linkedin/pinot/issues/161/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/pinot/issues/161/comments","events_url":"https://api.github.com/repos/linkedin/pinot/issues/161/events","html_url":"https://github.com/linkedin/pinot/issues/161","id":158860564,"number":161,"title":"Schema evolution: Default values for newly added columns ","user":{"login":"kishoreg","id":850791,"avatar_url":"https://avatars.githubusercontent.com/u/850791?v=3","gravatar_id":"","url":"https://api.github.com/users/kishoreg","html_url":"https://github.com/kishoreg","followers_url":"https://api.github.com/users/kishoreg/followers","following_url":"https://api.github.com/users/kishoreg/following{/other_user}","gists_url":"https://api.github.com/users/kishoreg/gists{/gist_id}","starred_url":"https://api.github.com/users/kishoreg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kishoreg/subscriptions","organizations_url":"https://api.github.com/users/kishoreg/orgs","repos_url":"https://api.github.com/users/kishoreg/repos","events_url":"https://api.github.com/users/kishoreg/events{/privacy}","received_events_url":"https://api.github.com/users/kishoreg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-06-07T07:51:11Z","updated_at":"2016-06-08T21:01:56Z","closed_at":null,"body":"One of the most common request from users \"when we add new columns, can the old segments have default values for these columns\". Currently Pinot forces users to re-bootstrap old data when new columns are added. Most users are ok with using default values in old segments for newly added columns.\r\n\r\nFor example, lets say user added a new column c_new to table that currently has d1, m1 columns.\r\nLets say we have two segments. S1_OLD (contains d1,m1) and S2_NEW (contains d1,m1, c_new)\r\n\r\nLets go over the queries and its current behavior in Pinot\r\nIf the query does not mention new column, results will be correct and as expected. \r\n\r\nIf the query specifies the new column, then we will not consider old data(w/o new column) while processing this query. In this case it depends on other filters in the query if the result is right or wrong.\r\n\r\nFor example, lets look at the current behavior in Pinot when a new column c_new is added at time T1\r\n```\r\nselect sum(m1) from Table //this will work in all cases\r\nselect sum(m1) from Table where c_new=x //this will be incorrect as pinot considers only new data\r\nselect sum(m1) from Table where c_new=x and time > T1 //this will be correct since all data after T1 will have c_new\r\nselect sum(c_new) from table //this works and returns correct result\r\nselect sum(c1) from table //this also works as expected\r\nbut\r\nselect sum(c_new), sum(m1)  from table //THIS WONT WORK AS EXPECTED\r\n```\r\n\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/linkedin/pinot/issues/comments/224726448","html_url":"https://github.com/linkedin/pinot/issues/161#issuecomment-224726448","issue_url":"https://api.github.com/repos/linkedin/pinot/issues/161","id":224726448,"user":{"login":"jfim","id":246807,"avatar_url":"https://avatars.githubusercontent.com/u/246807?v=3","gravatar_id":"","url":"https://api.github.com/users/jfim","html_url":"https://github.com/jfim","followers_url":"https://api.github.com/users/jfim/followers","following_url":"https://api.github.com/users/jfim/following{/other_user}","gists_url":"https://api.github.com/users/jfim/gists{/gist_id}","starred_url":"https://api.github.com/users/jfim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jfim/subscriptions","organizations_url":"https://api.github.com/users/jfim/orgs","repos_url":"https://api.github.com/users/jfim/repos","events_url":"https://api.github.com/users/jfim/events{/privacy}","received_events_url":"https://api.github.com/users/jfim/received_events","type":"User","site_admin":false},"created_at":"2016-06-08T21:01:56Z","updated_at":"2016-06-08T21:01:56Z","body":"`select * from Table` is also broken, it will give a set of columns that is not deterministic."}}
{"actor":{"display_login":"jfim"}}