{"action":"opened","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/133","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/133/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/133/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/133/events","html_url":"https://github.com/rstudio/sparklyr/issues/133","id":168075408,"number":133,"title":"Predict(...) on large newData seems unordered","user":{"login":"l4mun","id":15957925,"avatar_url":"https://avatars.githubusercontent.com/u/15957925?v=3","gravatar_id":"","url":"https://api.github.com/users/l4mun","html_url":"https://github.com/l4mun","followers_url":"https://api.github.com/users/l4mun/followers","following_url":"https://api.github.com/users/l4mun/following{/other_user}","gists_url":"https://api.github.com/users/l4mun/gists{/gist_id}","starred_url":"https://api.github.com/users/l4mun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/l4mun/subscriptions","organizations_url":"https://api.github.com/users/l4mun/orgs","repos_url":"https://api.github.com/users/l4mun/repos","events_url":"https://api.github.com/users/l4mun/events{/privacy}","received_events_url":"https://api.github.com/users/l4mun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-28T11:39:42Z","updated_at":"2016-07-28T11:39:42Z","closed_at":null,"body":"Hi,\r\n\r\nSorry for not providing a reproductible example. However the problems seems to arise when dataset are not \"small\".\r\nI tried by my own Sparklyr on a bigger dataset. I picked up Airlines2007 and 2008. Merge with Weather on Date. Trained a model on 2007 (300k rows) and predicted on 2008 ((300k rows). \r\n\r\nAt the end i train a linear regression and a RandomForest. \r\n\r\n````\r\nmy_ml_rf = ml_random_forest(x = train ,response = \"ArrDelay\", features = myFeatures)\r\np1 = predict(my_ml_rf, validation)\r\np2 = predict(my_ml_rf, validation)\r\nqplot(p1, p2)\r\n\r\nmy_ml_linear = ml_linear_regression(x = train ,response = \"ArrDelay\", features = myFeatures)\r\np11 = predict(my_ml_linear, validation)\r\np22 = predict(my_ml_linear, validation)\r\nqplot(p11, p22)\r\n````\r\n\r\nHowever and surprinsingly, here are the graphs\r\n![image](https://cloud.githubusercontent.com/assets/15957925/17210968/b16e1536-54c6-11e6-9610-77b5a1718b1d.png)\r\n\r\n![image](https://cloud.githubusercontent.com/assets/15957925/17211016/00daea04-54c7-11e6-9c6a-b3851c8f14c9.png)\r\n\r\n````\r\np11[120000:110000] == p22[120000:110000]\r\n````\r\n![image](https://cloud.githubusercontent.com/assets/15957925/17211252/a72b67b6-54c8-11e6-8f5f-855e3ea958a3.png)\r\n\r\nIt clearly seems that \"bloc\" order change sometimes. I guess it is due to the inner functionnement of Spark. The first returned block are the first processed. However it is an annoying problem.\r\n\r\nBest regards,\r\nJacques\r\n\r\n "}}
{"actor":{"display_login":"l4mun"}}