{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/1432","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/1432/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/1432/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/1432/events","html_url":"https://github.com/rstudio/sparklyr/issues/1432","id":316885551,"number":1432,"title":"spark_apply not working ","user":{"login":"northeastloon","id":35330313,"avatar_url":"https://avatars0.githubusercontent.com/u/35330313?v=4","gravatar_id":"","url":"https://api.github.com/users/northeastloon","html_url":"https://github.com/northeastloon","followers_url":"https://api.github.com/users/northeastloon/followers","following_url":"https://api.github.com/users/northeastloon/following{/other_user}","gists_url":"https://api.github.com/users/northeastloon/gists{/gist_id}","starred_url":"https://api.github.com/users/northeastloon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/northeastloon/subscriptions","organizations_url":"https://api.github.com/users/northeastloon/orgs","repos_url":"https://api.github.com/users/northeastloon/repos","events_url":"https://api.github.com/users/northeastloon/events{/privacy}","received_events_url":"https://api.github.com/users/northeastloon/received_events","type":"User","site_admin":false},"labels":[{"id":403946014,"url":"https://api.github.com/repos/rstudio/sparklyr/labels/distributed%20r","name":"distributed r","color":"006b75","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-04-23T16:22:01Z","updated_at":"2018-05-11T15:26:18Z","closed_at":null,"author_association":"NONE","body":"I have the following spark DataFrame cached in memory (with local mode):\r\n```r\r\nObservations: ??\r\nVariables: 12\r\n$ COUNTRY     <int> 218, 218, 218, 218, 218...\r\n$ illit       <dbl> 0, 0, 0, 0, 0...\r\n$ isol_illit  <dbl> 0, 0, 0, 0, 0...\r\n$ AGE         <int> 29, 25, 47, 39, 18...\r\n$ age_cat     <chr> \"25-35\", \"25-35\", \"45-54\", \"35-44\", \"15-24\", ...\r\n$ SEX         <dbl> 0, 1, 0, 1, 1...\r\n$ hh_size     <dbl> 4, 4, 6, 6, 6, ...\r\n$ perwt_std   <dbl> 1.340802e-06, 1.340802e-06, 1.340802e-06, 1.340802e-06, 1.340802e-06...\r\n$ rural       <dbl> 0, 0, 0, 0, 0, ...\r\n$ marst_rec   <int> 2, 2, 2, 2, 1...\r\n$ prop_unrel  <dbl> 0, 0, 0, 0, 0...\r\n$ nchild10_14 <dbl> 0, 0, 1, 1, 1...\r\n```\r\nI can run the following ml function fine\r\n```r\r\nmodel_pooled1 <- model_data %>%\r\n  ml_logistic_regression(formula = isol_illit ~ rural + AGE + SEX + hh_size + marst_rec + prop_unrel + nchild10_14,  weight_col = \"perwt_std\",    max_iter  = 25L)\r\n```\r\nHowever if I try do use glm through spark_apply...\r\n```r\r\nmodel_country1 <- model_data %>%\r\n  spark_apply(\r\n    function(e) broom::tidy(glm(formula = isol_illit ~ rural + AGE + SEX + hh_size + marst_rec + prop_unrel + nchild10_14, weights = perwt_std, family = \"binomial\", data = e)),\r\n    names = c(\"term\", \"estimate\", \"std.error\", \"statistic\", \"p.value\"),\r\n    group_by = \"COUNTRY\") \r\n```\r\n...I get the following error: \r\n```r\r\nError in file(con, \"r\") : cannot open the connection\r\nIn addition: Warning message:\r\nIn file(con, \"r\") :\r\n  cannot open file 'C:\\Users\\~\\AppData\\Local\\Temp\\RtmpUBKRZu\\file1fe07d436c2_spark.log': Permission denied\r\n```\r\nI get the same error removing the group_by command. It is also replicated on a windows 10 machine running r version 3.4.4.\r\n```r\r\nR version 3.4.3 (2017-11-30)\r\nPlatform: x86_64-w64-mingw32/x64 (64-bit)\r\nRunning under: Windows 7 x64 (build 7601) Service Pack 1\r\n\r\nMatrix products: default\r\n\r\nlocale:\r\n[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252\r\n[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    \r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n [1] bindrcpp_0.2.2      viridis_0.5.1       viridisLite_0.3.0   tmap_1.11-2         rgdal_1.2-18        sp_1.2-7           \r\n [7] stringr_1.3.0       tidyr_0.8.0         broom_0.4.4         countrycode_1.00.0  ggplot2_2.2.1       ipumsr_0.1.1       \r\n[13] dplyr_0.7.4         sparklyr_0.7.0-9026\r\n\r\nloaded via a namespace (and not attached):\r\n  [1] colorspace_1.3-2   deldir_0.1-15      class_7.3-14       gdalUtils_2.0.1.7  leaflet_1.1.0      rprojroot_1.3-2   \r\n  [7] satellite_1.0.1    base64enc_0.1-3    dichromat_2.0-0    rstudioapi_0.7     xml2_1.2.0         codetools_0.2-15  \r\n [13] splines_3.4.3      R.methodsS3_1.7.1  mnormt_1.5-5       knitr_1.20         config_0.3         geojsonlint_0.2.0 \r\n [19] jsonlite_1.5       tmaptools_1.2-4    dbplyr_1.2.1       png_0.1-7          R.oo_1.21.0        rgeos_0.3-26      \r\n [25] shiny_1.0.5        compiler_3.4.3     httr_1.3.1         backports_1.1.2    mapview_2.3.0      assertthat_0.2.0  \r\n [31] Matrix_1.2-12      lazyeval_0.2.1     cli_1.0.0          htmltools_0.3.6    tools_3.4.3        coda_0.19-1       \r\n [37] gtable_0.2.0       glue_1.2.0         reshape2_1.4.3     rappdirs_0.3.1     gmodels_2.16.2     V8_1.5            \r\n [43] Rcpp_0.12.16       raster_2.6-7       spdep_0.7-7        gdata_2.18.0       nlme_3.1-131       udunits2_0.13     \r\n [49] iterators_1.0.9    crosstalk_1.0.0    psych_1.8.3.3      mime_0.5           devtools_1.13.5    gtools_3.5.0      \r\n [55] XML_3.98-1.10      LearnBayes_2.15.1  MASS_7.3-48        scales_0.5.0       parallel_3.4.3     expm_0.999-2      \r\n [61] RColorBrewer_1.1-2 yaml_2.1.18        curl_3.2           memoise_1.1.0      geosphere_1.5-7    gridExtra_2.3     \r\n [67] stringi_1.1.7      jsonvalidate_1.0.0 foreach_1.4.4      e1071_1.6-8        boot_1.3-20        spData_0.2.8.3    \r\n [73] rlang_0.2.0        pkgconfig_2.0.1    bitops_1.0-6       lattice_0.20-35    purrr_0.2.4        bindr_0.1.1       \r\n [79] sf_0.6-1           htmlwidgets_1.0    tidyselect_0.2.4   osmar_1.1-7        plyr_1.8.4         magrittr_1.5      \r\n [85] R6_2.2.2           DBI_0.8            pillar_1.2.1       haven_1.1.1        foreign_0.8-69     withr_2.1.2       \r\n [91] units_0.5-1        RCurl_1.95-4.10    tibble_1.4.2       crayon_1.3.4       rmapshaper_0.4.0   utf8_1.1.3        \r\n [97] KernSmooth_2.23-15 grid_3.4.3         forcats_0.3.0      digest_0.6.15      classInt_0.1-24    webshot_0.5.0     \r\n[103] xtable_1.8-2       httpuv_1.3.6.2     R.utils_2.6.0      openssl_1.0.1      stats4_3.4.3       munsell_0.4.3  \r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/comments/388397564","html_url":"https://github.com/rstudio/sparklyr/issues/1432#issuecomment-388397564","issue_url":"https://api.github.com/repos/rstudio/sparklyr/issues/1432","id":388397564,"user":{"login":"marcoscattolin","id":7213771,"avatar_url":"https://avatars3.githubusercontent.com/u/7213771?v=4","gravatar_id":"","url":"https://api.github.com/users/marcoscattolin","html_url":"https://github.com/marcoscattolin","followers_url":"https://api.github.com/users/marcoscattolin/followers","following_url":"https://api.github.com/users/marcoscattolin/following{/other_user}","gists_url":"https://api.github.com/users/marcoscattolin/gists{/gist_id}","starred_url":"https://api.github.com/users/marcoscattolin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcoscattolin/subscriptions","organizations_url":"https://api.github.com/users/marcoscattolin/orgs","repos_url":"https://api.github.com/users/marcoscattolin/repos","events_url":"https://api.github.com/users/marcoscattolin/events{/privacy}","received_events_url":"https://api.github.com/users/marcoscattolin/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T15:26:18Z","updated_at":"2018-05-11T15:26:18Z","author_association":"NONE","body":"I ran into this as well. Looks like it's related to the fact that dataframe includes a column of type org.apache.spark.ml.linalg.VectorUDT\r\n\r\nCheck following code:\r\n\r\n# Load packages\r\nlibrary(sparklyr)\r\nlibrary(dplyr)\r\n\r\n\r\n# set spark context\r\nsc <- spark_connect(master = \"local\", version = \"2.2.0\")\r\n\r\n\r\n# simple data frame -- works fine\r\niris_tbl <- copy_to(sc, iris, \"iris\", overwrite = T) %>%\r\n        sdf_repartition(partitions = 3, partition_by = \"Species\") %>%\r\n        sdf_register(\"iris\")\t\t\t\t\r\n\r\niris_tbl %>%\r\n        spark_apply(function(x) head(x,1))\r\n\r\n# data frame with linalg -- gives \"Error in file(con, \"r\") : cannot open the connection ...\"\r\niris_vect_tbl <- iris_tbl %>%\r\n        sdf_repartition(partitions = 3, partition_by = \"Species\") %>%\r\n        ft_vector_assembler(input_cols = c(\"Sepal_Length\",\"Sepal_Width\",\"Petal_Length\",\"Petal_Width\"), \"vec\") %>%\r\n        sdf_register(\"iris_vec\")\r\n\r\niris_vect_tbl %>%\r\n        spark_apply(function(x) head(x,1))\r\n\r\nSession info ---------------------------------------------------------------------------------------------------------------------------------------------------------\r\n setting  value                       \r\n version  R version 3.4.1 (2017-06-30)\r\n system   x86_64, mingw32             \r\n ui       RStudio (1.1.423)           \r\n language (EN)                        \r\n collate  Italian_Italy.1252          \r\n tz       Europe/Berlin               \r\n date     2018-05-11                  \r\n\r\nPackages -------------------------------------------------------------------------------------------------------------------------------------------------------------\r\n package    * version date       source        \r\n assertthat   0.2.0   2017-04-11 CRAN (R 3.4.0)\r\n backports    1.1.1   2017-09-25 CRAN (R 3.4.1)\r\n base       * 3.4.1   2017-06-30 local         \r\n base64enc    0.1-3   2015-07-28 CRAN (R 3.4.1)\r\n bindr        0.1     2016-11-13 CRAN (R 3.4.0)\r\n bindrcpp   * 0.2     2017-06-17 CRAN (R 3.4.1)\r\n broom        0.4.2   2017-02-13 CRAN (R 3.4.0)\r\n cli          1.0.0   2017-11-05 CRAN (R 3.4.4)\r\n compiler     3.4.1   2017-06-30 local         \r\n config       0.2     2016-08-02 CRAN (R 3.4.2)\r\n crayon       1.3.4   2017-09-16 CRAN (R 3.4.2)\r\n datasets   * 3.4.1   2017-06-30 local         \r\n DBI        * 0.7     2017-06-18 CRAN (R 3.4.2)\r\n dbplyr       1.1.0   2017-06-27 CRAN (R 3.4.2)\r\n devtools     1.13.3  2017-08-02 CRAN (R 3.4.2)\r\n digest       0.6.15  2018-01-28 CRAN (R 3.4.3)\r\n dplyr      * 0.7.4   2017-09-28 CRAN (R 3.4.2)\r\n foreign      0.8-69  2017-06-22 CRAN (R 3.4.1)\r\n glue         1.1.1   2017-06-21 CRAN (R 3.4.1)\r\n graphics   * 3.4.1   2017-06-30 local         \r\n grDevices  * 3.4.1   2017-06-30 local         \r\n grid         3.4.1   2017-06-30 local         \r\n htmltools    0.3.6   2017-04-28 CRAN (R 3.4.0)\r\n httpuv       1.3.5   2017-07-04 CRAN (R 3.4.2)\r\n httr         1.3.1   2017-08-20 CRAN (R 3.4.2)\r\n jsonlite     1.5     2017-06-01 CRAN (R 3.4.0)\r\n lattice      0.20-35 2017-03-25 CRAN (R 3.4.1)\r\n lazyeval     0.2.1   2017-10-29 CRAN (R 3.4.4)\r\n magrittr     1.5     2014-11-22 CRAN (R 3.4.0)\r\n memoise      1.1.0   2017-04-21 CRAN (R 3.4.2)\r\n methods    * 3.4.1   2017-06-30 local         \r\n mime         0.5     2016-07-07 CRAN (R 3.4.0)\r\n mnormt       1.5-5   2016-10-15 CRAN (R 3.4.0)\r\n nlme         3.1-131 2017-02-06 CRAN (R 3.4.1)\r\n openssl      0.9.7   2017-09-06 CRAN (R 3.4.2)\r\n parallel     3.4.1   2017-06-30 local         \r\n pillar       1.2.1   2018-02-27 CRAN (R 3.4.4)\r\n pkgconfig    2.0.1   2017-03-21 CRAN (R 3.4.0)\r\n plyr         1.8.4   2016-06-08 CRAN (R 3.4.0)\r\n psych        1.7.8   2017-09-09 CRAN (R 3.4.2)\r\n purrr        0.2.4   2017-10-18 CRAN (R 3.4.2)\r\n R6           2.2.2   2017-06-17 CRAN (R 3.4.2)\r\n rappdirs     0.3.1   2016-03-28 CRAN (R 3.4.2)\r\n Rcpp         0.12.14 2017-11-23 CRAN (R 3.4.3)\r\n reshape2     1.4.2   2016-10-22 CRAN (R 3.4.0)\r\n rlang        0.2.0   2018-02-20 CRAN (R 3.4.4)\r\n rprojroot    1.2     2017-01-16 CRAN (R 3.4.2)\r\n rstudioapi   0.7     2017-09-07 CRAN (R 3.4.2)\r\n shiny        1.0.5   2017-08-23 CRAN (R 3.4.2)\r\n sparklyr   * 0.8.2   2018-05-06 CRAN (R 3.4.4)\r\n stats      * 3.4.1   2017-06-30 local         \r\n stringi      1.1.5   2017-04-07 CRAN (R 3.4.0)\r\n stringr      1.2.0   2017-02-18 CRAN (R 3.4.0)\r\n tibble       1.4.2   2018-01-22 CRAN (R 3.4.4)\r\n tidyr        0.7.2   2017-10-16 CRAN (R 3.4.2)\r\n tools        3.4.1   2017-06-30 local         \r\n utf8         1.1.3   2018-01-03 CRAN (R 3.4.4)\r\n utils      * 3.4.1   2017-06-30 local         \r\n withr        2.0.0   2017-07-28 CRAN (R 3.4.2)\r\n xtable       1.8-2   2016-02-05 CRAN (R 3.4.0)\r\n yaml         2.1.14  2016-11-12 CRAN (R 3.4.0)"}}
{"actor":{"display_login":"marcoscattolin"}}