{"action":"created","issue":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/2492","repository_url":"https://api.github.com/repos/mlr-org/mlr","labels_url":"https://api.github.com/repos/mlr-org/mlr/issues/2492/labels{/name}","comments_url":"https://api.github.com/repos/mlr-org/mlr/issues/2492/comments","events_url":"https://api.github.com/repos/mlr-org/mlr/issues/2492/events","html_url":"https://github.com/mlr-org/mlr/issues/2492","id":381993614,"node_id":"MDU6SXNzdWUzODE5OTM2MTQ=","number":2492,"title":"Lightgbm custom learner with hiperparameters tuning \"unserialize(node$con) : error reading from connection\"","user":{"login":"matkalinowski","id":28002118,"node_id":"MDQ6VXNlcjI4MDAyMTE4","avatar_url":"https://avatars3.githubusercontent.com/u/28002118?v=4","gravatar_id":"","url":"https://api.github.com/users/matkalinowski","html_url":"https://github.com/matkalinowski","followers_url":"https://api.github.com/users/matkalinowski/followers","following_url":"https://api.github.com/users/matkalinowski/following{/other_user}","gists_url":"https://api.github.com/users/matkalinowski/gists{/gist_id}","starred_url":"https://api.github.com/users/matkalinowski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matkalinowski/subscriptions","organizations_url":"https://api.github.com/users/matkalinowski/orgs","repos_url":"https://api.github.com/users/matkalinowski/repos","events_url":"https://api.github.com/users/matkalinowski/events{/privacy}","received_events_url":"https://api.github.com/users/matkalinowski/received_events","type":"User","site_admin":false},"labels":[{"id":54191382,"node_id":"MDU6TGFiZWw1NDE5MTM4Mg==","url":"https://api.github.com/repos/mlr-org/mlr/labels/type-question","name":"type-question","color":"cc317c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-11-18T19:52:30Z","updated_at":"2018-11-21T09:37:49Z","closed_at":null,"author_association":"NONE","body":"**How I run this**\r\n- open Rstudio;\r\n- create script & paste code;\r\n- add data from attachment;\r\n- run whole script (for me it failed everytime, it runs about 1 minute);\r\n- it turns down the session so i use new jobs feature in rstudio to see the output.\r\n\r\n**Informations about the code:**\r\n- First few methods are just overwriting of mlr methods due to error in makeRLearnerInternal, line: learner$fix.factors.prediction = F seems to be fixing the problem.\r\n- lightgbm integration comes from https://github.com/MelvinWoodley/mlr-lightgbm-Extension/blob/master/mlr_lightgbm_starter.R\r\n\r\n**Other informations:**\r\n- Data is added as an attachment **([data.zip](https://github.com/mlr-org/mlr/files/2593137/data.zip))**.\r\n- I haven't tried other parallelStart* options.\r\n- lower values of maxit (like 4) in makeTuneControlRandom are working- optimization finishes (in this example You can run makeTuneControlRandom or makeTuneControlGrid).\r\n- I still got a lot of RAM (4-5 GB) available while running this example.\r\n- It doesn't matter how many cores I've set.\r\n- sometimes traceback is not available;\r\n- sometimes logs are scattered like in the given example;\r\n- lightgbm library was installed using r package.\r\n- It fails in two ways:\r\n     - one is when i run it in parallel- then You can see traceback below (and whole output at the end)\r\n     - second is when i run it with commented lines (and whole output at the end):\r\n```r\r\nlibrary(parallelMap)\r\nparallelStartSocket(cpus = 2)\r\nparallelExport('trainLearner.regr.lightgbm', 'predictLearner.regr.lightgbm', 'lgb.Dataset')\r\n```\r\n\r\n**Traceback:**\r\n> 14: unserialize(node$con)\r\n> 13: recvData.SOCKnode(con)\r\n> 12: recvData(con)\r\n> 11: FUN(X[[i]], ...)\r\n> 10: lapply(cl[1:jobs], recvResult)\r\n> 9: staticClusterApply(cl, fun, vlen, argfun)\r\n> 8: clusterMap(cl = NULL, slaveWrapper, ..., .i = iters, MoreArgs = more.args, \r\n>        SIMPLIFY = FALSE, USE.NAMES = FALSE)\r\n> 7: parallelMap(evalOptimizationState, dobs, states, level = level, \r\n>        more.args = list(learner = learner, task = task, resampling = resampling, \r\n>            measures = measures, par.set = par.set, bits.to.features = bits.to.features, \r\n>            control = control, opt.path = opt.path, show.info = show.info, \r\n>            remove.nas = remove.nas, resample.fun = resample.fun))\r\n> 6: evalOptimizationStates(learner, task, resampling, measures, par.set, \r\n>        NULL, control, opt.path, show.info, states, dobs, eols, remove.nas, \r\n>        resample.fun, \"mlr.tuneParams\")\r\n> 5: evalOptimizationStatesTune(learner, task, resampling, measures, \r\n>        par.set, control, opt.path, show.info, xs, dobs = seq_along(xs), \r\n>        eols = NA_integer_, remove.nas = TRUE, resample.fun = resample.fun)\r\n> 4: sel.func(learner, task, resampling, measures, par.set, control, \r\n>        opt.path, show.info, resample.fun)\r\n> 3: tuneParams(learner = lgb_learner, task = trainTask, resampling = resample_desc, \r\n>        par.set = lgb_params_set, control = control, measures = rmse) at #3\r\n> 2: tryCatchList(expr, classes, parentenv, handlers)\r\n> 1: tryCatch({\r\n>        lgb_params_tuning <- tuneParams(learner = lgb_learner, task = trainTask, \r\n>            resampling = resample_desc, par.set = lgb_params_set, \r\n>            control = control, measures = rmse)\r\n>    }, finally = {\r\n>        print(\"Traceback below:\")\r\n>        print(traceback())\r\n>        print(\"Warnings below:\")\r\n>        print(warnings())\r\n>    })\r\n> \r\n\r\n**additional sources on this problem (sorted according to information gain):**\r\n- https://github.com/stan-dev/rstan/issues/243\r\n- https://github.com/HenrikBengtsson/future/issues/143\r\n- https://github.com/kenkellner/jagsUI/issues/8\r\n**about how to do parallel in r** (there is an information about this error stating: \"as you can see below when we create an intermediate variable that takes up storage space:\")\r\n- http://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/\r\n\r\n**sessionInfo:**\r\nR version 3.5.0 (2018-04-23)\r\nPlatform: x86_64-w64-mingw32/x64 (64-bit)\r\nRunning under: Windows 10 x64 (build 17134)\r\n\r\nMatrix products: default\r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n [1] **parallelMap_1.3**   bindrcpp_0.2.2    stringi_1.1.7     checkmate_1.8.5   BBmisc_1.11      \r\n [6] knitr_1.20        **lightgbm_2.2.3**    R6_2.3.0          forcats_0.3.0     stringr_1.3.1    \r\n[11] dplyr_0.7.7       purrr_0.2.5       readr_1.1.1       tidyr_0.8.2       tibble_1.4.2     \r\n[16] ggplot2_3.1.0     tidyverse_1.2.1   **mlr_2.13**          ParamHelpers_1.11 logging_0.7-103  \r\n\r\nloaded via a namespace (and not attached):\r\n [1] tidyselect_0.2.5  splines_3.5.0     haven_1.1.2       lattice_0.20-35   colorspace_1.3-2 \r\n [6] survival_2.41-3   XML_3.98-1.16     rlang_0.3.0.1     pillar_1.2.2      withr_2.1.2      \r\n[11] glue_1.3.0        modelr_0.1.2      readxl_1.1.0      bindr_0.1.1       plyr_1.8.4       \r\n[16] munsell_0.4.3     gtable_0.2.0      cellranger_1.1.0  rvest_0.3.2       **parallel_3.5.0**   \r\n[21] broom_0.5.0       Rcpp_0.12.17      scales_0.5.0      backports_1.1.2   jsonlite_1.5     \r\n[26] fastmatch_1.1-0   hms_0.4.2         grid_3.5.0        cli_1.0.0         tools_3.5.0      \r\n[31] magrittr_1.5      lazyeval_0.2.1    crayon_1.3.4      pkgconfig_2.0.2   Matrix_1.2-14    \r\n[36] data.table_1.11.8 xml2_1.2.0        lubridate_1.7.4   assertthat_0.2.0  httr_1.3.1       \r\n[41] rstudioapi_0.7    nlme_3.1-137      compiler_3.5.0  \r\n\r\n\r\n**Code:**\r\n```r\r\n# loading libraries ----------------------------------------------------------------------------------------------\r\nlibrary(mlr)\r\nconfigureMlr(on.learner.error = 'warn', on.error.dump = T)\r\n\r\nlibrary(tidyverse)\r\nlibrary(lightgbm)\r\nlibrary(knitr)\r\n\r\n# fixing mlr method -----------------------------------------------------------------------------------------------\r\nlibrary(BBmisc)\r\nlibrary(checkmate)\r\nlibrary(stringi)\r\n\r\nmakeParamHelpList <- function(funs, pkgs, par.set) {\r\n  help.list = list()\r\n  par.ids = getParamIds(par.set)\r\n  pkgs = stri_replace_all(pkgs, \"\", regex = \"[+!_]\")\r\n  h = NULL\r\n  for (f in rev(funs)) {\r\n    for (pkg_ref in pkgs) {\r\n      h = utils::help(f, package = (pkg_ref))\r\n      if (length(h) > 0) {\r\n        break\r\n      }\r\n    }\r\n    if (length(h) == 0) {\r\n      # catf(\"nothing found: %s\", f)\r\n      next\r\n    }\r\n    ghf = get(\".getHelpFile\", mode = \"function\", envir = getNamespace(\"utils\"))\r\n    html = capture.output(tools::Rd2HTML(ghf(h)))\r\n    html = XML::htmlParse(html)\r\n    # try to extract the 'R argblock' table\r\n    tab = XML::getNodeSet(html, \"//table[@summary='R argblock']\")\r\n    if (length(tab) < 1) {\r\n      next\r\n    }\r\n    tbl = do.call(rbind, lapply(tab, function(t) {\r\n      tbl = XML::readHTMLTable(t, header = FALSE, stringsAsFactors = FALSE)\r\n      if (identical(ncol(tbl), 2L)) {\r\n        tbl\r\n      }\r\n    }))\r\n    # we also try to extract generally all lists that begin with a code string.\r\n    # this gets appended to the front, so that the (more trustworthy) html table\r\n    # extract will overwrite the <li>-extract if a param occurs in both.\r\n    tbl = rbind(codeListToTable(html), tbl)\r\n    if (is.null(tbl)) {\r\n      next\r\n    }\r\n    for (row in seq_len(nrow(tbl))) {\r\n      # the following loop handles occasions where multiple parameters are mentioned in\r\n      # one row, separated by commas.\r\n      for (par.name in stri_split(tbl[row, 1], regex = \", *\")[[1]]) {\r\n        if (par.name %in% par.ids) {\r\n          help.list[[par.name]] = stri_join(\"Argument of: \",\r\n                                            pkg_ref, \"::\", f, \"\\n\\n\", prepareString(tbl[row, 2]))\r\n        } else {\r\n          # catf(\"not interesting: %s par %s\", f, par.name)\r\n        }\r\n      }\r\n    }\r\n  }\r\n  return(help.list)\r\n}\r\n\r\n\r\nmakeLearnerBaseConstructor <-  function(classes, id, type, package, properties, par.set, par.vals, predict.type) {\r\n  if (length(par.vals) == 0L)\r\n    names(par.vals) = character(0L)\r\n  \r\n  learner = makeS3Obj(c(classes, \"Learner\"),\r\n                      id = id,\r\n                      type = type,\r\n                      package = package,\r\n                      properties = unique(properties),\r\n                      par.set = par.set,\r\n                      par.vals = par.vals,\r\n                      predict.type = predict.type\r\n  )\r\n  return(learner)\r\n}\r\n\r\nmakeRLearnerInternal <-  function(id, type, package, par.set, par.vals, properties,\r\n                                  name = id, short.name = id, note = \"\", callees) {\r\n  \r\n  # must do that before accessing par.set\r\n  # one case where lazy eval is actually helpful...\r\n  assertCharacter(package, any.missing = FALSE)\r\n  requirePackages(package, why = stri_paste(\"learner\", id, sep = \" \"), default.method = \"load\")\r\n  \r\n  assertString(id)\r\n  assertChoice(type, choices = c(\"classif\", \"regr\", \"multilabel\", \"surv\", \"cluster\", \"costsens\"))\r\n  assertSubset(properties, listLearnerProperties(type))\r\n  assertClass(par.set, classes = \"ParamSet\")\r\n  checkListElementClass(par.set$pars, \"LearnerParam\")\r\n  assertList(par.vals)\r\n  if (!isProperlyNamed(par.vals))\r\n    stop(\"Argument par.vals must be a properly named list!\")\r\n  assertString(name)\r\n  assertString(short.name)\r\n  assertString(note)\r\n  assertCharacter(callees, any.missing = FALSE)\r\n  learner = makeLearnerBaseConstructor(\"RLearner\",\r\n                                       id = id,\r\n                                       type = type,\r\n                                       package = package,\r\n                                       properties = unique(properties),\r\n                                       par.set = par.set,\r\n                                       par.vals = par.vals,\r\n                                       predict.type = \"response\"\r\n  )\r\n  learner$name = name\r\n  learner$short.name = short.name\r\n  learner$note = note\r\n  learner$callees = callees\r\n  learner$help.list = makeParamHelpList(callees, package, par.set)\r\n  learner$fix.factors.prediction = F\r\n  return(learner)\r\n  \r\n}\r\n\r\n\r\n\r\n# preparing data --------------------------------------------------------------------------------------------------\r\n\r\nprepare_data_for_ml_tasks <- function(train_orig, test_orig) {\r\n  train <- train_orig %>%\r\n    mutate(dataset = \"train\")\r\n  \r\n  test <- test_orig %>%\r\n    mutate(dataset = \"test\")\r\n  \r\n  combined <- bind_rows(train, test)\r\n  \r\n  combined <- combined %>%\r\n    mutate_at(\r\n      .vars = vars(\"Product_ID\", \"Gender\", \"Age\", \"Occupation\", \"City_Category\", \"Stay_In_Current_City_Years\", \"Marital_Status\", \"Product_Category_1\", \"Product_Category_2\", \"Product_Category_3\"),\r\n      .funs = funs(as.factor(.))\r\n    ) %>%\r\n    select(-c(User_ID, Product_Category_2, Product_Category_3))\r\n  \r\n  train <- combined %>%\r\n    filter(dataset == \"train\") %>%\r\n    select(-dataset)\r\n  \r\n  test <- combined %>%\r\n    filter(dataset == \"test\") %>%\r\n    select(-dataset)\r\n  \r\n  train <- train %>%\r\n    mutate_at(\r\n      .vars = colnames(Filter(is.factor, train)),\r\n      .funs = funs(as.integer(.))\r\n    )\r\n  return(list(\r\n    train = train,\r\n    test = test\r\n  ))\r\n}\r\n\r\ntrain_orig <- read_csv(\"./data/train.csv\")\r\ntest_orig <- read_csv(\"./data/test.csv\")\r\n\r\ndata <- prepare_data_for_ml_tasks(train_orig, test_orig)\r\n\r\ntrain <- data$train\r\ntest <- data$test\r\n\r\n# creating mlr lightgbm integration -------------------------------------------------------------------------------\r\n\r\nmakeRLearnerRegr <-  function(cl, package, par.set, par.vals = list(), properties = character(0L), name = cl, short.name = cl, note = \"\", callees = character(0L)) {\r\n  addClasses(\r\n    makeRLearnerInternal(cl, \"regr\", package, par.set, par.vals, properties, name, short.name, note, callees),\r\n    c(cl, \"RLearnerRegr\")\r\n  )\r\n}\r\n\r\nmakeRLearner.regr.lightgbm <-  function() {\r\n  makeRLearnerRegr(\r\n    cl = \"regr.lightgbm\",\r\n    package = \"lightgbm\",\r\n    par.set = makeParamSet(\r\n      makeUntypedLearnerParam(\"early.stopping.data\"),\r\n      makeIntegerLearnerParam(\"nrounds\", lower = 1, default = 100),\r\n      makeDiscreteLearnerParam(\"metric\", values = c(\"l1\", \"l2\", \"l2_root\", \"quantile\", \"mape\", \"huber\", \"fair\")),\r\n      makeDiscreteLearnerParam(\"obj\", values = c(\"regression_l2\", \"regression_l1\", \"huber\", \"fair\", \"poisson\", \"quantile\", \"mape\", \"gamma\", \"tweedie\", default = \"regression_l2\")),\r\n      makeIntegerLearnerParam(\"verbose\", lower = -1, upper = 2, tunable = FALSE, default = 2),\r\n      makeLogicalLearnerParam(\"record\", default = TRUE, tunable = FALSE),\r\n      makeIntegerLearnerParam(\"eval_freq\", lower = 1, tunable = FALSE, requires = quote(verbose > 0)),\r\n      makeUntypedLearnerParam(\"init_model\"),\r\n      makeIntegerLearnerParam(\"early_stopping_rounds\", lower = 1),\r\n      makeDiscreteLearnerParam(\"boosting\", values = c(\"gbdt\", \"dart\", \"rf\"), default = \"gbdt\", requires = quote(boosting != \"rf\" || bagging_freq > 0 && bagging_fraction < 1 && feature_fraction < 1)),\r\n      makeNumericLearnerParam(\"learning_rate\", lower = 0, upper = 1, default = 0.1),\r\n      makeIntegerLearnerParam(\"num_leaves\", lower = 1, default = 31),\r\n      makeDiscreteLearnerParam(\"tree_learner\", values = c(\"serial\", \"feature\", \"data\", \"voting\"), default = \"serial\"),\r\n      makeIntegerLearnerParam(\"num_threads\", lower = 1),\r\n      makeDiscreteLearnerParam(\"device\", values = c(\"cpu\", \"gpu\"), default = \"gpu\"),\r\n      makeIntegerLearnerParam(\"max_depth\", lower = -1, default = -1),\r\n      makeIntegerLearnerParam(\"min_data_in_leaf\", lower = 1, default = 20),\r\n      makeNumericLearnerParam(\"min_sum_hessian_in_leaf\", lower = 0, default = 1e-3),\r\n      makeNumericLearnerParam(\"feature_fraction\", lower = 0, upper = 1, default = 1),\r\n      makeNumericLearnerParam(\"bagging_fraction\", lower = 0, upper = 1, default = 1),\r\n      makeIntegerLearnerParam(\"bagging_freq\", lower = 0, default = 0),\r\n      makeNumericLearnerParam(\"lambda_l1\", lower = 0, default = 0),\r\n      makeNumericLearnerParam(\"lambda_l2\", lower = 0, default = 0),\r\n      makeNumericLearnerParam(\"min_split_gain\", lower = 0, default = 0),\r\n      makeNumericLearnerParam(\"drop_rate\", lower = 0, upper = 1, default = 0.1, requires = quote(boosting == \"dart\")),\r\n      makeNumericLearnerParam(\"skip_drop\", lower = 0, upper = 1, default = 0.5, requires = quote(boosting == \"dart\")),\r\n      makeIntegerLearnerParam(\"max_drop\", lower = 0, default = 50, requires = quote(boosting == \"dart\")),\r\n      makeLogicalLearnerParam(\"xgboost_dart_mode\", default = FALSE),\r\n      makeIntegerLearnerParam(\"min_data_per_group\", lower = 1, default = 100),\r\n      makeIntegerLearnerParam(\"max_cat_threshold\", lower = 0, default = 32),\r\n      makeNumericLearnerParam(\"cat_l2\", lower = 0, default = 10),\r\n      makeIntegerLearnerParam(\"max_cat_to_onehot\", lower = 0, default = 4)\r\n    ),\r\n    properties = c(\"numerics\", \"weights\", \"featimp\", \"missings\", \"factors\"),\r\n    name = \"Light Gradient Boosting Machine\",\r\n    short.name = \"lightgbm\",\r\n    par.vals = list(obj = \"regression_l2\"),\r\n    note = \"\"\r\n  )\r\n}\r\n\r\npredictLearner.regr.lightgbm = function(.learner, .model, .newdata, ...) {\r\n  predict(.model$learner.model, as.matrix(.newdata))\r\n}\r\n\r\ntrainLearner.regr.lightgbm  <-  function(.learner, .task, .subset, .weights = NULL,  ...) {\r\n  f = getTaskDesc(.task)\r\n  data = getTaskData(.task, .subset,target.extra = TRUE)\r\n  lgb.data = lgb.Dataset(as.matrix(data$data), label = data$target)\r\n  lightgbm::lgb.train(\r\n    data = lgb.data,\r\n    objective = \"regression\",\r\n    # verbosity = 1,\r\n    verbose = 2,\r\n    record = TRUE,\r\n    # device = \"gpu\",\r\n    ...\r\n  )\r\n}\r\n\r\nregisterS3method(\"makeRLearner\", \"regr.lightgbm\", makeRLearner.regr.lightgbm)\r\nregisterS3method(\"trainLearner\", \"regr.lightgbm\", trainLearner.regr.lightgbm)\r\nregisterS3method(\"predictLearner\", \"regr.lightgbm\", predictLearner.regr.lightgbm)\r\n\r\n# instantiating mlr -----------------------------------------------------------------------------------------------\r\n\r\nlibrary(parallelMap)\r\nparallelStartSocket(cpus = 2)\r\nparallelExport('trainLearner.regr.lightgbm', 'predictLearner.regr.lightgbm', 'lgb.Dataset')\r\n\r\nrmse = makeMeasure(\r\n  id = \"rmse\",\r\n  minimize = TRUE,\r\n  best = 0,\r\n  worst = Inf,\r\n  properties = c(\"regr\", \"req.pred\", \"req.truth\"),\r\n  name = \"Root mean squared error\",\r\n  note = \"The RMSE is aggregated as sqrt(mean(rmse.vals.on.test.sets^2)). If you don't want that, you could also use `test.mean`.\",\r\n  fun = function(task, model, pred, feats, extra.args) {\r\n    measureRMSE(pred$data$truth, pred$data$response)\r\n  },\r\n  aggr = test.rmse\r\n)\r\n\r\nlgb_learner <- makeRLearner.regr.lightgbm()\r\ntrainTask <- makeRegrTask(data = data.frame(train), target = 'Purchase')\r\n\r\n# control <- makeTuneControlRandom(maxit = 5)\r\ncontrol <- makeTuneControlGrid()\r\nresample_desc <- makeResampleDesc(\"CV\", iters = 4)\r\nlgb_params_set <- makeParamSet(\r\n  makeIntegerParam(\"nrounds\", lower = 55, upper = 60)\r\n)\r\n\r\ntryCatch(\r\n  {\r\n    lgb_params_tuning <- tuneParams(\r\n      learner = lgb_learner,\r\n      task = trainTask,\r\n      resampling = resample_desc,\r\n      par.set = lgb_params_set,\r\n      control = control,\r\n      measures = rmse\r\n    )\r\n  },finally = {\r\n    # parallelMap::parallelStop()\r\n    print('Traceback below:')\r\n    print(traceback())\r\n    print('Warnings below:')\r\n    print(warnings())\r\n  }\r\n)\r\n```\r\n\r\n**output for multicore (whole output in [this file](https://github.com/mlr-org/mlr/files/2594025/parallel_mode_output.txt)):**\r\nLoading required package: ParamHelpers\r\n-- Attaching packages ---------------------------------------------------------- tidyverse 1.2.1 --\r\n<U+221A> ggplot2 3.1.0     <U+221A> purrr   0.2.5\r\n<U+221A> tibble  1.4.2     <U+221A> dplyr   0.7.7\r\n<U+221A> tidyr   0.8.2     <U+221A> stringr 1.3.1\r\n<U+221A> readr   1.1.1     <U+221A> forcats 0.3.0\r\n-- Conflicts ------------------------------------------------------------- tidyverse_conflicts() --\r\nx dplyr::filter() masks stats::filter()\r\nx dplyr::lag()    masks stats::lag()\r\nLoading required package: R6\r\n\r\nAttaching package: 'lightgbm'\r\n\r\nThe following object is masked from 'package:dplyr':\r\n\r\n    slice\r\n\r\n\r\nAttaching package: 'BBmisc'\r\n\r\nThe following objects are masked from 'package:dplyr':\r\n\r\n    coalesce, collapse\r\n\r\nThe following object is masked from 'package:base':\r\n\r\n    isFALSE\r\n\r\nParsed with column specification:\r\ncols(\r\n  User_ID = col_integer(),\r\n  Product_ID = col_character(),\r\n  Gender = col_character(),\r\n  Age = col_character(),\r\n  Occupation = col_integer(),\r\n  City_Category = col_character(),\r\n  Stay_In_Current_City_Years = col_character(),\r\n  Marital_Status = col_integer(),\r\n  Product_Category_1 = col_integer(),\r\n  Product_Category_2 = col_integer(),\r\n  Product_Category_3 = col_integer(),\r\n  Purchase = col_integer()\r\n)\r\nParsed with column specification:\r\ncols(\r\n  User_ID = col_integer(),\r\n  Product_ID = col_character(),\r\n  Gender = col_character(),\r\n  Age = col_character(),\r\n  Occupation = col_integer(),\r\n  City_Category = col_character(),\r\n  Stay_In_Current_City_Years = col_character(),\r\n  Marital_Status = col_integer(),\r\n  Product_Category_1 = col_integer(),\r\n  Product_Category_2 = col_integer(),\r\n  Product_Category_3 = col_integer()\r\n)\r\nStarting parallelization in mode=socket with cpus=2.\r\nExporting objects to slaves for mode socket: trainLearner.regr.lightgbm,predictLearner.regr.lightgbm,lgb.Dataset\r\n[Tune] Started tuning learner regr.lightgbm for parameter set:\r\n           Type len Def   Constr Req Tunable Trafo\r\nnrounds integer   -   - 55 to 60   -    TRUE     -\r\nWith control class: TuneControlGrid\r\nImputation value: Inf\r\nExporting objects to slaves for mode socket: .mlr.slave.options\r\nMapping in parallel: mode = socket; cpus = 2; elements = 6.\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.755429\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 9\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.170425\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 10\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9263.085131\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 10\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.755429\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 8\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9267.863867\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 9\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.170425\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 10\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9263.085131\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9267.863867\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 15\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.755429\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 8\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.170425\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.755429\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 11\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9262.170425\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 10\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 18\r\n[1] \"Traceback below:\"\r\nNo traceback available \r\nNULL\r\n[1] \"Warnings below:\"\r\nWarning messages:\r\n1: package 'mlr' was built under R version 3.5.1\r\n2: package 'ParamHelpers' was built under R version 3.5.1\r\n3: package 'tidyverse' was built under R version 3.5.1\r\n4: package 'ggplot2' was built under R version 3.5.1[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 8\r\n\r\n5: package 'tidyr' was built under R version 3.5.1\r\n6: package 'readr' was built under R version 3.5.1\r\n7: package 'purrr' was built under R version 3.5.1\r\n8: package 'dplyr' was built under R version 3.5.1\r\n9: package 'forcats' was built under R version 3.5.1\r\n10: package 'R6' was built under R version 3.5.1\r\n11: package 'BBmisc' was built under R version 3.5.1\r\n12: package 'bindrcpp' was built under R version 3.5.1\r\n13: package 'parallelMap' was built under R version 3.5.1\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 13\r\nB��d w poleceniu 'unserialize(node$con)':b��d czytania z po��czenia\r\nWywo�ania: sourceWithProgress ... FUN -> recvData -> recvData.SOCKnode -> unserialize\r\nDodatkowo: By�o 13 ostrze�enie (u�yj 'warnings()' aby je zobaczy�)\r\nWykonywanie wstrzymane\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 14\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 17\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n\r\n\r\n**Output for commented lines (whole output in [this file](https://github.com/mlr-org/mlr/files/2594022/single_core_output.txt)):**\r\n```r\r\nlibrary(parallelMap)\r\nparallelStartSocket(cpus = 2)\r\nparallelExport('trainLearner.regr.lightgbm', 'predictLearner.regr.lightgbm', 'lgb.Dataset')\r\n```\r\n\r\nLoading required package: ParamHelpers\r\n-- Attaching packages ---------------------------------------------------------- tidyverse 1.2.1 --\r\n<U+221A> ggplot2 3.1.0     <U+221A> purrr   0.2.5\r\n<U+221A> tibble  1.4.2     <U+221A> dplyr   0.7.7\r\n<U+221A> tidyr   0.8.2     <U+221A> stringr 1.3.1\r\n<U+221A> readr   1.1.1     <U+221A> forcats 0.3.0\r\n-- Conflicts ------------------------------------------------------------- tidyverse_conflicts() --\r\nx dplyr::filter() masks stats::filter()\r\nx dplyr::lag()    masks stats::lag()\r\nLoading required package: R6\r\n\r\nAttaching package: 'lightgbm'\r\n\r\nThe following object is masked from 'package:dplyr':\r\n\r\n    slice\r\n\r\n\r\nAttaching package: 'BBmisc'\r\n\r\nThe following objects are masked from 'package:dplyr':\r\n\r\n    coalesce, collapse\r\n\r\nThe following object is masked from 'package:base':\r\n\r\n    isFALSE\r\n\r\nParsed with column specification:\r\ncols(\r\n  User_ID = col_integer(),\r\n  Product_ID = col_character(),\r\n  Gender = col_character(),\r\n  Age = col_character(),\r\n  Occupation = col_integer(),\r\n  City_Category = col_character(),\r\n  Stay_In_Current_City_Years = col_character(),\r\n  Marital_Status = col_integer(),\r\n  Product_Category_1 = col_integer(),\r\n  Product_Category_2 = col_integer(),\r\n  Product_Category_3 = col_integer(),\r\n  Purchase = col_integer()\r\n)\r\nParsed with column specification:\r\ncols(\r\n  User_ID = col_integer(),\r\n  Product_ID = col_character(),\r\n  Gender = col_character(),\r\n  Age = col_character(),\r\n  Occupation = col_integer(),\r\n  City_Category = col_character(),\r\n  Stay_In_Current_City_Years = col_character(),\r\n  Marital_Status = col_integer(),\r\n  Product_Category_1 = col_integer(),\r\n  Product_Category_2 = col_integer(),\r\n  Product_Category_3 = col_integer()\r\n)\r\n[Tune] Started tuning learner regr.lightgbm for parameter set:\r\n           Type len Def   Constr Req Tunable Trafo\r\nnrounds integer   -   - 55 to 60   -    TRUE     -\r\nWith control class: TuneControlGrid\r\nImputation value: Inf\r\n[Tune-x] 1: nrounds=55\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9263.103679\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 9\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9272.597160\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 10\r\n[..]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 8\r\n[LightGBM] [Info] Total Bins 322\r\n[LightGBM] [Info] Number of data: 412551, number of used features: 8\r\n[LightGBM] [Info] Start training from score 9258.360421\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 12\r\n[...]\r\n[LightGBM] [Debug] Trained a tree with leaves = 31 and max_depth = 13\r\n"},"comment":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/comments/440598063","html_url":"https://github.com/mlr-org/mlr/issues/2492#issuecomment-440598063","issue_url":"https://api.github.com/repos/mlr-org/mlr/issues/2492","id":440598063,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDU5ODA2Mw==","user":{"login":"matkalinowski","id":28002118,"node_id":"MDQ6VXNlcjI4MDAyMTE4","avatar_url":"https://avatars3.githubusercontent.com/u/28002118?v=4","gravatar_id":"","url":"https://api.github.com/users/matkalinowski","html_url":"https://github.com/matkalinowski","followers_url":"https://api.github.com/users/matkalinowski/followers","following_url":"https://api.github.com/users/matkalinowski/following{/other_user}","gists_url":"https://api.github.com/users/matkalinowski/gists{/gist_id}","starred_url":"https://api.github.com/users/matkalinowski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matkalinowski/subscriptions","organizations_url":"https://api.github.com/users/matkalinowski/orgs","repos_url":"https://api.github.com/users/matkalinowski/repos","events_url":"https://api.github.com/users/matkalinowski/events{/privacy}","received_events_url":"https://api.github.com/users/matkalinowski/received_events","type":"User","site_admin":false},"created_at":"2018-11-21T09:37:49Z","updated_at":"2018-11-21T09:37:49Z","author_association":"NONE","body":"So here is the traceback for the parallel case:\r\n\r\n> 14: unserialize(node$con)\r\n13: recvData.SOCKnode(con)\r\n12: recvData(con)\r\n11: FUN(X[[i]], ...)\r\n10: lapply(cl[1:jobs], recvResult)\r\n9: staticClusterApply(cl, fun, vlen, argfun)\r\n8: clusterMap(cl = NULL, slaveWrapper, ..., .i = iters, MoreArgs = more.args,\r\nSIMPLIFY = FALSE, USE.NAMES = FALSE)\r\n7: parallelMap(evalOptimizationState, dobs, states, level = level,\r\nmore.args = list(learner = learner, task = task, resampling = resampling,\r\nmeasures = measures, par.set = par.set, bits.to.features = bits.to.features,\r\ncontrol = control, opt.path = opt.path, show.info = show.info,\r\nremove.nas = remove.nas, resample.fun = resample.fun))\r\n6: evalOptimizationStates(learner, task, resampling, measures, par.set,\r\nNULL, control, opt.path, show.info, states, dobs, eols, remove.nas,\r\nresample.fun, \"mlr.tuneParams\")\r\n5: evalOptimizationStatesTune(learner, task, resampling, measures,\r\npar.set, control, opt.path, show.info, xs, dobs = seq_along(xs),\r\neols = NA_integer_, remove.nas = TRUE, resample.fun = resample.fun)\r\n4: sel.func(learner, task, resampling, measures, par.set, control,\r\nopt.path, show.info, resample.fun)\r\n3: tuneParams(learner = lgb_learner, task = trainTask, resampling = resample_desc,\r\npar.set = lgb_params_set, control = control, measures = rmse) at #3\r\n2: tryCatchList(expr, classes, parentenv, handlers)\r\n1: tryCatch({\r\nlgb_params_tuning <- tuneParams(learner = lgb_learner, task = trainTask,\r\nresampling = resample_desc, par.set = lgb_params_set,\r\ncontrol = control, measures = rmse)\r\n}, finally = {\r\nprint(\"Traceback below:\")\r\nprint(traceback())\r\nprint(\"Warnings below:\")\r\nprint(warnings())\r\n})\r\n\r\nI couldn't get a traceback for the case where I comment those lines (basically turining off the parallelization as far as I would expect) but I wouldn't be suprised if it was the same error in two of those situations:\r\n```r\r\nlibrary(parallelMap)\r\nparallelStartSocket(cpus = 2)\r\nparallelExport('trainLearner.regr.lightgbm', 'predictLearner.regr.lightgbm', 'lgb.Dataset')\r\n```\r\n\r\nFor this data it fails everytime. Can you reproduce it? Do you need any other informations?"}}
{"actor":{"display_login":"matkalinowski"}}