{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/tfruns/issues/24","repository_url":"https://api.github.com/repos/rstudio/tfruns","labels_url":"https://api.github.com/repos/rstudio/tfruns/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/tfruns/issues/24/comments","events_url":"https://api.github.com/repos/rstudio/tfruns/issues/24/events","html_url":"https://github.com/rstudio/tfruns/issues/24","id":317841013,"number":24,"title":"Flags not saved upon completion of run","user":{"login":"mantisboxer","id":16738708,"avatar_url":"https://avatars2.githubusercontent.com/u/16738708?v=4","gravatar_id":"","url":"https://api.github.com/users/mantisboxer","html_url":"https://github.com/mantisboxer","followers_url":"https://api.github.com/users/mantisboxer/followers","following_url":"https://api.github.com/users/mantisboxer/following{/other_user}","gists_url":"https://api.github.com/users/mantisboxer/gists{/gist_id}","starred_url":"https://api.github.com/users/mantisboxer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mantisboxer/subscriptions","organizations_url":"https://api.github.com/users/mantisboxer/orgs","repos_url":"https://api.github.com/users/mantisboxer/repos","events_url":"https://api.github.com/users/mantisboxer/events{/privacy}","received_events_url":"https://api.github.com/users/mantisboxer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-26T00:54:16Z","updated_at":"2018-04-26T13:11:28Z","closed_at":null,"author_association":"NONE","body":"After running a tuning file calling from flags as set up below, I get a dataframe that contains only the training metrics, but not the flags they are associated with. Any idea what's going on? I'm planning to make several hundred runs and I'm really going to need to filter by the flag values to make sense of the results.\r\n\r\nOne other thing that might be relevant....I'm using a custom loss function (actually a wrapped Python function).\r\n\r\nThanks,\r\n\r\nMarlon\r\n=====================================================\r\nWarning messages:\r\nThere were 50 or more warnings (use warnings() to see the first 50)\r\n> warnings()\r\nWarning messages:\r\n1: Unable to log model info: AttributeError: 'function' object has no attribute 'func_name'\r\n2: Unable to log model info: AttributeError: 'function' object has no attribute 'func_name'\r\n3: Unable to log model info: AttributeError: 'function' object has no attribute 'func_name'\r\n....\r\n\r\n\r\nHere are my flags:\r\nFLAGS <- flags(\r\n  flag_integer(\"nb_units\",20),\r\n  flag_integer(\"mn_units\",20),\r\n  flag_integer(\"lg_units\",20),\r\n  flag_integer(\"num_epochs\",10),\r\n  flag_numeric(\"mn_l1\",0.001),\r\n  flag_numeric(\"mn_l2\",0.001),\r\n  flag_numeric(\"val_split\",0.3),\r\n  flag_numeric(\"tst_split\",0.7),\r\n  flag_string(\"target\",\"PKPC\"),\r\n  flag_string(\"key_metric\",\"Melt_Visc_STD\")\r\n)\r\n\r\nHere is my call to tuning_run:\r\ncert_tune <- tuning_run(\"cert_yield_pred.R\",runs_dir = \"cert_yield_tune_tst_2\",\r\n                        flags = list(\r\n                          num_epochs=c(10),\r\n                          mn_l1=c(0.001,0.01,0.1),\r\n                          mn_l2=c(0.001,0.01,0.1),\r\n                          key_metric=c(\"Melt_Flow_Rate\")\r\n                        ))\r\n\r\nHere is the structure of the cert_tune frame:\r\nClasses ‘tfruns_runs_df’ and 'data.frame':\t9 obs. of  23 variables:\r\n $ run_dir                          : chr  \"cert_yield_tune_tst_2/2018-04-25T21-27-43Z\" \"cert_yield_tune_tst_2/2018-04-25T21-27-30Z\" \"cert_yield_tune_tst_2/2018-04-25T21-27-16Z\" \"cert_yield_tune_tst_2/2018-04-25T21-27-02Z\" ...\r\n $ metric_loss                      : num  1.052 0.927 1.245 2.502 2.872 ...\r\n $ metric_cert_yield_nb_out_loss    : num  0.976 0.896 1.088 0.926 1.2 ...\r\n $ metric_cert_yield_mn_out_loss    : num  0.808 0.692 0.998 2.237 2.632 ...\r\n $ metric_cert_yield_lg_out_loss    : num  1.55 4.71 7.35 2.66 2.79 ...\r\n $ metric_val_loss                  : num  1.6 1.94 2.83 13.93 3.84 ...\r\n $ metric_val_cert_yield_nb_out_loss: num  2.37 1.32 2.71 2.17 2.74 ...\r\n $ metric_val_cert_yield_mn_out_loss: num  1.35 1.71 2.59 13.67 3.6 ...\r\n $ metric_val_cert_yield_lg_out_loss: num  2.97 4.35 2.84 10.94 4.05 ...\r\n $ samples                          : int  800 800 800 800 800 800 800 800 800\r\n $ validation_samples               : int  344 344 344 344 344 344 344 344 344\r\n $ batch_size                       : int  126 126 126 126 126 126 126 126 126\r\n $ epochs                           : int  10 10 10 10 10 10 10 10 10\r\n $ epochs_completed                 : int  10 10 10 10 10 10 10 10 10\r\n $ metrics                          : chr  \"cert_yield_tune_tst_2/2018-04-25T21-27-43Z/tfruns.d/metrics.json\" \"cert_yield_tune_tst_2/2018-04-25T21-27-30Z/tfruns.d/metrics.json\" \"cert_yield_tune_tst_2/2018-04-25T21-27-16Z/tfruns.d/metrics.json\" \"cert_yield_tune_tst_2/2018-04-25T21-27-02Z/tfruns.d/metrics.json\" ...\r\n $ script                           : chr  \"cert_yield_pred.R\" \"cert_yield_pred.R\" \"cert_yield_pred.R\" \"cert_yield_pred.R\" ...\r\n $ start                            : POSIXct, format: \"2018-04-25 21:27:47\" \"2018-04-25 21:27:34\" \"2018-04-25 21:27:20\" \"2018-04-25 21:27:07\" ...\r\n $ end                              : POSIXct, format: \"2018-04-25 21:27:55\" \"2018-04-25 21:27:43\" \"2018-04-25 21:27:29\" \"2018-04-25 21:27:16\" ...\r\n $ completed                        : logi  TRUE TRUE TRUE TRUE TRUE TRUE ...\r\n $ output                           : chr  \"\\n> resin_raw_set <- total_full_norm_raw %>% filter(!is.na(!!!syms(FLAGS$key_metric))) %>% group_by(Workcenter,\"| __truncated__ \"\\n> resin_raw_set <- total_full_norm_raw %>% filter(!is.na(!!!syms(FLAGS$key_metric))) %>% group_by(Workcenter,\"| __truncated__ \"\\n> resin_raw_set <- total_full_norm_raw %>% filter(!is.na(!!!syms(FLAGS$key_metric))) %>% group_by(Workcenter,\"| __truncated__ \"\\n> resin_raw_set <- total_full_norm_raw %>% filter(!is.na(!!!syms(FLAGS$key_metric))) %>% group_by(Workcenter,\"| __truncated__ ...\r\n $ source_code                      : chr  \"cert_yield_tune_tst_2/2018-04-25T21-27-43Z/tfruns.d/source.tar.gz\" \"cert_yield_tune_tst_2/2018-04-25T21-27-30Z/tfruns.d/source.tar.gz\" \"cert_yield_tune_tst_2/2018-04-25T21-27-16Z/tfruns.d/source.tar.gz\" \"cert_yield_tune_tst_2/2018-04-25T21-27-02Z/tfruns.d/source.tar.gz\" ...\r\n $ context                          : chr  \"local\" \"local\" \"local\" \"local\" ...\r\n $ type                             : chr  \"training\" \"training\" \"training\" \"training\" ...\r\n - attr(*, \"original_cols\")= chr  \"run_dir\" \"metric_loss\" \"metric_cert_yield_nb_out_loss\" \"metric_cert_yield_mn_out_loss\" ..."},"comment":{"url":"https://api.github.com/repos/rstudio/tfruns/issues/comments/384635581","html_url":"https://github.com/rstudio/tfruns/issues/24#issuecomment-384635581","issue_url":"https://api.github.com/repos/rstudio/tfruns/issues/24","id":384635581,"user":{"login":"mantisboxer","id":16738708,"avatar_url":"https://avatars2.githubusercontent.com/u/16738708?v=4","gravatar_id":"","url":"https://api.github.com/users/mantisboxer","html_url":"https://github.com/mantisboxer","followers_url":"https://api.github.com/users/mantisboxer/followers","following_url":"https://api.github.com/users/mantisboxer/following{/other_user}","gists_url":"https://api.github.com/users/mantisboxer/gists{/gist_id}","starred_url":"https://api.github.com/users/mantisboxer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mantisboxer/subscriptions","organizations_url":"https://api.github.com/users/mantisboxer/orgs","repos_url":"https://api.github.com/users/mantisboxer/repos","events_url":"https://api.github.com/users/mantisboxer/events{/privacy}","received_events_url":"https://api.github.com/users/mantisboxer/received_events","type":"User","site_admin":false},"created_at":"2018-04-26T13:11:28Z","updated_at":"2018-04-26T13:11:28Z","author_association":"NONE","body":"Here is my script:\r\n\r\n```\r\nresin_raw_set <- total_full_norm_raw %>% filter(!is.na(!!!syms(FLAGS$key_metric))) %>% group_by(Workcenter,M__ex_PRT,Base_Component,Material,!!!syms(FLAGS$key_metric),Parts_Cyc) %>% \r\n  summarize_at(vars(Total,Process_Scrap,RT_Total),funs(sum),na.rm=T) %>% ungroup() %>% filter(!is.na(Parts_Cyc),Total>0) %>% mutate(PKPC=ifelse(Total!=0,ceiling(Process_Scrap/(Total/1000)),0),PHR=ifelse(RT_Total!=0,ceiling(Process_Scrap/(RT_Total)),0))\r\n\r\nif(FLAGS$target==\"Process_Scrap\"){\r\n  core_factors <- c(\"Workcenter\",\"M__ex_PRT\",\"Base_Component\",\"Material\",\"Total\",\"RT_Total\")\r\n}else{\r\n  core_factors <- c(\"Workcenter\",\"M__ex_PRT\",\"Base_Component\",\"Material\")\r\n}\r\nset.seed(42)\r\nraw_split <- initial_split(resin_raw_set %>% mutate_at(vars(Base_Component,Material),funs(as.character(.))) %>% select(!!!syms(c(core_factors,FLAGS$key_metric,FLAGS$target))),prop = FLAGS$tst_split)\r\nraw_train <- training(raw_split)\r\nraw_test <- testing(raw_split)\r\n\r\nrec_form <- as.formula(paste0(FLAGS$target,\"~.\"))\r\n\r\nraw_recipe <- recipe(rec_form,data = raw_train) %>% \r\n  step_zv(all_predictors(),-all_outcomes()) %>% \r\n  step_other(all_nominal(),-all_outcomes(),threshold = 0.01) %>%\r\n  step_dummy(all_nominal(),-all_outcomes()) %>% \r\n  step_center(all_predictors(),-all_outcomes()) %>% step_scale(all_predictors(),-all_outcomes()) %>% prep(data = raw_train)\r\n\r\nx_raw_train <- bake(raw_recipe,raw_train) %>% select(-one_of(FLAGS$target))\r\nx_raw_test <- bake(raw_recipe,raw_test) %>% select(-one_of(FLAGS$target))\r\nx_raw_trn_resp <- raw_train %>% pull(FLAGS$target)\r\nx_raw_tst_resp <- raw_test %>% pull(FLAGS$target)\r\n\r\ncert_yield_in <- layer_input(shape = c(ncol(x_raw_train)),name = \"cert_yield_in\")\r\ncert_yield_nb <- cert_yield_in %>% layer_dense(FLAGS$nb_units,activation = \"softplus\", name = \"cert_yield_nb\")\r\ncert_yield_nb_out <- cert_yield_nb %>% layer_dense(1,activation = \"softplus\",name = \"cert_yield_nb_out\")\r\ncert_yield_mn <- cert_yield_in %>% layer_dense(FLAGS$mn_units,activation = tf$exp, kernel_regularizer = regularizer_l1_l2(l1=FLAGS$mn_l1,l2=FLAGS$mn_l2), name = \"cert_yield_mn\")\r\ncert_yield_mn_out <- cert_yield_mn %>% layer_dense(1,activation = tf$exp,name = \"cert_yield_mn_out\")\r\ncert_yield_lg <- cert_yield_in %>% layer_dense(FLAGS$lg_units,activation = \"sigmoid\", name = \"cert_yield_lg\")\r\ncert_yield_lg_out <- cert_yield_mn %>% layer_dense(1,activation = \"sigmoid\",name = \"cert_yield_lg_out\")\r\n\r\ncert_yield_model <- keras_model(inputs=list(cert_yield_in),outputs = list(cert_yield_nb_out,cert_yield_mn_out,cert_yield_lg_out))\r\ncert_yield_loss <- py$ZINB(pi=cert_yield_lg_out, theta=cert_yield_nb_out)\r\ncert_yield_model$compile(optimizer = optimizer_adadelta(),loss = cert_yield_loss$loss,loss_weights = list(0,1,0))\r\ncert_yield_history <- fit(cert_yield_model,x=list(as.matrix(x_raw_train)),\r\n                      y=list(x_raw_trn_resp,x_raw_trn_resp,x_raw_trn_resp),\r\n                      validation_split = FLAGS$val_split,\r\n                      batch_size = 126,\r\n                      epochs = FLAGS$num_epochs)\r\n```\r\n\r\n\r\nI am indeed using a custom loss function. I'm trying to predict counts, so using a loss for zero inflated negative binomal. The code is actually Python code that I'm wrapping with reticulate. I've struggled to get an R version of this working properly. The code is in two parts - the first is the negative binomial loss formulation.\r\n\r\n```\r\nclass NB(object):\r\n    def __init__(self, theta=None, theta_init=[0.0], scale_factor=1.0, scope='nbinom_loss/', debug=False, **theta_kwargs):\r\n                          # for numerical stability\r\n                          self.eps = 1e-10\r\n                          self.scale_factor = scale_factor\r\n                          self.debug = debug\r\n                          self.scope = scope\r\n                          \r\n                          with tf.name_scope(self.scope):\r\n                              # a variable may be given by user or it can be created here\r\n                              if theta is None:\r\n                                theta = tf.Variable(theta_init, dtype=tf.float32,\r\n                                name='theta', **theta_kwargs)\r\n                              \r\n                              # keep a reference to the variable itself\r\n                              self.theta_variable = theta\r\n                              \r\n                              # to keep dispersion always non-negative\r\n                              self.theta = tf.nn.softplus(theta)\r\n                          \r\n    def loss(self, y_true, y_pred, reduce=True):\r\n        scale_factor = self.scale_factor\r\n        eps = self.eps\r\n\r\n        with tf.name_scope(self.scope):\r\n            y_true = tf.cast(y_true, tf.float32)\r\n            y_pred = tf.cast(y_pred, tf.float32) * scale_factor\r\n                                  \r\n            theta = 1.0/(self.theta+eps)\r\n                                  \r\n            t1 = -tf.lgamma(y_true+theta+eps) \r\n            t2 = tf.lgamma(theta+eps)\r\n            t3 = tf.lgamma(y_true+1.0) \r\n            t4 = -(theta * (tf.log(theta+eps)))\r\n            t5 = -(y_true * (tf.log(y_pred+eps)))\r\n            t6 = (theta+y_true) * tf.log(theta+y_pred+eps)      \r\n                                  \r\n            if self.debug:\r\n                tf.summary.histogram('t1', t1)\r\n                tf.summary.histogram('t2', t2)\r\n                tf.summary.histogram('t3', t3)\r\n                tf.summary.histogram('t4', t4)\r\n                tf.summary.histogram('t5', t5)\r\n                tf.summary.histogram('t6', t6)\r\n                              \r\n            final = t1 + t2 + t3 + t4 + t5 + t6\r\n                              \r\n            if reduce:\r\n                final = tf.reduce_mean(final)\r\n                              \r\n        return final\r\n```\r\n\r\nThe second basically decides whether to pick a zero or some value from the negative binomial distribution (which could also be zero):\r\n\r\n\r\n```\r\nfrom keras.engine.topology import Layer\r\nfrom keras.layers import Lambda\r\nfrom keras import backend as K\r\nimport tensorflow as tf\r\n\r\nclass ZINB(NB):\r\n  def __init__(self, pi, scope='zinb_loss/', **kwargs):\r\n        super().__init__(scope=scope, **kwargs)\r\n        self.pi = pi\r\n                \r\n  def loss(self, y_true, y_pred):\r\n      scale_factor = self.scale_factor\r\n      eps = self.eps\r\n                \r\n      with tf.name_scope(self.scope):\r\n          # reuse existing NB neg.log.lik.\r\n          nb_case = super().loss(y_true, y_pred, reduce=False) - tf.log(1.0-self.pi+eps)\r\n                \r\n          y_true = tf.cast(y_true, tf.float32)\r\n          y_pred = tf.cast(y_pred, tf.float32) * scale_factor\r\n          theta = 1.0/(self.theta+eps)\r\n                \r\n          zero_nb = tf.pow(theta/(theta+y_pred+eps), theta)\r\n          zero_case = -tf.log(self.pi + ((1.0-self.pi)*zero_nb)+eps)\r\n          result = tf.where(tf.less(y_true, 1e-8), zero_case, nb_case)\r\n          result = tf.reduce_mean(result)\r\n                \r\n          if self.debug:\r\n              tf.summary.histogram('nb_case', nb_case)\r\n              tf.summary.histogram('zero_nb', zero_nb)\r\n              tf.summary.histogram('zero_case', zero_case)\r\n                \r\n      return result\r\n```\r\n\r\nThe code is based on a tutorial I found [here](https://github.com/gokceneraslan/neuralnet_countmodels/blob/master/Count%20models%20with%20neuralnets.ipynb)."}}
{"actor":{"display_login":"mantisboxer"}}