{"action":"created","issue":{"url":"https://api.github.com/repos/IDSIA/sacred/issues/291","repository_url":"https://api.github.com/repos/IDSIA/sacred","labels_url":"https://api.github.com/repos/IDSIA/sacred/issues/291/labels{/name}","comments_url":"https://api.github.com/repos/IDSIA/sacred/issues/291/comments","events_url":"https://api.github.com/repos/IDSIA/sacred/issues/291/events","html_url":"https://github.com/IDSIA/sacred/issues/291","id":323196904,"number":291,"title":"Overwrite in MongoObserver deletes run info","user":{"login":"JarnoRFB","id":8550270,"avatar_url":"https://avatars0.githubusercontent.com/u/8550270?v=4","gravatar_id":"","url":"https://api.github.com/users/JarnoRFB","html_url":"https://github.com/JarnoRFB","followers_url":"https://api.github.com/users/JarnoRFB/followers","following_url":"https://api.github.com/users/JarnoRFB/following{/other_user}","gists_url":"https://api.github.com/users/JarnoRFB/gists{/gist_id}","starred_url":"https://api.github.com/users/JarnoRFB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JarnoRFB/subscriptions","organizations_url":"https://api.github.com/users/JarnoRFB/orgs","repos_url":"https://api.github.com/users/JarnoRFB/repos","events_url":"https://api.github.com/users/JarnoRFB/events{/privacy}","received_events_url":"https://api.github.com/users/JarnoRFB/received_events","type":"User","site_admin":false},"labels":[{"id":89407952,"url":"https://api.github.com/repos/IDSIA/sacred/labels/feature","name":"feature","color":"009800","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-15T12:31:41Z","updated_at":"2018-05-16T15:38:04Z","closed_at":null,"author_association":"NONE","body":"At my university we have a grid system on which computation jobs can be scheduled. After a certain wall time (usually 1:30h) the job is killed and has to be restarted. To use sacred and still write logs to same experiment we want to use the `overwrite` parameter of the `MongoObserver` as recommended in #144. However, this leads to `run.info` being empty in the mongo db once the run is overwritten. This is especially bad since information about metrics is lost, so that the metrics plots won't be displayed in sacredboard, even so the metrics are there in the metrics collection.\r\n\r\nA more direct way to resume experiments than overwriting would be preferred still. "},"comment":{"url":"https://api.github.com/repos/IDSIA/sacred/issues/comments/389564897","html_url":"https://github.com/IDSIA/sacred/issues/291#issuecomment-389564897","issue_url":"https://api.github.com/repos/IDSIA/sacred/issues/291","id":389564897,"user":{"login":"JarnoRFB","id":8550270,"avatar_url":"https://avatars0.githubusercontent.com/u/8550270?v=4","gravatar_id":"","url":"https://api.github.com/users/JarnoRFB","html_url":"https://github.com/JarnoRFB","followers_url":"https://api.github.com/users/JarnoRFB/followers","following_url":"https://api.github.com/users/JarnoRFB/following{/other_user}","gists_url":"https://api.github.com/users/JarnoRFB/gists{/gist_id}","starred_url":"https://api.github.com/users/JarnoRFB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JarnoRFB/subscriptions","organizations_url":"https://api.github.com/users/JarnoRFB/orgs","repos_url":"https://api.github.com/users/JarnoRFB/repos","events_url":"https://api.github.com/users/JarnoRFB/events{/privacy}","received_events_url":"https://api.github.com/users/JarnoRFB/received_events","type":"User","site_admin":false},"created_at":"2018-05-16T15:38:04Z","updated_at":"2018-05-16T15:38:04Z","author_association":"NONE","body":"Thanks for your response!\r\n\r\n> Overwriting the info dict is intended behavior\r\n\r\nI understand that `overwrite` should lead to everything being overwritten, but why does the `info dict` end up being empty in the second run?\r\n\r\n> We could provide an option to resume an experiment by id, such as --resume 47. Observers could then use that to append to the respective info and metrics.\r\n\r\nThat's exactly what would be needed. For us it would be important to be able to pass that option directly to the observer similar to `overwrite` as the restart usually happens programmatically.  \r\n\r\nHowever, I noted that metrics are already appended, even with a duplicate step index. Not sure if this is the intended behavior.\r\n\r\n> how to handle differences in the rest of the entry? Maybe one of the dependencies was upgraded, or the code changed. Or even the config. Should Sacred refuse to continue in this case? That seems to be a good safeguard against mistakenly continuing the wrong experiment.\r\n\r\nYes it would make sense to check against differences in the sources and especially the config. Changes would mean to me that it is no longer the same experiment, so refusing to resume in this case makes sense.\r\n\r\n> But what if you had to fix a minor problem in the code, or update some library because of a bug? Maybe we would also need a --force-resume option?\r\n\r\nThis could make sense for changes in the sources. However, I think it is not a major requirement right now, as the main use case for us would be to continue experiments that were interrupted due to external factors (wall time) rather than internal factors (software bugs). A useful `--resume` feature could be implemented first, without a `--force-resume` option.\r\n\r\n> Most code cannot be interrupted and continued at any point, but instead relies on checkpoints. What to do with info entries and metrics between the checkpoint and the interruption? They would be duplicated in an naive implementation. Maybe sacred should have a way of storing the times of the last checkpoint, so that --resume would overwrite all entries after that checkpoint time?\r\n\r\nFor our use case (training networks with tf.keras) the check pointing is done at the end of an epoch. Therefore duplicate entries in the metrics are avoided by providing an explicit step argument to the e.g. `log_scalar` function. By making explicitly provided steps a precondition for resuming and then overwrite based on the step, I think major clashes could be avoided. This would offload the responsibility of sensibly handing the metrics to the programmer. Not sure if there are other logs that would be affected by that issue.\r\n\r\nIf we can work out the requirements for this, and you could point me to the right places in the code I would be happy to submit a PR. \r\n \r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"JarnoRFB"}}