{"action":"created","issue":{"url":"https://api.github.com/repos/automl/HpBandSter/issues/32","repository_url":"https://api.github.com/repos/automl/HpBandSter","labels_url":"https://api.github.com/repos/automl/HpBandSter/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/automl/HpBandSter/issues/32/comments","events_url":"https://api.github.com/repos/automl/HpBandSter/issues/32/events","html_url":"https://github.com/automl/HpBandSter/issues/32","id":389711587,"node_id":"MDU6SXNzdWUzODk3MTE1ODc=","number":32,"title":"Cannot warmstart when some configurations raised exceptions","user":{"login":"arberzela","id":19250934,"node_id":"MDQ6VXNlcjE5MjUwOTM0","avatar_url":"https://avatars1.githubusercontent.com/u/19250934?v=4","gravatar_id":"","url":"https://api.github.com/users/arberzela","html_url":"https://github.com/arberzela","followers_url":"https://api.github.com/users/arberzela/followers","following_url":"https://api.github.com/users/arberzela/following{/other_user}","gists_url":"https://api.github.com/users/arberzela/gists{/gist_id}","starred_url":"https://api.github.com/users/arberzela/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arberzela/subscriptions","organizations_url":"https://api.github.com/users/arberzela/orgs","repos_url":"https://api.github.com/users/arberzela/repos","events_url":"https://api.github.com/users/arberzela/events{/privacy}","received_events_url":"https://api.github.com/users/arberzela/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-11T11:10:49Z","updated_at":"2018-12-13T08:30:07Z","closed_at":null,"author_association":"NONE","body":"Consider warmstarting from the evaluations present in some `results.json` file and some of the sampled evaluations during optimization raised an error (e.g. `OutOfMemory` error) and the following line is appended to the `results.json` for this configuration and a `np.inf` loss is given to this configuration to mark it as a bad one:\r\n\r\n`[[10, 0, 0], 38.0, {\"finished\": 1544192253.4073746, \"started\": 1544191887.3157759, \"submitted\": 1544191887.3152575}, null, \"Traceback (most recent call last):\\n  File \\\"/home/zelaa/HpBandSter/hpbandster/core/worker.py\\\", line 206, in start_computation\\n    result = {'result': self.compute(*args, config_id=id, **kwargs),\\n  File \\\"/home/zelaa/Thesis/bohb-darts/cond/workers/darts_worker.py\\\", line 35, in compute\\n    darts_source=self.darts_path)\\n  File \\\"/home/zelaa/Thesis/bohb-darts/cond/workers/helper.py\\\", line 81, in darts_cifar10\\n    subprocess.check_call(\\\" \\\".join(bash_strings), shell=True)\\n  File \\\"/home/zelaa/anaconda3/envs/pytorch-0.3.1-cu8/lib/python3.5/subprocess.py\\\", line 271, in check_call\\n    raise CalledProcessError(retcode, cmd)\\nsubprocess.CalledProcessError: Command 'cd /home/zelaa/Thesis/bohb-darts/cond/workers/lib/darts_space; python train.py --cutout --auxiliary --save /home/zelaa/Thesis/bohb-darts/cond/data/BOHB/run_4420/10_0_0 --epochs 38 --edge_normal_0 sep_conv_5x5' returned non-zero exit status 1\\n\"]`\r\n\r\nWhen trying to warmstart [line 133](https://github.com/automl/HpBandSter/blob/master/hpbandster/core/base_iteration.py#L133) in `core/base_iteration.py` will raise a `TypeError` exception since `result['loss'] = None` for the aforementioned configuration. Adding `(not result['loss'] is None)` to the condition in [line 133](https://github.com/automl/HpBandSter/blob/master/hpbandster/core/base_iteration.py#L133) and capturing the exception in [line 292](https://github.com/automl/HpBandSter/blob/master/hpbandster/optimizers/config_generators/bohb.py#L292) in `optimizers/config_generators/bohb.py` as follow:\r\n\r\n<pre>\r\ntry:\r\n    loss = job.result[\"loss\"] if np.isfinite(job.result[\"loss\"]) else np.inf\r\nexcept TypeError:\r\n    loss = np.inf\r\n</pre>\r\n\r\nwill remove the issue and BOHB will warmstart succesfully."},"comment":{"url":"https://api.github.com/repos/automl/HpBandSter/issues/comments/446883346","html_url":"https://github.com/automl/HpBandSter/issues/32#issuecomment-446883346","issue_url":"https://api.github.com/repos/automl/HpBandSter/issues/32","id":446883346,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0Njg4MzM0Ng==","user":{"login":"sfalkner","id":4528989,"node_id":"MDQ6VXNlcjQ1Mjg5ODk=","avatar_url":"https://avatars1.githubusercontent.com/u/4528989?v=4","gravatar_id":"","url":"https://api.github.com/users/sfalkner","html_url":"https://github.com/sfalkner","followers_url":"https://api.github.com/users/sfalkner/followers","following_url":"https://api.github.com/users/sfalkner/following{/other_user}","gists_url":"https://api.github.com/users/sfalkner/gists{/gist_id}","starred_url":"https://api.github.com/users/sfalkner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sfalkner/subscriptions","organizations_url":"https://api.github.com/users/sfalkner/orgs","repos_url":"https://api.github.com/users/sfalkner/repos","events_url":"https://api.github.com/users/sfalkner/events{/privacy}","received_events_url":"https://api.github.com/users/sfalkner/received_events","type":"User","site_admin":false},"created_at":"2018-12-13T08:30:07Z","updated_at":"2018-12-13T08:30:07Z","author_association":"CONTRIBUTOR","body":"Good point. I think it would be better to actively check for None and catch that, would you agree?"}}
{"actor":{"display_login":"sfalkner"}}