{"action":"created","issue":{"url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/11","repository_url":"https://api.github.com/repos/carpedm20/ENAS-pytorch","labels_url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/11/comments","events_url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/11/events","html_url":"https://github.com/carpedm20/ENAS-pytorch/issues/11","id":305039339,"number":11,"title":"Cannot use test and reproduce the result?","user":{"login":"renqianluo","id":8542159,"avatar_url":"https://avatars1.githubusercontent.com/u/8542159?v=4","gravatar_id":"","url":"https://api.github.com/users/renqianluo","html_url":"https://github.com/renqianluo","followers_url":"https://api.github.com/users/renqianluo/followers","following_url":"https://api.github.com/users/renqianluo/following{/other_user}","gists_url":"https://api.github.com/users/renqianluo/gists{/gist_id}","starred_url":"https://api.github.com/users/renqianluo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/renqianluo/subscriptions","organizations_url":"https://api.github.com/users/renqianluo/orgs","repos_url":"https://api.github.com/users/renqianluo/repos","events_url":"https://api.github.com/users/renqianluo/events{/privacy}","received_events_url":"https://api.github.com/users/renqianluo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-14T06:37:37Z","updated_at":"2018-03-14T13:03:29Z","closed_at":null,"author_association":"NONE","body":"@dukebw Hi, thanks for your code. I download it and run it but meet with 3 main problems:\r\n1. It seems that it cannot reproduce the result in the paper? I run it using the run.sh by default and the eval ppl is around 80~100 until the end of the training (150 epochs)\r\n2. There is no test function in the Trainer class. I add one using the evaluation method (by passing self.test_data as the arg). However,  the ppl is around 1500. Even when I pass self.train_data, self.eval_data or self.valid_data, it`s also around 1500.\r\n3. After training is done, when I call either test() or derive() and pass the arg --load_path, the self.shared.load_state_dict in load_model() throws out an error as \"KeyError: unexpected key batch_norm.weight in state_dict\". Moreover, I print the self.shared.state_dict.keys() and the content loaded by torch.load from the checkpoint, and find that, parameters stored in the checkpoint contain 4 parameters related with batch normalization as \"batch_norm.weight\", \"batch_norm.bias\", \"bath_norm.running_mean\", \"batch_norm.running_var\", while the parameters shown by self.shared.load_state_dict not.\r\n\r\nIt would be great if you could help check these."},"comment":{"url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/comments/373012953","html_url":"https://github.com/carpedm20/ENAS-pytorch/issues/11#issuecomment-373012953","issue_url":"https://api.github.com/repos/carpedm20/ENAS-pytorch/issues/11","id":373012953,"user":{"login":"dukebw","id":8805584,"avatar_url":"https://avatars0.githubusercontent.com/u/8805584?v=4","gravatar_id":"","url":"https://api.github.com/users/dukebw","html_url":"https://github.com/dukebw","followers_url":"https://api.github.com/users/dukebw/followers","following_url":"https://api.github.com/users/dukebw/following{/other_user}","gists_url":"https://api.github.com/users/dukebw/gists{/gist_id}","starred_url":"https://api.github.com/users/dukebw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dukebw/subscriptions","organizations_url":"https://api.github.com/users/dukebw/orgs","repos_url":"https://api.github.com/users/dukebw/repos","events_url":"https://api.github.com/users/dukebw/events{/privacy}","received_events_url":"https://api.github.com/users/dukebw/received_events","type":"User","site_admin":false},"created_at":"2018-03-14T13:03:29Z","updated_at":"2018-03-14T13:03:29Z","author_association":"CONTRIBUTOR","body":"Hi Ren Qian,\r\n\r\nTo address 1. and 2., it is necessary to re-train the model from scratch after discovering a good architecture. See Section 4.5 of this early revision of the ENAS paper: https://openreview.net/references/pdf?id=BykVS-WC-. This re-training is not implemented yet.\r\n\r\nMy results on the validation of the _shared_ model are similar to yours, with the best validation perplexity I have seen being 82.\r\n\r\nThe third point you raised sounds like a genuine bug. I think it has to do with these lines in `models/shared_rnn.py`: https://github.com/carpedm20/ENAS-pytorch/blob/5736a24ef6d1967e4e9eefe2f6662a872a98586b/models/shared_rnn.py#L185-L188.\r\n\r\nThank you for pointing out the issues."}}
{"actor":{"display_login":"dukebw"}}