{"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/11843","repository_url":"https://api.github.com/repos/PaddlePaddle/Paddle","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/11843/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/11843/comments","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/11843/events","html_url":"https://github.com/PaddlePaddle/Paddle/issues/11843","id":336907103,"node_id":"MDU6SXNzdWUzMzY5MDcxMDM=","number":11843,"title":"First round parameters of trainer and pserver was NOT identical","user":{"login":"velconia","id":1979255,"node_id":"MDQ6VXNlcjE5NzkyNTU=","avatar_url":"https://avatars1.githubusercontent.com/u/1979255?v=4","gravatar_id":"","url":"https://api.github.com/users/velconia","html_url":"https://github.com/velconia","followers_url":"https://api.github.com/users/velconia/followers","following_url":"https://api.github.com/users/velconia/following{/other_user}","gists_url":"https://api.github.com/users/velconia/gists{/gist_id}","starred_url":"https://api.github.com/users/velconia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velconia/subscriptions","organizations_url":"https://api.github.com/users/velconia/orgs","repos_url":"https://api.github.com/users/velconia/repos","events_url":"https://api.github.com/users/velconia/events{/privacy}","received_events_url":"https://api.github.com/users/velconia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-29T08:20:40Z","updated_at":"2018-06-29T09:08:27Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The parameters and bias in our program will be initialised with the code below:\r\n\r\n```python\r\n    def set_default_param_initializer(self):                                    \r\n        \"\"\"                                                                     \r\n        Set the default initializer for the parameter with Xavier.              \r\n                                                                                \r\n        Args:                                                                   \r\n            None.                                                               \r\n                                                                                \r\n        Returns:                                                                \r\n            None.                                                               \r\n        \"\"\"                                                                     \r\n        self.set_default_initializer(Xavier())                                  \r\n                                                                                \r\n    def set_default_bias_initializer(self):                                     \r\n        \"\"\"                                                                     \r\n        Set the default initializer for the bias with Constant(0.0).            \r\n                                                                                \r\n        Args:                                                                   \r\n            None.                                                               \r\n                                                                                \r\n        Returns:                                                                \r\n            None.                                                               \r\n        \"\"\"                                                                     \r\n        self.set_default_initializer(Constant(0.0)) \r\n```\r\n\r\nThe bias was inited with 0.0f;\r\n\r\nAnd the params was inited with uniform_random_seed(0):\r\n\r\n```CPP\r\n    if (seed == 0) {                                                               \r\n      seed = std::random_device()();                                               \r\n    }      \r\n```\r\n\r\nSo it's true random if the program was ran in single-machine mode.\r\n\r\nHowever, in distributed mode, the pserver will initialise all parameters randomly too, which will lead to the difference between params in pserver and params in trainer:\r\n\r\n```\r\nWeigth w1 in Trainer -> uniform_random_seed(0) -> [0.4, 0.3, 0.5]\r\nWeigth w1 in Trainer -> uniform_random_seed(0) -> [0.4, 0.3, 0.5]\r\n```\r\n\r\nThis difference will be eliminated after the first round of training, but it means that the first round params was NOT identical and the training in first round means nothing..."},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments/401296446","html_url":"https://github.com/PaddlePaddle/Paddle/issues/11843#issuecomment-401296446","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/11843","id":401296446,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTI5NjQ0Ng==","user":{"login":"velconia","id":1979255,"node_id":"MDQ6VXNlcjE5NzkyNTU=","avatar_url":"https://avatars1.githubusercontent.com/u/1979255?v=4","gravatar_id":"","url":"https://api.github.com/users/velconia","html_url":"https://github.com/velconia","followers_url":"https://api.github.com/users/velconia/followers","following_url":"https://api.github.com/users/velconia/following{/other_user}","gists_url":"https://api.github.com/users/velconia/gists{/gist_id}","starred_url":"https://api.github.com/users/velconia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velconia/subscriptions","organizations_url":"https://api.github.com/users/velconia/orgs","repos_url":"https://api.github.com/users/velconia/repos","events_url":"https://api.github.com/users/velconia/events{/privacy}","received_events_url":"https://api.github.com/users/velconia/received_events","type":"User","site_admin":false},"created_at":"2018-06-29T09:08:27Z","updated_at":"2018-06-29T09:08:27Z","author_association":"CONTRIBUTOR","body":"@typhoonzero which name would u prefer here, ```remote_initializer``` or the ```pserver_initializer```?"}}
{"actor":{"display_login":"velconia"}}