{"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/10876","repository_url":"https://api.github.com/repos/PaddlePaddle/Paddle","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/10876/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/10876/comments","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/10876/events","html_url":"https://github.com/PaddlePaddle/Paddle/issues/10876","id":325654060,"node_id":"MDU6SXNzdWUzMjU2NTQwNjA=","number":10876,"title":"run \"fit a line\" failed on tag v0.12.0 ","user":{"login":"xjqbest","id":12492564,"node_id":"MDQ6VXNlcjEyNDkyNTY0","avatar_url":"https://avatars0.githubusercontent.com/u/12492564?v=4","gravatar_id":"","url":"https://api.github.com/users/xjqbest","html_url":"https://github.com/xjqbest","followers_url":"https://api.github.com/users/xjqbest/followers","following_url":"https://api.github.com/users/xjqbest/following{/other_user}","gists_url":"https://api.github.com/users/xjqbest/gists{/gist_id}","starred_url":"https://api.github.com/users/xjqbest/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xjqbest/subscriptions","organizations_url":"https://api.github.com/users/xjqbest/orgs","repos_url":"https://api.github.com/users/xjqbest/repos","events_url":"https://api.github.com/users/xjqbest/events{/privacy}","received_events_url":"https://api.github.com/users/xjqbest/received_events","type":"User","site_admin":false},"labels":[{"id":651040732,"node_id":"MDU6TGFiZWw2NTEwNDA3MzI=","url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels/User","name":"User","color":"5319e7","default":false},{"id":863941226,"node_id":"MDU6TGFiZWw4NjM5NDEyMjY=","url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels/%E7%83%AB","name":"烫","color":"b60205","default":false}],"state":"closed","locked":false,"assignee":{"login":"typhoonzero","id":13348433,"node_id":"MDQ6VXNlcjEzMzQ4NDMz","avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"assignees":[{"login":"typhoonzero","id":13348433,"node_id":"MDQ6VXNlcjEzMzQ4NDMz","avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-05-23T10:58:29Z","updated_at":"2018-05-31T07:56:14Z","closed_at":"2018-05-31T07:56:14Z","author_association":"NONE","body":"when doing distributed cpu training on k8s, using 2 pservers and 2 trainers, one trainer succeeded and the other trainer failed.\r\nthe succeeded trainer's log is:\r\n```\r\n...\r\n[245.54922]\r\n[106.66316]\r\n[78.47267]\r\n[86.7585]\r\n[6.032805]\r\n('infer shape: ', (10, 1))\r\n('infer results: ', array([[-46.680927],\r\n       [-51.959793],\r\n       [-33.93647 ],\r\n       [-31.00824 ],\r\n       [-38.361206],\r\n       [-46.148376],\r\n       [-28.084446],\r\n       [-35.666855],\r\n       [-33.81091 ],\r\n       [-50.53453 ]], dtype=float32))\r\n```\r\nthe failed trainer's log is:\r\n```\r\n...\r\n81.19149]\r\n[95.01187]\r\n[104.61168]\r\n[139.18037]\r\n[113.60637]\r\n[71.95655]\r\n[13.743915]\r\nE0523 18:38:30.050364  1005 grpc_client.cc:236] proc param error:name:[fc_0.b_0] ep:[10.255.92.12:30002] grpc error:Deadline Exceeded\r\nE0523 18:38:30.051322  1004 grpc_client.cc:236] proc param error:name:[fc_0.w_0] ep:[10.255.93.18:30002] grpc error:Deadline Exceeded\r\nTraceback (most recent call last):\r\n  File \"train.py\", line 175, in <module>\r\n    main(use_cuda, is_local)\r\n  File \"train.py\", line 168, in main\r\n    train(use_cuda, save_dirname, is_local)\r\n  File \"train.py\", line 121, in train\r\n    train_loop(t.get_trainer_program())\r\n  File \"train.py\", line 85, in train_loop\r\n    fetch_list=[avg_cost])\r\n  File \"/usr/local/lib/python2.7/site-packages/paddle/fluid/executor.py\", line 336, in run\r\n    self.executor.run(program.desc, scope, 0, True, True)\r\npaddle.fluid.core.EnforceNotMet:  at [/paddle/paddle/fluid/operators/send_op.cc:82]\r\nPaddlePaddle Call Stacks: \r\n0       0x7f9830615566p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486\r\n1       0x7f9830c754edp paddle::operators::SendOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 3197\r\n2       0x7f9830ce5818p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 56\r\n3       0x7f983069cf90p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool) + 336\r\n4       0x7f983069d6d4p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 100\r\n5       0x7f983062c01bp _ZZN8pybind1112cpp_function10initializeIZNS0_C1IvN6paddle9framework8ExecutorEJRKNS4_11ProgramDescEPNS4_5ScopeEibbEJNS_4nameENS_9is_methodENS_7siblingEEEEMT0_FT_DpT1_EDpRKT2_EUlPS5_S8_SA_ibbE_vJSO_S8_SA_ibbEJSB_SC_SD_EEEvOSF_PFSE_SH_ESN_ENUlRNS_6detail13function_callEE1_4_FUNESV_ + 555\r\n6       0x7f9830625d84p pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 2596\r\n7       0x7f98c4fe2631p PyEval_EvalFrameEx + 24497\r\n8       0x7f98c4fe3bcep PyEval_EvalCodeEx + 2190\r\n9       0x7f98c4fe220ap PyEval_EvalFrameEx + 23434\r\n10      0x7f98c4fe3bcep PyEval_EvalCodeEx + 2190\r\n11      0x7f98c4fe220ap PyEval_EvalFrameEx + 23434\r\n12      0x7f98c4fe3bcep PyEval_EvalCodeEx + 2190\r\n13      0x7f98c4fe220ap PyEval_EvalFrameEx + 23434\r\n14      0x7f98c4fe3bcep PyEval_EvalCodeEx + 2190\r\n15      0x7f98c4fe220ap PyEval_EvalFrameEx + 23434\r\n16      0x7f98c4fe3bcep PyEval_EvalCodeEx + 2190\r\n17      0x7f98c4fe3ce2p PyEval_EvalCode + 50\r\n18      0x7f98c50039e0p PyRun_FileExFlags + 176\r\n19      0x7f98c5003bbfp PyRun_SimpleFileExFlags + 239\r\n20      0x7f98c5019454p Py_Main + 3188\r\n21      0x7f98c42cdcddp __libc_start_main + 253\r\n22            0x400649p\r\n\r\n```\r\n\r\nand no errors in pservers' log"},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments/393444074","html_url":"https://github.com/PaddlePaddle/Paddle/issues/10876#issuecomment-393444074","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/10876","id":393444074,"node_id":"MDEyOklzc3VlQ29tbWVudDM5MzQ0NDA3NA==","user":{"login":"typhoonzero","id":13348433,"node_id":"MDQ6VXNlcjEzMzQ4NDMz","avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"created_at":"2018-05-31T07:56:14Z","updated_at":"2018-05-31T07:56:14Z","author_association":"CONTRIBUTOR","body":"Done, remove the following lines in the code, which may cause different trainers run different numbers of batches.\r\n\r\n```\r\nif avg_loss_value[0] < 10.0:\r\n                    if save_dirname is not None:\r\n                        fluid.io.save_inference_model(save_dirname, ['x'],\r\n                                                      [y_predict], exe)\r\n```"}}
{"actor":{"display_login":"typhoonzero"}}