{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23471","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23471/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23471/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23471/events","html_url":"https://github.com/tensorflow/tensorflow/issues/23471","id":377017350,"node_id":"MDU6SXNzdWUzNzcwMTczNTA=","number":23471,"title":"Won't build tf r1.12rc2 because of MPI support","user":{"login":"inscite","id":20734988,"node_id":"MDQ6VXNlcjIwNzM0OTg4","avatar_url":"https://avatars2.githubusercontent.com/u/20734988?v=4","gravatar_id":"","url":"https://api.github.com/users/inscite","html_url":"https://github.com/inscite","followers_url":"https://api.github.com/users/inscite/followers","following_url":"https://api.github.com/users/inscite/following{/other_user}","gists_url":"https://api.github.com/users/inscite/gists{/gist_id}","starred_url":"https://api.github.com/users/inscite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inscite/subscriptions","organizations_url":"https://api.github.com/users/inscite/orgs","repos_url":"https://api.github.com/users/inscite/repos","events_url":"https://api.github.com/users/inscite/events{/privacy}","received_events_url":"https://api.github.com/users/inscite/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"node_id":"MDU6TGFiZWwzODYxOTE4ODc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"f4b400","default":false},{"id":473173351,"node_id":"MDU6TGFiZWw0NzMxNzMzNTE=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:build/install","name":"type:build/install","color":"159b2e","default":false}],"state":"open","locked":false,"assignee":{"login":"harshini-gadige","id":42781361,"node_id":"MDQ6VXNlcjQyNzgxMzYx","avatar_url":"https://avatars1.githubusercontent.com/u/42781361?v=4","gravatar_id":"","url":"https://api.github.com/users/harshini-gadige","html_url":"https://github.com/harshini-gadige","followers_url":"https://api.github.com/users/harshini-gadige/followers","following_url":"https://api.github.com/users/harshini-gadige/following{/other_user}","gists_url":"https://api.github.com/users/harshini-gadige/gists{/gist_id}","starred_url":"https://api.github.com/users/harshini-gadige/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshini-gadige/subscriptions","organizations_url":"https://api.github.com/users/harshini-gadige/orgs","repos_url":"https://api.github.com/users/harshini-gadige/repos","events_url":"https://api.github.com/users/harshini-gadige/events{/privacy}","received_events_url":"https://api.github.com/users/harshini-gadige/received_events","type":"User","site_admin":false},"assignees":[{"login":"harshini-gadige","id":42781361,"node_id":"MDQ6VXNlcjQyNzgxMzYx","avatar_url":"https://avatars1.githubusercontent.com/u/42781361?v=4","gravatar_id":"","url":"https://api.github.com/users/harshini-gadige","html_url":"https://github.com/harshini-gadige","followers_url":"https://api.github.com/users/harshini-gadige/followers","following_url":"https://api.github.com/users/harshini-gadige/following{/other_user}","gists_url":"https://api.github.com/users/harshini-gadige/gists{/gist_id}","starred_url":"https://api.github.com/users/harshini-gadige/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshini-gadige/subscriptions","organizations_url":"https://api.github.com/users/harshini-gadige/orgs","repos_url":"https://api.github.com/users/harshini-gadige/repos","events_url":"https://api.github.com/users/harshini-gadige/events{/privacy}","received_events_url":"https://api.github.com/users/harshini-gadige/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-11-03T03:03:07Z","updated_at":"2018-11-12T08:42:04Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- OS Platform and Distribution: CentOS Linux release 7.5.1804 (Core)\r\n- TensorFlow installed from (source or binary): source, official release r1.12rc2\r\n- TensorFlow version: r1.12rc2\r\n- Python version: 3.6.6 (anaconda)\r\n- Installed using virtualenv? pip? conda?: conda\r\n- Bazel version (if compiling from source): 0.18.1\r\n- GCC/Compiler version (if compiling from source): 8.2.0 with experimental clang support (of configure),\r\nbut also not working on gcc4 (centos7-native), gcc6 (rh-scl-devtoolset-6), gcc7 (rh-scl-devtoolset-7)\r\n- CUDA/cuDNN version: None, integrated nGraph\r\n- GPU model and memory: None, CPU: AMD Ryzen 7 1800X\r\n\r\n**Describe the problem**\r\nA released source won't build if I enable MPI support flags while configuring.\r\nTried multiple gcc versions, however there was no luck.\r\nOpenMPI was also installed from official OMPI src release v3.1.2.\r\n\r\n**Provide the exact sequence of commands / steps that you executed before running into the problem**\r\n1> Usual ./configure after tarball extract\r\n2> config: Apache Ignite (Y) / XLA JIT (Y) / SYCL (N) / ROCm (N) / CUDA (N) / clang (Y) / MPI (Y) / COPT (-march=native) / Android (N)\r\n3>  bazel --server_javabase=$JAVA_HOME build --cxxopt=\"-D_GLIBCXX_USE_CXX11_ABI=0\" --config=opt --config=ngraph //tensorflow/tools/pip_package:build_pip_package\r\n4> (optional) if I disable MPI support while ./configure -ing, then tf r1.12rc2 builds successfully.\r\n\r\n**Any other info / logs**\r\n_Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached._\r\n\r\n[Manual Patch]\r\nI applied this manual fix: [[Pull req #20147](https://github.com/tensorflow/tensorflow/pull/20147/files)] of tensorflow/contrib/mpi_collectives/kernels/mpi_ops.cc only.\r\n\r\n[Logs]\r\nINFO: From Compiling tensorflow/stream_executor/device_description.cc [for host]:\r\ntensorflow/stream_executor/device_description.cc:148:18: warning: 'DivideCeil' is deprecated: Use MathUtil::CeilOfRatio directly instead. [-Wdeprecated-declarations]\r\n  *block_count = DivideCeil(element_count, *threads_per_block);\r\n                 ^\r\n./tensorflow/stream_executor/device_description.h:362:1: note: 'DivideCeil' has been explicitly marked deprecated here\r\nABSL_DEPRECATED(\"Use MathUtil::CeilOfRatio directly instead.\")\r\n^\r\nexternal/com_google_absl/absl/base/macros.h:149:49: note: expanded from macro 'ABSL_DEPRECATED'\r\n#define ABSL_DEPRECATED(message) __attribute__((deprecated(message)))\r\n                                                ^\r\n1 warning generated.\r\nERROR: /home/xo/pyenv-ngraph/mpicol/tensorflow-1.12.0-rc2/tensorflow/contrib/mpi_collectives/BUILD:40:1: C++ compilation of rule '//tensorflow/contrib/mpi_collectives:python/ops/_mpi_ops.so' failed (Exit 1)\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:58:6: error: explicit specialization of 'CopyTensorData<Eigen::ThreadPoolDevice>' after instantiation**\r\nvoid CopyTensorData<CPUDevice>(void* dst, void* src, size_t size) {\r\n     ^\r\n**./tensorflow/contrib/mpi_collectives/kernels/ring.h:165:3:** note: implicit instantiation first required here\r\n  CopyTensorData<Device>((void*)buffer, (void*)input->tensor_data().data(),\r\n  ^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:71:1: error: explicit specialization of 'AccumulateTensorData<Eigen::ThreadPoolDevice, int>' after instantiation**\r\nGENERATE_ACCUMULATE(int);\r\n^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:65:8:** note: expanded from macro 'GENERATE_ACCUMULATE'\r\n  void AccumulateTensorData<CPUDevice, type>(type * dst, type * src, \\\r\n       ^\r\n**./tensorflow/contrib/mpi_collectives/kernels/ring.h:221:5:** note: implicit instantiation first required here\r\n    AccumulateTensorData<Device, T>(segment_update, segment_recv,\r\n    ^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:72:1: error: explicit specialization of 'AccumulateTensorData<Eigen::ThreadPoolDevice, long long>' after instantiation**\r\nGENERATE_ACCUMULATE(long long);\r\n^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:65:8:** note: expanded from macro 'GENERATE_ACCUMULATE'\r\n  void AccumulateTensorData<CPUDevice, type>(type * dst, type * src, \\\r\n       ^\r\n**./tensorflow/contrib/mpi_collectives/kernels/ring.h:221:5:** note: implicit instantiation first required here\r\n    AccumulateTensorData<Device, T>(segment_update, segment_recv,\r\n    ^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:73:1: error: explicit specialization of 'AccumulateTensorData<Eigen::ThreadPoolDevice, float>' after instantiation**\r\nGENERATE_ACCUMULATE(float);\r\n^\r\n**tensorflow/contrib/mpi_collectives/kernels/ring.cc:65:8:** note: expanded from macro 'GENERATE_ACCUMULATE'\r\n  void AccumulateTensorData<CPUDevice, type>(type * dst, type * src, \\\r\n       ^\r\n**./tensorflow/contrib/mpi_collectives/kernels/ring.h:221:5:** note: implicit instantiation first required here\r\n    AccumulateTensorData<Device, T>(segment_update, segment_recv,\r\n    ^\r\n4 errors generated.\r\nTarget //tensorflow/tools/pip_package:build_pip_package failed to build\r\nUse --verbose_failures to see the command lines of failed build steps.\r\n**INFO:** Elapsed time: 1532.667s, Critical Path: 295.58s\r\n**INFO:** 6912 processes: 6912 local.\r\n**FAILED:** Build did NOT complete successfully"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/437799431","html_url":"https://github.com/tensorflow/tensorflow/issues/23471#issuecomment-437799431","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/23471","id":437799431,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzc5OTQzMQ==","user":{"login":"inscite","id":20734988,"node_id":"MDQ6VXNlcjIwNzM0OTg4","avatar_url":"https://avatars2.githubusercontent.com/u/20734988?v=4","gravatar_id":"","url":"https://api.github.com/users/inscite","html_url":"https://github.com/inscite","followers_url":"https://api.github.com/users/inscite/followers","following_url":"https://api.github.com/users/inscite/following{/other_user}","gists_url":"https://api.github.com/users/inscite/gists{/gist_id}","starred_url":"https://api.github.com/users/inscite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inscite/subscriptions","organizations_url":"https://api.github.com/users/inscite/orgs","repos_url":"https://api.github.com/users/inscite/repos","events_url":"https://api.github.com/users/inscite/events{/privacy}","received_events_url":"https://api.github.com/users/inscite/received_events","type":"User","site_admin":false},"created_at":"2018-11-12T08:42:04Z","updated_at":"2018-11-12T08:42:04Z","author_association":"NONE","body":"@harshini-gadige Sorry for late response. Currently, I omitted the MPI support by skipping build flags (not to use distributed tensorflow, for a while.) Moreover, the brand-new release 1.12.0 came out recently; therefore, I have lots of configurations to test.\r\n\r\nAs @jthestness mentioned about several configurations for baseline, I'll try multiple builds for my desire.\r\n- OS: CentOS 7.5 x86_64\r\n- Host: KVM (due to the lack of physical hosts however it won't give any hardship)\r\n- GCC: 8.2.0 (custom build, since native 4.8.x won't build r1.12.0rc2 in my experience)\r\n- TF: 1.8/1.9/1.10.0/1.11.0/1.12.0\r\n\r\nIf there are any progresses or unexpected results, I'll leave some comments.\r\nThx for your consideration.\r\n\r\nRegards,\r\n@inscite\r\n"}}
{"actor":{"display_login":"inscite"}}