<p dir="auto">I try to build tensorflow from the source code, as <a href="https://www.tensorflow.org/install/install_sources" rel="nofollow">https://www.tensorflow.org/install/install_sources</a> shows. But it fails at basel building like following.</p>
<p dir="auto">commands: bazel build --config=opt --config=cuda --verbose_failures //tensorflow/tools/pip_package:build_pip_package<br>
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See <a href="http://bazel.build/docs/bazel-user-manual.html#sandboxing" rel="nofollow">http://bazel.build/docs/bazel-user-manual.html#sandboxing</a> for more information. You can turn off this warning via --ignore_unsupported_sandboxing.<br>
INFO: Found 1 target...<br>
ERROR: /local/xian_titan/.cache/bazel/bazel_xian_titan/d96b63d3ff03346cfbf37aac2a75fe2c/external/protobuf/BUILD:230:1: C++ compilation of rule '@protobuf//:js_embed' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command<br>
(cd /local/xian_titan/.cache/bazel/bazel_xian_titan/d96b63d3ff03346cfbf37aac2a75fe2c/execroot/tensorflow &amp;&amp;<br>
exec env -<br>
LD_LIBRARY_PATH=/usr/local/lib/:/usr/local/cuda-8.0/lib64<br>
PATH=/usr/local/cuda-8.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/local/xian_titan/bin<br>
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++11' -MD -MF bazel-out/host/bin/external/protobuf/objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.d '-frandom-seed=bazel-out/host/bin/external/protobuf/objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.o' -iquote external/protobuf -iquote bazel-out/host/genfiles/external/protobuf -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE="redacted"' '-D__TIMESTAMP="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/protobuf/src/google/protobuf/compiler/js/embed.cc -o bazel-out/host/bin/external/protobuf/_objs/js_embed/external/protobuf/src/google/protobuf/compiler/js/embed.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.<br>
gcc: error trying to exec 'cc1plus': execvp: No such file or directory<br>
Target //tensorflow/tools/pip_package:build_pip_package failed to build<br>
INFO: Elapsed time: 2.283s, Critical Path: 1.68s</p>
<p dir="auto">I found some other people also had similar problems before. But I cannot successfully address the problem from previous threads.</p>
<p dir="auto">I tried bazel 0.4.4 and 0.4.2 releases, they have the same issues.</p>
<p dir="auto">My system is Red Hat Enterprise Linux Workstation release 6.7 (Santiago),<br>
java version "1.8.0_121",<br>
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)<br>
g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)</p>
<p dir="auto">Hope someone can figure out where's wrong</p>
<h1 dir="auto">Description</h1>
<p dir="auto">Hi,</p>
<p dir="auto">I am trying to compile TensorFlow with non-default GCC. 5.4.0, and get follwing error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gcc: error trying to exec 'as': execvp: No such file or directory
ERROR: /local/cvsupport/_bazel_cvsupport/c61d2ac558d6d30ef2694b9af72e4144/external/nccl_archive/BUILD.bazel:33:1: output 'external/nccl_archive/_objs/nccl/external/nccl_archive/src/libwrap.cu.pic.o' was not created.
ERROR: /local/cvsupport/_bazel_cvsupport/c61d2ac558d6d30ef2694b9af72e4144/external/nccl_archive/BUILD.bazel:33:1: not all outputs were created or valid.
Target //tensorflow/tools/pip_package:build_pip_package failed to build"><pre class="notranslate"><code class="notranslate">gcc: error trying to exec 'as': execvp: No such file or directory
ERROR: /local/cvsupport/_bazel_cvsupport/c61d2ac558d6d30ef2694b9af72e4144/external/nccl_archive/BUILD.bazel:33:1: output 'external/nccl_archive/_objs/nccl/external/nccl_archive/src/libwrap.cu.pic.o' was not created.
ERROR: /local/cvsupport/_bazel_cvsupport/c61d2ac558d6d30ef2694b9af72e4144/external/nccl_archive/BUILD.bazel:33:1: not all outputs were created or valid.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
</code></pre></div>
<h1 dir="auto">Environment</h1>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (AltArch)

$ uname -a
Linux power004.cluster 3.10.0-514.6.1.el7.ppc64le #1 SMP Thu Jan 19 14:34:54 GMT 2017 ppc64le ppc64le ppc64le GNU/Linux

$ ls -l /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcud*
-rw-r--r-- 1 root root 559800 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root     16 Feb 14 23:26 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so -&gt; libcudart.so.8.0
lrwxrwxrwx 1 root root     19 Feb 14 23:26 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so.8.0 -&gt; libcudart.so.8.0.54
-rwxr-xr-x 1 root root 476024 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so.8.0.54
-rw-r--r-- 1 root root 966166 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart_static.a

$ git rev-parse HEAD
16485a3fb5ffcbaa244e55c388e43279d2770982

$ bazel version
INFO: $TEST_TMPDIR defined: output root default is '/local/cvsupport'.
................
Build label: 0.4.4- (@non-git)
Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Feb 9 23:48:24 2017 (1486684104)
Build timestamp: 1486684104
Build timestamp as int: 1486684104"><pre class="notranslate"><code class="notranslate">$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (AltArch)

$ uname -a
Linux power004.cluster 3.10.0-514.6.1.el7.ppc64le #1 SMP Thu Jan 19 14:34:54 GMT 2017 ppc64le ppc64le ppc64le GNU/Linux

$ ls -l /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcud*
-rw-r--r-- 1 root root 559800 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root     16 Feb 14 23:26 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so -&gt; libcudart.so.8.0
lrwxrwxrwx 1 root root     19 Feb 14 23:26 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so.8.0 -&gt; libcudart.so.8.0.54
-rwxr-xr-x 1 root root 476024 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart.so.8.0.54
-rw-r--r-- 1 root root 966166 Oct 29 10:22 /trinity/shared/apps/cv-ppc64le/nvidia/cuda/8.0/lib64/libcudart_static.a

$ git rev-parse HEAD
16485a3fb5ffcbaa244e55c388e43279d2770982

$ bazel version
INFO: $TEST_TMPDIR defined: output root default is '/local/cvsupport'.
................
Build label: 0.4.4- (@non-git)
Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Feb 9 23:48:24 2017 (1486684104)
Build timestamp: 1486684104
Build timestamp as int: 1486684104
</code></pre></div>
<h1 dir="auto">Steps to reproduce</h1>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ module display gcc/5.4.0
-------------------------------------------------------------------
/trinity/shared/modulefiles/cv-ppc64le/gcc/5.4.0:

module-whatis    loads the gcc/5.4.0 environment 
prepend-path     PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin 
prepend-path     LD_LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib 
prepend-path     LD_LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib64 
prepend-path     LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib 
prepend-path     LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib64 
prepend-path     C_INCLUDE_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     INCLUDE /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     CPATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     MANPATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/share/man 

$ cat ../build_vars.sh
export TEST_TMPDIR=/local/cvsupport
export PYTHON_BIN_PATH=/usr/bin/python3.4
export PYTHON_LIB_PATH=/usr/lib64/python3.4/site-packages
export TF_NEED_JEMALLOC=1
export TF_NEED_GCP=0
export TF_NEED_HDFS=0
export TF_NEED_OPENCL=0
export TF_NEED_CUDA=1
export TF_ENABLE_XLA=0
export CC_OPT_FLAGS=&quot;-march=native&quot;
export TF_CUDA_VERSION=8.0
export TF_CUDNN_VERSION=5.1.10
export TF_CUDA_COMPUTE_CAPABILITIES=6.0
export GCC_HOST_COMPILER_PATH=/trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin/gcc
export CUDA_TOOLKIT_PATH=/trinity/shared/apps/cv-ppc64le/nvidia/cuda/${TF_CUDA_VERSION}
export CUDNN_INSTALL_PATH=/trinity/shared/apps/cv-ppc64le/nvidia/cudnn/${TF_CUDA_VERSION}

$ module load nvidia/cuda/8.0 nvidia/cudnn/8.0 gcc/5.4.0 bazel/0.4.4
$ source ../build_vars.sh
$ ./configure
$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package"><pre class="notranslate"><code class="notranslate">$ module display gcc/5.4.0
-------------------------------------------------------------------
/trinity/shared/modulefiles/cv-ppc64le/gcc/5.4.0:

module-whatis    loads the gcc/5.4.0 environment 
prepend-path     PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin 
prepend-path     LD_LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib 
prepend-path     LD_LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib64 
prepend-path     LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib 
prepend-path     LIBRARY_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/lib64 
prepend-path     C_INCLUDE_PATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     INCLUDE /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     CPATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/include 
prepend-path     MANPATH /trinity/shared/apps/cv-ppc64le/gcc/5.4.0/share/man 

$ cat ../build_vars.sh
export TEST_TMPDIR=/local/cvsupport
export PYTHON_BIN_PATH=/usr/bin/python3.4
export PYTHON_LIB_PATH=/usr/lib64/python3.4/site-packages
export TF_NEED_JEMALLOC=1
export TF_NEED_GCP=0
export TF_NEED_HDFS=0
export TF_NEED_OPENCL=0
export TF_NEED_CUDA=1
export TF_ENABLE_XLA=0
export CC_OPT_FLAGS="-march=native"
export TF_CUDA_VERSION=8.0
export TF_CUDNN_VERSION=5.1.10
export TF_CUDA_COMPUTE_CAPABILITIES=6.0
export GCC_HOST_COMPILER_PATH=/trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin/gcc
export CUDA_TOOLKIT_PATH=/trinity/shared/apps/cv-ppc64le/nvidia/cuda/${TF_CUDA_VERSION}
export CUDNN_INSTALL_PATH=/trinity/shared/apps/cv-ppc64le/nvidia/cudnn/${TF_CUDA_VERSION}

$ module load nvidia/cuda/8.0 nvidia/cudnn/8.0 gcc/5.4.0 bazel/0.4.4
$ source ../build_vars.sh
$ ./configure
$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
</code></pre></div>
<h1 dir="auto">Investigation</h1>
<p dir="auto"><code class="notranslate">as</code> is installed on my system, but wasn't built by gcc-5.4, I am using system-default one:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ rpm -qf /usr/bin/as
binutils-2.25.1-22.base.el7.ppc64le"><pre class="notranslate"><code class="notranslate">$ rpm -qf /usr/bin/as
binutils-2.25.1-22.base.el7.ppc64le
</code></pre></div>
<p dir="auto">Digging into stace output, It seems like builder is trying to find <code class="notranslate">as</code> using some incorrect assumptions, and failed:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="134767 execve(&quot;external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/../open64/bin/as&quot;, [&quot;as&quot;, &quot;-I&quot;, &quot;.&quot;, &quot;-I&quot;, &quot;external/nccl_archive/src&quot;, &quot;-I&quot;, &quot;external/local_config_cuda/cross&quot;..., &quot;-a64&quot;, &quot;-mppc64&quot;, &quot;-many&quot;, &quot;-mlittle&quot;, &quot;-o&quot;, &quot;bazel-out/local_linux-py3-opt/bi&quot;..., &quot;/tmp/cc7ZvHce.s&quot;], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve(&quot;external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/../nvvm/bin/as&quot;, [&quot;as&quot;, &quot;-I&quot;, &quot;.&quot;, &quot;-I&quot;, &quot;external/nccl_archive/src&quot;, &quot;-I&quot;, &quot;external/local_config_cuda/cross&quot;..., &quot;-a64&quot;, &quot;-mppc64&quot;, &quot;-many&quot;, &quot;-mlittle&quot;, &quot;-o&quot;, &quot;bazel-out/local_linux-py3-opt/bi&quot;..., &quot;/tmp/cc7ZvHce.s&quot;], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve(&quot;external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/as&quot;, [&quot;as&quot;, &quot;-I&quot;, &quot;.&quot;, &quot;-I&quot;, &quot;external/nccl_archive/src&quot;, &quot;-I&quot;, &quot;external/local_config_cuda/cross&quot;..., &quot;-a64&quot;, &quot;-mppc64&quot;, &quot;-many&quot;, &quot;-mlittle&quot;, &quot;-o&quot;, &quot;bazel-out/local_linux-py3-opt/bi&quot;..., &quot;/tmp/cc7ZvHce.s&quot;], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve(&quot;/trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin/as&quot;, [&quot;as&quot;, &quot;-I&quot;, &quot;.&quot;, &quot;-I&quot;, &quot;external/nccl_archive/src&quot;, &quot;-I&quot;, &quot;external/local_config_cuda/cross&quot;..., &quot;-a64&quot;, &quot;-mppc64&quot;, &quot;-many&quot;, &quot;-mlittle&quot;, &quot;-o&quot;, &quot;bazel-out/local_linux-py3-opt/bi&quot;..., &quot;/tmp/cc7ZvHce.s&quot;], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)"><pre class="notranslate"><code class="notranslate">134767 execve("external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/../open64/bin/as", ["as", "-I", ".", "-I", "external/nccl_archive/src", "-I", "external/local_config_cuda/cross"..., "-a64", "-mppc64", "-many", "-mlittle", "-o", "bazel-out/local_linux-py3-opt/bi"..., "/tmp/cc7ZvHce.s"], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve("external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/../nvvm/bin/as", ["as", "-I", ".", "-I", "external/nccl_archive/src", "-I", "external/local_config_cuda/cross"..., "-a64", "-mppc64", "-many", "-mlittle", "-o", "bazel-out/local_linux-py3-opt/bi"..., "/tmp/cc7ZvHce.s"], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve("external/local_config_cuda/crosstool/clang/bin/../../../cuda/bin/as", ["as", "-I", ".", "-I", "external/nccl_archive/src", "-I", "external/local_config_cuda/cross"..., "-a64", "-mppc64", "-many", "-mlittle", "-o", "bazel-out/local_linux-py3-opt/bi"..., "/tmp/cc7ZvHce.s"], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
134767 execve("/trinity/shared/apps/cv-ppc64le/gcc/5.4.0/bin/as", ["as", "-I", ".", "-I", "external/nccl_archive/src", "-I", "external/local_config_cuda/cross"..., "-a64", "-mppc64", "-many", "-mlittle", "-o", "bazel-out/local_linux-py3-opt/bi"..., "/tmp/cc7ZvHce.s"], [/* 19 vars */] &lt;unfinished ...&gt;
134767 &lt;... execve resumed&gt; )           = -1 ENOENT (No such file or directory)
</code></pre></div>