{"action":"created","issue":{"url":"https://api.github.com/repos/bazelbuild/bazel/issues/1871","repository_url":"https://api.github.com/repos/bazelbuild/bazel","labels_url":"https://api.github.com/repos/bazelbuild/bazel/issues/1871/labels{/name}","comments_url":"https://api.github.com/repos/bazelbuild/bazel/issues/1871/comments","events_url":"https://api.github.com/repos/bazelbuild/bazel/issues/1871/events","html_url":"https://github.com/bazelbuild/bazel/issues/1871","id":180766508,"number":1871,"title":"Bazel build failure for Tensorflow while building from source","user":{"login":"sskgit","id":11601821,"avatar_url":"https://avatars.githubusercontent.com/u/11601821?v=3","gravatar_id":"","url":"https://api.github.com/users/sskgit","html_url":"https://github.com/sskgit","followers_url":"https://api.github.com/users/sskgit/followers","following_url":"https://api.github.com/users/sskgit/following{/other_user}","gists_url":"https://api.github.com/users/sskgit/gists{/gist_id}","starred_url":"https://api.github.com/users/sskgit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sskgit/subscriptions","organizations_url":"https://api.github.com/users/sskgit/orgs","repos_url":"https://api.github.com/users/sskgit/repos","events_url":"https://api.github.com/users/sskgit/events{/privacy}","received_events_url":"https://api.github.com/users/sskgit/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-03T22:47:15Z","updated_at":"2016-10-06T10:04:22Z","closed_at":"2016-10-06T10:04:22Z","body":"While creating Tensorflow build from the source bazel build repeatedly fails.\r\n\r\nI am trying Tensoflow from the source and receive build error for \r\na) C++ compilation of rule '@grpc//:gpr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc\r\n\r\nb) ERROR: I/O error while writing action log: No space left on device.\r\n\r\nEnviornment:\r\n\r\nCuda 8.0\r\nCuDNN 5\r\nUbuntu 16.04\r\nbazel 0.31\r\nNvidia K80\r\nAzure VM N6 (56 GB Memory)\r\n\r\nTried ./configure and build several times. Configure is successful but build fails.\r\n\r\nAlso tried, bazel clean, bazel clean --expunge and ran the build with reduced number of jobs\r\nbazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package\r\n\r\nbut the error continues..\r\n\r\nLooked at this thread and also\r\n\r\n#190\r\n\r\nhttps://github.com/tensorflow/tensorflow/issues/4105\r\n\r\nHere is the full error message:\r\n\r\n52adb8ea4f53b1b72067611e8a7eb020/external/grpc/BUILD:69:1: C++ compilation of rule '@grpc//:gpr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 38 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.\r\nERROR: I/O error while writing action log: No space left on device.\r\njava.util.logging.ErrorManager: 2\r\njava.io.IOException: No space left on device\r\nat java.io.FileOutputStream.writeBytes(Native Method)\r\nat java.io.FileOutputStream.write(FileOutputStream.java:326)\r\nat java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\r\nat java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)\r\nat java.util.logging.FileHandler$MeteredStream.flush(FileHandler.java:196)\r\nat sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)\r\nat sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)\r\nat java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)\r\nat java.util.logging.StreamHandler.flush(StreamHandler.java:259)\r\nat java.util.logging.FileHandler.publish(FileHandler.java:683)\r\nat java.util.logging.Logger.log(Logger.java:738)\r\nat java.util.logging.Logger.doLog(Logger.java:765)\r\nat java.util.logging.Logger.log(Logger.java:788)\r\nat java.util.logging.Logger.info(Logger.java:1489)\r\nat com.google.devtools.build.lib.profiler.AutoProfiler$LoggingElapsedTimeReceiver.accept(AutoProfiler.java:315)\r\nat com.google.devtools.build.lib.profiler.AutoProfiler$SequencedElapsedTimeReceiver.accept(AutoProfiler.java:262)\r\nat com.google.devtools.build.lib.profiler.AutoProfiler.completeAndGetElapsedTimeNanos(AutoProfiler.java:226)\r\nat com.google.devtools.build.lib.buildtool.ExecutionTool.saveCaches(ExecutionTool.java:725)\r\nat com.google.devtools.build.lib.buildtool.ExecutionTool.executeBuild(ExecutionTool.java:470)\r\nat com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:201)\r\nat com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:333)\r\nat com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:69)\r\nat com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:488)\r\nat com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:324)\r\nat com.google.devtools.build.lib.runtime.CommandExecutor.exec(CommandExecutor.java:49)\r\nat com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:70)\r\n\r\n============\r\n\r\nUpdate 1:\r\n\r\ndf\r\nFilesystem 1K-blocks Used Available Use% Mounted on\r\nudev 28837212 0 28837212 0% /dev\r\ntmpfs 5771196 9244 5761952 1% /run\r\n/dev/sda1 29711408 29329736 365288 99% /\r\ntmpfs 28855968 0 28855968 0% /dev/shm\r\ntmpfs 5120 0 5120 0% /run/lock\r\ntmpfs 28855968 0 28855968 0% /sys/fs/cgroup\r\nnone 64 0 64 0% /etc/network/interfaces.dynamic.d\r\n/dev/sdb1 356513788 135524 356378264 1% /mnt\r\ntmpfs 5771196 0 5771196 0% /run/user/1000\r\n\r\nIt shows mounted on / has almost no space, and I think $VM/Username (i.e. $HOME) is on the same mount /, as there is no $HOME mounted on, in the output. Is this the right command?\r\n\r\nAt / (used space)\r\n\r\ndu -sch\r\n2.3G .\r\n2.3G total\r\n\r\nAt $HOME (used space)\r\n\r\ndu -sch\r\n4.4G .\r\n4.4G total\r\n\r\nSo, I am not sure what is occupying the remaining space? Total storage on this VM is 380 GB.\r\n\r\nIs there any way to get rid of bazel logs? Is it causing the space issue? If so, where? \r\nAlso, if it is how do I free up some space?\r\n\r\nUpdate 2:\r\n\r\n\r\nused bazel clean --expunge (this cleaned 500MB space) and re-configured tensorflow using ./configure\r\n\r\nRun the bazel build again..\r\n\r\nbazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package\r\n\r\nERROR: /$HOME/Downloads/tensorflow/tensorflow/core/kernels/BUILD:1710:1: error while parsing .d file: /$HOME/.cache/bazel/_bazel_gpuadmin/52adb8ea4f53b1b72067611e8a7eb020/execroot/tensorflow/bazel-out/local_linux-opt/bin/tensorflow/core/kernels/_objs/depth_space_ops_gpu/tensorflow/core/kernels/depthtospace_op_gpu.cu.pic.d (No such file or directory).\r\nnvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.\r\n: fatal error: when writing output to : No space left on device\r\ncompilation terminated.\r\nTarget //tensorflow/tools/pip_package:build_pip_package failed to build\r\n\r\nAt $HOME:\r\ndu -sch\r\n4.6G .\r\n4.6G total\r\n\r\nAt /:\r\ndu -sch\r\n2.3G .\r\n2.3G total\r\n\r\ndf -h\r\nFilesystem Size Used Avail Use% Mounted on\r\nudev 28G 0 28G 0% /dev\r\ntmpfs 5.6G 9.1M 5.5G 1% /run\r\n/dev/sda1 29G 29G 9.8M 100% /\r\ntmpfs 28G 0 28G 0% /dev/shm\r\ntmpfs 5.0M 0 5.0M 0% /run/lock\r\ntmpfs 28G 0 28G 0% /sys/fs/cgroup\r\nnone 64K 0 64K 0% /etc/network/interfaces.dynamic.d\r\n/dev/sdb1 340G 133M 340G 1% /mnt\r\ntmpfs 5.6G 0 5.6G 0% /run/user/1000\r\n\r\nThis is a fairly new machine with few installer software (<200MB)\r\n\r\nAm I missing something to complete the build???\r\n\r\nThanks,"},"comment":{"url":"https://api.github.com/repos/bazelbuild/bazel/issues/comments/251919581","html_url":"https://github.com/bazelbuild/bazel/issues/1871#issuecomment-251919581","issue_url":"https://api.github.com/repos/bazelbuild/bazel/issues/1871","id":251919581,"user":{"login":"damienmg","id":3721087,"avatar_url":"https://avatars.githubusercontent.com/u/3721087?v=3","gravatar_id":"","url":"https://api.github.com/users/damienmg","html_url":"https://github.com/damienmg","followers_url":"https://api.github.com/users/damienmg/followers","following_url":"https://api.github.com/users/damienmg/following{/other_user}","gists_url":"https://api.github.com/users/damienmg/gists{/gist_id}","starred_url":"https://api.github.com/users/damienmg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damienmg/subscriptions","organizations_url":"https://api.github.com/users/damienmg/orgs","repos_url":"https://api.github.com/users/damienmg/repos","events_url":"https://api.github.com/users/damienmg/events{/privacy}","received_events_url":"https://api.github.com/users/damienmg/received_events","type":"User","site_admin":false},"created_at":"2016-10-06T10:04:22Z","updated_at":"2016-10-06T10:04:22Z","body":"What is /mnt? It's is totally full (350G)!\r\n\r\nWhere is home?"}}
{"actor":{"display_login":"damienmg"}}