{"action":"created","issue":{"url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/61","repository_url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow","labels_url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/61/comments","events_url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/61/events","html_url":"https://github.com/ROCmSoftwarePlatform/hiptensorflow/issues/61","id":291819317,"number":61,"title":"convnet-benchmarks: Illegal instruction (core dumped)","user":{"login":"yige-hu","id":5649050,"avatar_url":"https://avatars2.githubusercontent.com/u/5649050?v=4","gravatar_id":"","url":"https://api.github.com/users/yige-hu","html_url":"https://github.com/yige-hu","followers_url":"https://api.github.com/users/yige-hu/followers","following_url":"https://api.github.com/users/yige-hu/following{/other_user}","gists_url":"https://api.github.com/users/yige-hu/gists{/gist_id}","starred_url":"https://api.github.com/users/yige-hu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yige-hu/subscriptions","organizations_url":"https://api.github.com/users/yige-hu/orgs","repos_url":"https://api.github.com/users/yige-hu/repos","events_url":"https://api.github.com/users/yige-hu/events{/privacy}","received_events_url":"https://api.github.com/users/yige-hu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"parallelo","id":2438230,"avatar_url":"https://avatars3.githubusercontent.com/u/2438230?v=4","gravatar_id":"","url":"https://api.github.com/users/parallelo","html_url":"https://github.com/parallelo","followers_url":"https://api.github.com/users/parallelo/followers","following_url":"https://api.github.com/users/parallelo/following{/other_user}","gists_url":"https://api.github.com/users/parallelo/gists{/gist_id}","starred_url":"https://api.github.com/users/parallelo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/parallelo/subscriptions","organizations_url":"https://api.github.com/users/parallelo/orgs","repos_url":"https://api.github.com/users/parallelo/repos","events_url":"https://api.github.com/users/parallelo/events{/privacy}","received_events_url":"https://api.github.com/users/parallelo/received_events","type":"User","site_admin":false},"assignees":[{"login":"parallelo","id":2438230,"avatar_url":"https://avatars3.githubusercontent.com/u/2438230?v=4","gravatar_id":"","url":"https://api.github.com/users/parallelo","html_url":"https://github.com/parallelo","followers_url":"https://api.github.com/users/parallelo/followers","following_url":"https://api.github.com/users/parallelo/following{/other_user}","gists_url":"https://api.github.com/users/parallelo/gists{/gist_id}","starred_url":"https://api.github.com/users/parallelo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/parallelo/subscriptions","organizations_url":"https://api.github.com/users/parallelo/orgs","repos_url":"https://api.github.com/users/parallelo/repos","events_url":"https://api.github.com/users/parallelo/events{/privacy}","received_events_url":"https://api.github.com/users/parallelo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-01-26T07:29:14Z","updated_at":"2018-02-14T01:47:58Z","closed_at":null,"author_association":"NONE","body":"Operating System: Ubuntu 16.04.3 LTS, Linux kernel 4.13.0\r\nGPU: AMD RX 580\r\nROCm backend.\r\n\r\nI installed hipTensorflow from the source, following the steps in:\r\nhttps://github.com/ROCmSoftwarePlatform/hiptensorflow/blob/hip/rocm_docs/tensorflow-build-from-source.md\r\nThe commit hash of hipTensorflow: 28243d43a933eefbd1b4a413a74677939cc74aa2\r\nBazel version: 0.4.5\r\n\r\nI'm trying to run alexNet benchmark from convnet-benchmarks:\r\nhttps://github.com/soumith/convnet-benchmarks/blob/master/tensorflow/benchmark_alexnet.py\r\n\r\nAnd got an 'Illegal instruction (core dumped)':\r\n> I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libhipblas.so locally\r\n> I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libhipfft.so locally\r\n> I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libhip_hcc.so locally\r\n> I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libhiprng.so locally\r\n> I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libMIOpen.so locally\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\r\n> W tensorflow/stream_executor/cuda/cuda_driver.cc:633] creating context when one is currently active; existing: 0x7f1c648938c0\r\n> I tensorflow/core/common_runtime/gpu/gpu_device.cc:892] Found device 0 with properties: \r\n> name: Device 67df\r\n> major: 3 minor: 0 memoryClockRate (GHz) 1.36\r\n> pciBusID 0000:01:00.0\r\n> Total memory: 8.00GiB\r\n> Free memory: 7.75GiB\r\n> I tensorflow/core/common_runtime/gpu/gpu_device.cc:913] DMA: 0 \r\n> I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] 0:   Y \r\n> I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Device 67df, pci bus id: 0000:01:00.0)\r\n> Illegal instruction (core dumped)\r\n\r\nI got the same error with many other benchmarks, including other convnet-benchmarks for Tensorflow.\r\n\r\nWith 'gdb program core' I got:\r\n> Program terminated with signal SIGILL, Illegal instruction.\r\n> #0  0x00007f72496c8d0d in ?? ()\r\n\r\nThen, with 'layout asm' I got:\r\n> Cannot access memory at address 0x7f72496c8d0d\r\n\r\nThus, I feel the core dump is not caused by unsupported, architecture-specific instructions (I'm using a x86_64 machine), it's more like the program jumps in a random address with plain data or in code but not in the start of the opcode.\r\n\r\nDoes anyone have an idea on what could be wrong? I'd appreciate your help!"},"comment":{"url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/comments/365468927","html_url":"https://github.com/ROCmSoftwarePlatform/hiptensorflow/issues/61#issuecomment-365468927","issue_url":"https://api.github.com/repos/ROCmSoftwarePlatform/hiptensorflow/issues/61","id":365468927,"user":{"login":"yige-hu","id":5649050,"avatar_url":"https://avatars2.githubusercontent.com/u/5649050?v=4","gravatar_id":"","url":"https://api.github.com/users/yige-hu","html_url":"https://github.com/yige-hu","followers_url":"https://api.github.com/users/yige-hu/followers","following_url":"https://api.github.com/users/yige-hu/following{/other_user}","gists_url":"https://api.github.com/users/yige-hu/gists{/gist_id}","starred_url":"https://api.github.com/users/yige-hu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yige-hu/subscriptions","organizations_url":"https://api.github.com/users/yige-hu/orgs","repos_url":"https://api.github.com/users/yige-hu/repos","events_url":"https://api.github.com/users/yige-hu/events{/privacy}","received_events_url":"https://api.github.com/users/yige-hu/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T01:47:58Z","updated_at":"2018-02-14T01:47:58Z","author_association":"NONE","body":"I tried cherry-pick the commit (as mentioned in #67)\r\n`git cherry-pick 417dfce2180a49e1bbe38db10025b5da4101c153`\r\nAnd hipTensorflow compiles.\r\n\r\nIt seems there is no illegal instruction error. However, when running tensorflow/examples/learn/mnist.py, I got the followed error message.\r\n```\r\nterminate called after throwing an instance of 'std::runtime_error'\r\n  what():  No device code available for function: _ZN5Eigen8internal15EigenMetaKernelINS_15TensorEvaluatorIKNS_14TensorAssignOpINS_9TensorMapINS_6TensorIfLi1ELi1EiEELi16ENS_11MakePointerEEEKNS_18TensorCwiseUnaryOpINS0_12scalar_rightIffNS0_18scalar_quotient_opIffEEEEKNS4_INS5_IKfLi1ELi1EiEELi16ES7_EEEEEENS_9GpuDeviceEEEiEEvT_T0_\r\nAborted (core dumped)\r\n```\r\n\r\nAlso, when running\r\n```\r\nconvnet-benchmarks/tensorflow/benchmark_alexnet.py\r\nconvnet-benchmarks/tensorflow/benchmark_googlenet.py\r\n```\r\nThey hang after the followed messages printed out.\r\n```\r\nI tensorflow/core/kernels/conv_ops.cc:604] running auto-tune for Convolve\r\nI tensorflow/core/kernels/conv_ops.cc:604] running auto-tune for Convolve\r\n```\r\n\r\nThanks,\r\nYige"}}
{"actor":{"display_login":"yige-hu"}}