{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/60","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/60/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/60/events","html_url":"https://github.com/GoogleCloudPlatform/container-engine-accelerators/issues/60","id":305235435,"number":60,"title":"Consider adding cuda libraries to path for container mounts","user":{"login":"cmluciano","id":4790487,"avatar_url":"https://avatars1.githubusercontent.com/u/4790487?v=4","gravatar_id":"","url":"https://api.github.com/users/cmluciano","html_url":"https://github.com/cmluciano","followers_url":"https://api.github.com/users/cmluciano/followers","following_url":"https://api.github.com/users/cmluciano/following{/other_user}","gists_url":"https://api.github.com/users/cmluciano/gists{/gist_id}","starred_url":"https://api.github.com/users/cmluciano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmluciano/subscriptions","organizations_url":"https://api.github.com/users/cmluciano/orgs","repos_url":"https://api.github.com/users/cmluciano/repos","events_url":"https://api.github.com/users/cmluciano/events{/privacy}","received_events_url":"https://api.github.com/users/cmluciano/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-14T16:32:06Z","updated_at":"2018-03-14T17:43:40Z","closed_at":null,"author_association":"NONE","body":"It would be helpful if we can copy or mount in other paths that CUDA (and by extension TensorFlow) need to run CUDA jobs. \r\n\r\nTensorflow will look for libraries like libcupti within `LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64`.\r\n\r\nI'm not sure how GKE is solving the CUDA availability. I think it might be sufficient to manually copy the libraries from `/usr/local/cuda/extras/CUPTI/lib64` that contains \r\n`libcupti.so  libcupti.so.9.0  libcupti.so.9.0.176`. \r\n\r\nFew options:\r\n\r\n- Simply document the need to copy libs to the correct path\r\n- Collect the necessary CUDA libraries at nvidia-installer runtime and move copy them at install-time (this assumes you installed CUDA on the host)\r\n- Add a new flag that can be used to mount CUDA libraries from the desired host-path  \r\n\r\nI can submit a PR for this if anyone finds it useful."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/comments/373112681","html_url":"https://github.com/GoogleCloudPlatform/container-engine-accelerators/issues/60#issuecomment-373112681","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/container-engine-accelerators/issues/60","id":373112681,"user":{"login":"cmluciano","id":4790487,"avatar_url":"https://avatars1.githubusercontent.com/u/4790487?v=4","gravatar_id":"","url":"https://api.github.com/users/cmluciano","html_url":"https://github.com/cmluciano","followers_url":"https://api.github.com/users/cmluciano/followers","following_url":"https://api.github.com/users/cmluciano/following{/other_user}","gists_url":"https://api.github.com/users/cmluciano/gists{/gist_id}","starred_url":"https://api.github.com/users/cmluciano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmluciano/subscriptions","organizations_url":"https://api.github.com/users/cmluciano/orgs","repos_url":"https://api.github.com/users/cmluciano/repos","events_url":"https://api.github.com/users/cmluciano/events{/privacy}","received_events_url":"https://api.github.com/users/cmluciano/received_events","type":"User","site_admin":false},"created_at":"2018-03-14T17:43:40Z","updated_at":"2018-03-14T17:43:40Z","author_association":"NONE","body":"Ok. Before I close this, is it possible to mount the files from the package installed on the host or would that be a nightmare for potential CUDA compatibility problems?\r\n\r\ncc @mindprince @jiayingz Not sure what you are doing in GKE"}}
{"actor":{"display_login":"cmluciano"}}