{"action":"created","issue":{"url":"https://api.github.com/repos/apache/arrow/issues/1948","repository_url":"https://api.github.com/repos/apache/arrow","labels_url":"https://api.github.com/repos/apache/arrow/issues/1948/labels{/name}","comments_url":"https://api.github.com/repos/apache/arrow/issues/1948/comments","events_url":"https://api.github.com/repos/apache/arrow/issues/1948/events","html_url":"https://github.com/apache/arrow/pull/1948","id":317798582,"number":1948,"title":"ARROW-2512: [Python] Using Pytorch as Python interface for GPU objects","user":{"login":"Wapaul1","id":14142309,"avatar_url":"https://avatars2.githubusercontent.com/u/14142309?v=4","gravatar_id":"","url":"https://api.github.com/users/Wapaul1","html_url":"https://github.com/Wapaul1","followers_url":"https://api.github.com/users/Wapaul1/followers","following_url":"https://api.github.com/users/Wapaul1/following{/other_user}","gists_url":"https://api.github.com/users/Wapaul1/gists{/gist_id}","starred_url":"https://api.github.com/users/Wapaul1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Wapaul1/subscriptions","organizations_url":"https://api.github.com/users/Wapaul1/orgs","repos_url":"https://api.github.com/users/Wapaul1/repos","events_url":"https://api.github.com/users/Wapaul1/events{/privacy}","received_events_url":"https://api.github.com/users/Wapaul1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-25T21:21:43Z","updated_at":"2018-04-25T22:30:01Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/arrow/pulls/1948","html_url":"https://github.com/apache/arrow/pull/1948","diff_url":"https://github.com/apache/arrow/pull/1948.diff","patch_url":"https://github.com/apache/arrow/pull/1948.patch"},"body":"This is currently a work in progress. Pytorch 0.4.0 is required for this to work.\r\n\r\n## Instructions\r\nFirst, go to `arrow/python/pytorch_example` and run `python setup.py install`, which will install the extension to pytorch. Then build arrow normally with plasma and gpu enabled as below:\r\n```\r\ncmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \\\r\n      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \\\r\n      -DARROW_PYTHON=on \\\r\n      -DARROW_PLASMA=on \\\r\n      -DARROW_BUILD_TESTS=OFF \\\r\n      -DARROW_GPU=ON \\\r\n      ..\r\n```\r\nand build pyarrow with `python setup.py build_ext --build-type=$ARROW_BUILD_TYPE        --with-plasma --inplace` (this may not be necessary, this is just how I installed pyarrow).\r\n\r\nThen in ipython, create a plasma store and a client:\r\n```python\r\nimport os\r\nimport random\r\nimport shutil\r\nimport signal                       \r\nimport subprocess\r\nimport tempfile\r\nimport time\r\n \r\nimport numpy as np\r\nimport pyarrow as pa\r\nimport pandas as pd\r\n \r\n \r\nDEFAULT_PLASMA_STORE_MEMORY = 10 ** 8\r\nUSE_VALGRIND = os.getenv(\"PLASMA_VALGRIND\") == \"1\"\r\n \r\n \r\ndef random_name():\r\n    return str(random.randint(0, 99999999))\r\ndef start_plasma_store(plasma_store_memory=DEFAULT_PLASMA_STORE_MEMORY,\r\n                        stdout_file=None, stderr_file=None):\r\n     plasma_store_executable = os.path.join(pa.__path__[0], \"plasma_store\")\r\n     plasma_store_name = \"/tmp/plasma_store{}\".format(random_name())\r\n     command = [plasma_store_executable,\r\n               \"-s\", plasma_store_name,\r\n               \"-m\", str(plasma_store_memory)]\r\n     pid = subprocess.Popen(command, stdout=stdout_file, stderr=stderr_file)\r\n     time.sleep(0.1)\r\n     return plasma_store_name, pid\r\nimport pyarrow.plasma as plasma\r\n        # Start Plasma store.\r\nplasma_store_name, p = start_plasma_store()\r\nclient = plasma.connect(plasma_store_name, \"\", 64)\r\n\r\n```\r\n\r\nNow simply create a numpy float32 array, and pass it to `client.gpu_example`\r\n```\r\narr = np.random.rand(20,20).astype('float32')\r\nresult = client.gpu_example(arr)\r\n```\r\nwhich should return a pytorch tensor."},"comment":{"url":"https://api.github.com/repos/apache/arrow/issues/comments/384454163","html_url":"https://github.com/apache/arrow/pull/1948#issuecomment-384454163","issue_url":"https://api.github.com/repos/apache/arrow/issues/1948","id":384454163,"user":{"login":"pcmoritz","id":113316,"avatar_url":"https://avatars1.githubusercontent.com/u/113316?v=4","gravatar_id":"","url":"https://api.github.com/users/pcmoritz","html_url":"https://github.com/pcmoritz","followers_url":"https://api.github.com/users/pcmoritz/followers","following_url":"https://api.github.com/users/pcmoritz/following{/other_user}","gists_url":"https://api.github.com/users/pcmoritz/gists{/gist_id}","starred_url":"https://api.github.com/users/pcmoritz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcmoritz/subscriptions","organizations_url":"https://api.github.com/users/pcmoritz/orgs","repos_url":"https://api.github.com/users/pcmoritz/repos","events_url":"https://api.github.com/users/pcmoritz/events{/privacy}","received_events_url":"https://api.github.com/users/pcmoritz/received_events","type":"User","site_admin":false},"created_at":"2018-04-25T22:30:01Z","updated_at":"2018-04-25T22:30:01Z","author_association":"CONTRIBUTOR","body":"@pitrou Thanks for the feedback! You are right, we need to do some refactoring before this gets merged. Also pytorch won't be a dependency of plasma, the plan is to make it to have an example that shows how to load a tensor from plasma GPU memory directly into pytorch."}}
{"actor":{"display_login":"pcmoritz"}}