{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-systemml/issues/211","repository_url":"https://api.github.com/repos/apache/incubator-systemml","labels_url":"https://api.github.com/repos/apache/incubator-systemml/issues/211/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-systemml/issues/211/comments","events_url":"https://api.github.com/repos/apache/incubator-systemml/issues/211/events","html_url":"https://github.com/apache/incubator-systemml/pull/211","id":170970755,"number":211,"title":"[SYSTEMML-853] Python API for new MLContext","user":{"login":"MechCoder","id":1867024,"avatar_url":"https://avatars.githubusercontent.com/u/1867024?v=3","gravatar_id":"","url":"https://api.github.com/users/MechCoder","html_url":"https://github.com/MechCoder","followers_url":"https://api.github.com/users/MechCoder/followers","following_url":"https://api.github.com/users/MechCoder/following{/other_user}","gists_url":"https://api.github.com/users/MechCoder/gists{/gist_id}","starred_url":"https://api.github.com/users/MechCoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MechCoder/subscriptions","organizations_url":"https://api.github.com/users/MechCoder/orgs","repos_url":"https://api.github.com/users/MechCoder/repos","events_url":"https://api.github.com/users/MechCoder/events{/privacy}","received_events_url":"https://api.github.com/users/MechCoder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-12T22:26:26Z","updated_at":"2016-08-19T19:54:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-systemml/pulls/211","html_url":"https://github.com/apache/incubator-systemml/pull/211","diff_url":"https://github.com/apache/incubator-systemml/pull/211.diff","patch_url":"https://github.com/apache/incubator-systemml/pull/211.patch"},"body":"This is the Python API for the new MLContext\r\n\r\nHere is a quick example\r\n\r\n```python\r\nsums = \"\"\"\r\ns1 = sum(m1)\r\ns2 = sum(m2)\r\ns3 = 'whatever'\r\n\"\"\"\r\nml = MLContext(sc)\r\nrdd1 = sc.parallelize([\"1.0,2.0\", \"3.0,4.0\"])\r\nrdd2 = sc.parallelize([\"5.0,6.0\", \"7.0,8.0\"])\r\ndml = Dml(sums).input(m1=rdd1).input(m2=rdd2).out(\"s1\", \"s2\", \"s3\")\r\nml.execute(dml).get(\"s1\", \"s2\", \"s3\")\r\n[10.0, 26.0, \"whatever\"]\r\n```\r\n\r\nThe tests cases should explain the other use cases."},"comment":{"url":"https://api.github.com/repos/apache/incubator-systemml/issues/comments/241118588","html_url":"https://github.com/apache/incubator-systemml/pull/211#issuecomment-241118588","issue_url":"https://api.github.com/repos/apache/incubator-systemml/issues/211","id":241118588,"user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"created_at":"2016-08-19T19:54:22Z","updated_at":"2016-08-19T19:54:22Z","body":"@MechCoder This is great!  Here's a list of things I tried out, along with some additional thoughts:\r\n\r\n1. Tried this out the example on Python 3.\r\n  * Can you update the example in the initial post to reflect the recent update, as in:\r\n    \r\n      ```\r\n      sums = \"\"\"\r\n      s1 = sum(m1)\r\n      s2 = sum(m2)\r\n      s3 = 'whatever'\r\n      \"\"\"\r\n      ml = MLContext(sc)\r\n      rdd1 = sc.parallelize([\"1.0,2.0\", \"3.0,4.0\"])\r\n      rdd2 = sc.parallelize([\"5.0,6.0\", \"7.0,8.0\"])\r\n      dml = Dml(sums).input(m1=rdd1).input(m2=rdd2).out(\"s1\", \"s2\", \"s3\")\r\n      ml.execute(dml).get(\"s1\", \"s2\", \"s3\")\r\n      [10.0, 26.0, \"whatever\"]\r\n      ```\r\n\r\n2. Ran the Python tests with `spark-submit` (after adding the missing comma).\r\n  * I used `PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path $SYSTEMML_HOME/target/SystemML.jar --py-files=$SYSTEMML_HOME/python/SystemML.py $SYSTEMML_HOME/python/SystemMLtests.py`\r\n3. Tried to alter the script so that `s2` would return a matrix, i.e. `s2 = m1 * 3`.\r\n  * Right now, this just returns a `Java object` reflecting a `mlcontext.Matrix` type that was added (  @deroneriksson correct if this is wrong).  We should add a `Matrix` type to our Python API here, and then add some logic to convert this to a DataFrame, RDD, etc. as is done on the Java/Scala side.  Should just be another simple wrapper over \\[1].\r\n\r\nSome other thoughts:\r\n\r\n1. Can you move the files from `python/` to `src/main/python` just for consistency with the Java and Scala code?  I know Spark has a `python/` folder, but they also have the project split into multiple modules, so they have to consolidate the Python code into a single top-level `python/` folder to allow for imports to work properly.  We only have one single module, so that won't be a problem.\r\n2. In a follow-up PR, could you look into adding a `setup.py` file and any other requirements so that we could create a `pip` package?  Then we could allow for `pip` installs directly from the git repo, using a subdirectory flag pointing to `src/main/python` \\[2].\r\n\r\n\\[1]: http://apache.github.io/incubator-systemml/spark-mlcontext-programming-guide#matrix-output\r\n\\[2]: http://stackoverflow.com/a/36074026"}}
{"actor":{"display_login":"dusenberrymw"}}