{"action":"created","comment":{"url":"https://api.github.com/repos/databricks/mlflow/pulls/comments/198358882","pull_request_review_id":132275914,"id":198358882,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5ODM1ODg4Mg==","diff_hunk":"@@ -0,0 +1,116 @@\n+\"\"\"\n+MLflow integration for SparkML models.\n+\n+SparkML models are saved and loaded using native SparkML persistence.\n+The models can be exported as pyfunc for out-of Spark deployment or it can be loaded back as Spark\n+Transformer in order to score it in Spark. The pyfunc flavor instantiates SparkContext internally\n+and reads the input data as Spark DataFrame prior to scoring.\n+\n+Note: SparkML model can not be exported as spark_udf as those two formats are incompatible.","path":"mlflow/sparkml.py","position":9,"original_position":9,"commit_id":"3a8e94e6a3f9269f539370683dd11fe5dd51ba95","original_commit_id":"3a8e94e6a3f9269f539370683dd11fe5dd51ba95","user":{"login":"tomasatdatabricks","id":33237569,"node_id":"MDQ6VXNlcjMzMjM3NTY5","avatar_url":"https://avatars3.githubusercontent.com/u/33237569?v=4","gravatar_id":"","url":"https://api.github.com/users/tomasatdatabricks","html_url":"https://github.com/tomasatdatabricks","followers_url":"https://api.github.com/users/tomasatdatabricks/followers","following_url":"https://api.github.com/users/tomasatdatabricks/following{/other_user}","gists_url":"https://api.github.com/users/tomasatdatabricks/gists{/gist_id}","starred_url":"https://api.github.com/users/tomasatdatabricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomasatdatabricks/subscriptions","organizations_url":"https://api.github.com/users/tomasatdatabricks/orgs","repos_url":"https://api.github.com/users/tomasatdatabricks/repos","events_url":"https://api.github.com/users/tomasatdatabricks/events{/privacy}","received_events_url":"https://api.github.com/users/tomasatdatabricks/received_events","type":"User","site_admin":false},"body":"Yes I did run it. The spark context is not a problem but lack of column names inside of spark_udf is. Since most spark ml models require column names it still does not work. I could add column names to the model or just this flavor and set them on the Dataframe before it is passed to the model. There is no way to extract feature names from PipelineModel, so user would have to provide those explicitly.","created_at":"2018-06-27T03:42:17Z","updated_at":"2018-06-27T03:42:17Z","html_url":"https://github.com/databricks/mlflow/pull/72#discussion_r198358882","pull_request_url":"https://api.github.com/repos/databricks/mlflow/pulls/72","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/comments/198358882"},"html":{"href":"https://github.com/databricks/mlflow/pull/72#discussion_r198358882"},"pull_request":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/72"}},"in_reply_to_id":198335254},"pull_request":{"url":"https://api.github.com/repos/databricks/mlflow/pulls/72","id":196353978,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk2MzUzOTc4","html_url":"https://github.com/databricks/mlflow/pull/72","diff_url":"https://github.com/databricks/mlflow/pull/72.diff","patch_url":"https://github.com/databricks/mlflow/pull/72.patch","issue_url":"https://api.github.com/repos/databricks/mlflow/issues/72","number":72,"state":"open","locked":false,"title":"Initial commit for the SparkML model export.","user":{"login":"tomasatdatabricks","id":33237569,"node_id":"MDQ6VXNlcjMzMjM3NTY5","avatar_url":"https://avatars3.githubusercontent.com/u/33237569?v=4","gravatar_id":"","url":"https://api.github.com/users/tomasatdatabricks","html_url":"https://github.com/tomasatdatabricks","followers_url":"https://api.github.com/users/tomasatdatabricks/followers","following_url":"https://api.github.com/users/tomasatdatabricks/following{/other_user}","gists_url":"https://api.github.com/users/tomasatdatabricks/gists{/gist_id}","starred_url":"https://api.github.com/users/tomasatdatabricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomasatdatabricks/subscriptions","organizations_url":"https://api.github.com/users/tomasatdatabricks/orgs","repos_url":"https://api.github.com/users/tomasatdatabricks/repos","events_url":"https://api.github.com/users/tomasatdatabricks/events{/privacy}","received_events_url":"https://api.github.com/users/tomasatdatabricks/received_events","type":"User","site_admin":false},"body":"Initial stab at exporting SparkML models. Also as part of this moved spark_udf outside of pyfunc and created a separate flavor for it. The flavor is required because not all pyfuncs have to support spark udf mode, e.g. the pyfunc generated by SparkML flavor does not.\r\n\r\nFew open questions / TODOs:\r\n  1. how to specify jar dependencies? For now unimplemented.\r\n  2. What should be the model output? For now only produce 1-d prediction to be consistent with other flavors. We should probably revisit that for all the models in one go.\r\n 3. Maybe we should add load-model-as-spark-transformer interface in addition to existing spark_udf.\r\n","created_at":"2018-06-21T07:04:24Z","updated_at":"2018-06-27T03:42:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"91fded5129fceb34d0f90ccb9773a30b4515007b","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/databricks/mlflow/pulls/72/commits","review_comments_url":"https://api.github.com/repos/databricks/mlflow/pulls/72/comments","review_comment_url":"https://api.github.com/repos/databricks/mlflow/pulls/comments{/number}","comments_url":"https://api.github.com/repos/databricks/mlflow/issues/72/comments","statuses_url":"https://api.github.com/repos/databricks/mlflow/statuses/3a8e94e6a3f9269f539370683dd11fe5dd51ba95","head":{"label":"tomasatdatabricks:tomas/sparkml_export","ref":"tomas/sparkml_export","sha":"3a8e94e6a3f9269f539370683dd11fe5dd51ba95","user":{"login":"tomasatdatabricks","id":33237569,"node_id":"MDQ6VXNlcjMzMjM3NTY5","avatar_url":"https://avatars3.githubusercontent.com/u/33237569?v=4","gravatar_id":"","url":"https://api.github.com/users/tomasatdatabricks","html_url":"https://github.com/tomasatdatabricks","followers_url":"https://api.github.com/users/tomasatdatabricks/followers","following_url":"https://api.github.com/users/tomasatdatabricks/following{/other_user}","gists_url":"https://api.github.com/users/tomasatdatabricks/gists{/gist_id}","starred_url":"https://api.github.com/users/tomasatdatabricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomasatdatabricks/subscriptions","organizations_url":"https://api.github.com/users/tomasatdatabricks/orgs","repos_url":"https://api.github.com/users/tomasatdatabricks/repos","events_url":"https://api.github.com/users/tomasatdatabricks/events{/privacy}","received_events_url":"https://api.github.com/users/tomasatdatabricks/received_events","type":"User","site_admin":false},"repo":{"id":136542352,"node_id":"MDEwOlJlcG9zaXRvcnkxMzY1NDIzNTI=","name":"mlflow","full_name":"tomasatdatabricks/mlflow","owner":{"login":"tomasatdatabricks","id":33237569,"node_id":"MDQ6VXNlcjMzMjM3NTY5","avatar_url":"https://avatars3.githubusercontent.com/u/33237569?v=4","gravatar_id":"","url":"https://api.github.com/users/tomasatdatabricks","html_url":"https://github.com/tomasatdatabricks","followers_url":"https://api.github.com/users/tomasatdatabricks/followers","following_url":"https://api.github.com/users/tomasatdatabricks/following{/other_user}","gists_url":"https://api.github.com/users/tomasatdatabricks/gists{/gist_id}","starred_url":"https://api.github.com/users/tomasatdatabricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomasatdatabricks/subscriptions","organizations_url":"https://api.github.com/users/tomasatdatabricks/orgs","repos_url":"https://api.github.com/users/tomasatdatabricks/repos","events_url":"https://api.github.com/users/tomasatdatabricks/events{/privacy}","received_events_url":"https://api.github.com/users/tomasatdatabricks/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tomasatdatabricks/mlflow","description":"Open source platform for the complete machine learning lifecycle","fork":true,"url":"https://api.github.com/repos/tomasatdatabricks/mlflow","forks_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/forks","keys_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/teams","hooks_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/hooks","issue_events_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/issues/events{/number}","events_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/events","assignees_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/assignees{/user}","branches_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/branches{/branch}","tags_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/tags","blobs_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/git/refs{/sha}","trees_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/statuses/{sha}","languages_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/languages","stargazers_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/stargazers","contributors_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/contributors","subscribers_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/subscribers","subscription_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/subscription","commits_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/commits{/sha}","git_commits_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/git/commits{/sha}","comments_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/comments{/number}","issue_comment_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/issues/comments{/number}","contents_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/contents/{+path}","compare_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/merges","archive_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/downloads","issues_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/issues{/number}","pulls_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/pulls{/number}","milestones_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/milestones{/number}","notifications_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/labels{/name}","releases_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/releases{/id}","deployments_url":"https://api.github.com/repos/tomasatdatabricks/mlflow/deployments","created_at":"2018-06-07T23:36:27Z","updated_at":"2018-06-07T23:36:29Z","pushed_at":"2018-06-26T18:40:35Z","git_url":"git://github.com/tomasatdatabricks/mlflow.git","ssh_url":"git@github.com:tomasatdatabricks/mlflow.git","clone_url":"https://github.com/tomasatdatabricks/mlflow.git","svn_url":"https://github.com/tomasatdatabricks/mlflow","homepage":"https://mlflow.org","size":3129,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"databricks:master","ref":"master","sha":"1040d425d41f1299e444cdd972d56a95971aee70","user":{"login":"databricks","id":4998052,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ5OTgwNTI=","avatar_url":"https://avatars2.githubusercontent.com/u/4998052?v=4","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"repo":{"id":136202695,"node_id":"MDEwOlJlcG9zaXRvcnkxMzYyMDI2OTU=","name":"mlflow","full_name":"databricks/mlflow","owner":{"login":"databricks","id":4998052,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ5OTgwNTI=","avatar_url":"https://avatars2.githubusercontent.com/u/4998052?v=4","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/databricks/mlflow","description":"Open source platform for the complete machine learning lifecycle","fork":false,"url":"https://api.github.com/repos/databricks/mlflow","forks_url":"https://api.github.com/repos/databricks/mlflow/forks","keys_url":"https://api.github.com/repos/databricks/mlflow/keys{/key_id}","collaborators_url":"https://api.github.com/repos/databricks/mlflow/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/databricks/mlflow/teams","hooks_url":"https://api.github.com/repos/databricks/mlflow/hooks","issue_events_url":"https://api.github.com/repos/databricks/mlflow/issues/events{/number}","events_url":"https://api.github.com/repos/databricks/mlflow/events","assignees_url":"https://api.github.com/repos/databricks/mlflow/assignees{/user}","branches_url":"https://api.github.com/repos/databricks/mlflow/branches{/branch}","tags_url":"https://api.github.com/repos/databricks/mlflow/tags","blobs_url":"https://api.github.com/repos/databricks/mlflow/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/databricks/mlflow/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/databricks/mlflow/git/refs{/sha}","trees_url":"https://api.github.com/repos/databricks/mlflow/git/trees{/sha}","statuses_url":"https://api.github.com/repos/databricks/mlflow/statuses/{sha}","languages_url":"https://api.github.com/repos/databricks/mlflow/languages","stargazers_url":"https://api.github.com/repos/databricks/mlflow/stargazers","contributors_url":"https://api.github.com/repos/databricks/mlflow/contributors","subscribers_url":"https://api.github.com/repos/databricks/mlflow/subscribers","subscription_url":"https://api.github.com/repos/databricks/mlflow/subscription","commits_url":"https://api.github.com/repos/databricks/mlflow/commits{/sha}","git_commits_url":"https://api.github.com/repos/databricks/mlflow/git/commits{/sha}","comments_url":"https://api.github.com/repos/databricks/mlflow/comments{/number}","issue_comment_url":"https://api.github.com/repos/databricks/mlflow/issues/comments{/number}","contents_url":"https://api.github.com/repos/databricks/mlflow/contents/{+path}","compare_url":"https://api.github.com/repos/databricks/mlflow/compare/{base}...{head}","merges_url":"https://api.github.com/repos/databricks/mlflow/merges","archive_url":"https://api.github.com/repos/databricks/mlflow/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/databricks/mlflow/downloads","issues_url":"https://api.github.com/repos/databricks/mlflow/issues{/number}","pulls_url":"https://api.github.com/repos/databricks/mlflow/pulls{/number}","milestones_url":"https://api.github.com/repos/databricks/mlflow/milestones{/number}","notifications_url":"https://api.github.com/repos/databricks/mlflow/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/databricks/mlflow/labels{/name}","releases_url":"https://api.github.com/repos/databricks/mlflow/releases{/id}","deployments_url":"https://api.github.com/repos/databricks/mlflow/deployments","created_at":"2018-06-05T16:05:58Z","updated_at":"2018-06-27T02:53:21Z","pushed_at":"2018-06-27T02:53:18Z","git_url":"git://github.com/databricks/mlflow.git","ssh_url":"git@github.com:databricks/mlflow.git","clone_url":"https://github.com/databricks/mlflow.git","svn_url":"https://github.com/databricks/mlflow","homepage":"https://mlflow.org","size":3065,"stargazers_count":1333,"watchers_count":1333,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":169,"mirror_url":null,"archived":false,"open_issues_count":22,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":169,"open_issues":22,"watchers":1333,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/72"},"html":{"href":"https://github.com/databricks/mlflow/pull/72"},"issue":{"href":"https://api.github.com/repos/databricks/mlflow/issues/72"},"comments":{"href":"https://api.github.com/repos/databricks/mlflow/issues/72/comments"},"review_comments":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/72/comments"},"review_comment":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/databricks/mlflow/pulls/72/commits"},"statuses":{"href":"https://api.github.com/repos/databricks/mlflow/statuses/3a8e94e6a3f9269f539370683dd11fe5dd51ba95"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"tomasatdatabricks"}}