{"action":"created","comment":{"url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments/197683456","pull_request_review_id":131472396,"id":197683456,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NzY4MzQ1Ng==","diff_hunk":"@@ -141,13 +173,92 @@ class ModelBroadcast[T: ClassTag](applyProtoBuffer: Boolean = false)\n       Array()\n     }\n   }\n-\n }\n \n-\n object ModelBroadcast {\n   def apply[@specialized(Float, Double) T: ClassTag](applyProtoBuffer: Boolean = false)\n         (implicit ev: TensorNumeric[T]) : ModelBroadcast[T] = {\n     new ModelBroadcast(applyProtoBuffer)\n   }\n }\n+\n+class ModelInfo[T: ClassTag](val uuid: String, @transient var model: Module[T])(","path":"spark/dl/src/main/scala/com/intel/analytics/bigdl/models/utils/ModelBroadcast.scala","position":128,"original_position":128,"commit_id":"c19a8ed843054e288d7e67ac80e0b9217ae091e3","original_commit_id":"c19a8ed843054e288d7e67ac80e0b9217ae091e3","user":{"login":"wzhongyuan","id":1837261,"node_id":"MDQ6VXNlcjE4MzcyNjE=","avatar_url":"https://avatars3.githubusercontent.com/u/1837261?v=4","gravatar_id":"","url":"https://api.github.com/users/wzhongyuan","html_url":"https://github.com/wzhongyuan","followers_url":"https://api.github.com/users/wzhongyuan/followers","following_url":"https://api.github.com/users/wzhongyuan/following{/other_user}","gists_url":"https://api.github.com/users/wzhongyuan/gists{/gist_id}","starred_url":"https://api.github.com/users/wzhongyuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wzhongyuan/subscriptions","organizations_url":"https://api.github.com/users/wzhongyuan/orgs","repos_url":"https://api.github.com/users/wzhongyuan/repos","events_url":"https://api.github.com/users/wzhongyuan/events{/privacy}","received_events_url":"https://api.github.com/users/wzhongyuan/received_events","type":"User","site_admin":false},"body":"Why transient ? will it be ignored while serialized and deserialized between driver and executor ?","created_at":"2018-06-25T05:38:56Z","updated_at":"2018-06-25T05:38:57Z","html_url":"https://github.com/intel-analytics/BigDL/pull/2557#discussion_r197683456","pull_request_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments/197683456"},"html":{"href":"https://github.com/intel-analytics/BigDL/pull/2557#discussion_r197683456"},"pull_request":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557"}},"in_reply_to_id":197683289},"pull_request":{"url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557","id":194847517,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk0ODQ3NTE3","html_url":"https://github.com/intel-analytics/BigDL/pull/2557","diff_url":"https://github.com/intel-analytics/BigDL/pull/2557.diff","patch_url":"https://github.com/intel-analytics/BigDL/pull/2557.patch","issue_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2557","number":2557,"state":"open","locked":false,"title":"fix: memory leak in `model.predictImageSet`.","user":{"login":"i8run","id":2551195,"node_id":"MDQ6VXNlcjI1NTExOTU=","avatar_url":"https://avatars3.githubusercontent.com/u/2551195?v=4","gravatar_id":"","url":"https://api.github.com/users/i8run","html_url":"https://github.com/i8run","followers_url":"https://api.github.com/users/i8run/followers","following_url":"https://api.github.com/users/i8run/following{/other_user}","gists_url":"https://api.github.com/users/i8run/gists{/gist_id}","starred_url":"https://api.github.com/users/i8run/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/i8run/subscriptions","organizations_url":"https://api.github.com/users/i8run/orgs","repos_url":"https://api.github.com/users/i8run/repos","events_url":"https://api.github.com/users/i8run/events{/privacy}","received_events_url":"https://api.github.com/users/i8run/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\nThere're three reasons of memory leak.\r\n\r\n1. repeat allocations in bigquant, which will be fixed in BigDL-core.\r\n2. repeat clone module but no release. `model.predictImageSet` will new\r\n   Predictor again and again.\r\n3. share weights.\r\n\r\nBecause of multi models architecture, every `LocalPredictor` will clone models.\r\nFor efficiency all cloned models and origin model will share the same parameters.\r\nIf we iterate and release all cloned models, the parameters of origin model will\r\nbe released also. So we should clone a new model first in `LocalPredictor`.\r\n\r\nThis patch add a `StorageManager` which contains a concurrent hash map\r\nto maintain all allocations of native memory/resources and prevent\r\nduplicate release. It's also helpful for debug.\r\n\r\n## How was this patch tested?\r\n\r\nAdd a new test case and run Jenkins.\r\n\r\n## Related links or issues (optional)\r\nfixed #2535\r\n\r\n","created_at":"2018-06-14T10:29:01Z","updated_at":"2018-06-25T05:38:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8897bb9d674056a7678e64611e5eb571ff1a197f","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557/commits","review_comments_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557/comments","review_comment_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments{/number}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2557/comments","statuses_url":"https://api.github.com/repos/intel-analytics/BigDL/statuses/c19a8ed843054e288d7e67ac80e0b9217ae091e3","head":{"label":"i8run:quant-mem-leak","ref":"quant-mem-leak","sha":"c19a8ed843054e288d7e67ac80e0b9217ae091e3","user":{"login":"i8run","id":2551195,"node_id":"MDQ6VXNlcjI1NTExOTU=","avatar_url":"https://avatars3.githubusercontent.com/u/2551195?v=4","gravatar_id":"","url":"https://api.github.com/users/i8run","html_url":"https://github.com/i8run","followers_url":"https://api.github.com/users/i8run/followers","following_url":"https://api.github.com/users/i8run/following{/other_user}","gists_url":"https://api.github.com/users/i8run/gists{/gist_id}","starred_url":"https://api.github.com/users/i8run/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/i8run/subscriptions","organizations_url":"https://api.github.com/users/i8run/orgs","repos_url":"https://api.github.com/users/i8run/repos","events_url":"https://api.github.com/users/i8run/events{/privacy}","received_events_url":"https://api.github.com/users/i8run/received_events","type":"User","site_admin":false},"repo":{"id":78086686,"node_id":"MDEwOlJlcG9zaXRvcnk3ODA4NjY4Ng==","name":"BigDL-1","full_name":"i8run/BigDL-1","owner":{"login":"i8run","id":2551195,"node_id":"MDQ6VXNlcjI1NTExOTU=","avatar_url":"https://avatars3.githubusercontent.com/u/2551195?v=4","gravatar_id":"","url":"https://api.github.com/users/i8run","html_url":"https://github.com/i8run","followers_url":"https://api.github.com/users/i8run/followers","following_url":"https://api.github.com/users/i8run/following{/other_user}","gists_url":"https://api.github.com/users/i8run/gists{/gist_id}","starred_url":"https://api.github.com/users/i8run/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/i8run/subscriptions","organizations_url":"https://api.github.com/users/i8run/orgs","repos_url":"https://api.github.com/users/i8run/repos","events_url":"https://api.github.com/users/i8run/events{/privacy}","received_events_url":"https://api.github.com/users/i8run/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/i8run/BigDL-1","description":"BigDL: Distributed Deep learning Library for Apache Spark","fork":true,"url":"https://api.github.com/repos/i8run/BigDL-1","forks_url":"https://api.github.com/repos/i8run/BigDL-1/forks","keys_url":"https://api.github.com/repos/i8run/BigDL-1/keys{/key_id}","collaborators_url":"https://api.github.com/repos/i8run/BigDL-1/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/i8run/BigDL-1/teams","hooks_url":"https://api.github.com/repos/i8run/BigDL-1/hooks","issue_events_url":"https://api.github.com/repos/i8run/BigDL-1/issues/events{/number}","events_url":"https://api.github.com/repos/i8run/BigDL-1/events","assignees_url":"https://api.github.com/repos/i8run/BigDL-1/assignees{/user}","branches_url":"https://api.github.com/repos/i8run/BigDL-1/branches{/branch}","tags_url":"https://api.github.com/repos/i8run/BigDL-1/tags","blobs_url":"https://api.github.com/repos/i8run/BigDL-1/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/i8run/BigDL-1/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/i8run/BigDL-1/git/refs{/sha}","trees_url":"https://api.github.com/repos/i8run/BigDL-1/git/trees{/sha}","statuses_url":"https://api.github.com/repos/i8run/BigDL-1/statuses/{sha}","languages_url":"https://api.github.com/repos/i8run/BigDL-1/languages","stargazers_url":"https://api.github.com/repos/i8run/BigDL-1/stargazers","contributors_url":"https://api.github.com/repos/i8run/BigDL-1/contributors","subscribers_url":"https://api.github.com/repos/i8run/BigDL-1/subscribers","subscription_url":"https://api.github.com/repos/i8run/BigDL-1/subscription","commits_url":"https://api.github.com/repos/i8run/BigDL-1/commits{/sha}","git_commits_url":"https://api.github.com/repos/i8run/BigDL-1/git/commits{/sha}","comments_url":"https://api.github.com/repos/i8run/BigDL-1/comments{/number}","issue_comment_url":"https://api.github.com/repos/i8run/BigDL-1/issues/comments{/number}","contents_url":"https://api.github.com/repos/i8run/BigDL-1/contents/{+path}","compare_url":"https://api.github.com/repos/i8run/BigDL-1/compare/{base}...{head}","merges_url":"https://api.github.com/repos/i8run/BigDL-1/merges","archive_url":"https://api.github.com/repos/i8run/BigDL-1/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/i8run/BigDL-1/downloads","issues_url":"https://api.github.com/repos/i8run/BigDL-1/issues{/number}","pulls_url":"https://api.github.com/repos/i8run/BigDL-1/pulls{/number}","milestones_url":"https://api.github.com/repos/i8run/BigDL-1/milestones{/number}","notifications_url":"https://api.github.com/repos/i8run/BigDL-1/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/i8run/BigDL-1/labels{/name}","releases_url":"https://api.github.com/repos/i8run/BigDL-1/releases{/id}","deployments_url":"https://api.github.com/repos/i8run/BigDL-1/deployments","created_at":"2017-01-05T06:36:06Z","updated_at":"2017-05-10T08:50:43Z","pushed_at":"2018-06-25T02:01:02Z","git_url":"git://github.com/i8run/BigDL-1.git","ssh_url":"git@github.com:i8run/BigDL-1.git","clone_url":"https://github.com/i8run/BigDL-1.git","svn_url":"https://github.com/i8run/BigDL-1","homepage":"","size":22644,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"intel-analytics:master","ref":"master","sha":"2e5f18cae5b55e3c972438fd15eb79ea831c9b9a","user":{"login":"intel-analytics","id":10941215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwOTQxMjE1","avatar_url":"https://avatars3.githubusercontent.com/u/10941215?v=4","gravatar_id":"","url":"https://api.github.com/users/intel-analytics","html_url":"https://github.com/intel-analytics","followers_url":"https://api.github.com/users/intel-analytics/followers","following_url":"https://api.github.com/users/intel-analytics/following{/other_user}","gists_url":"https://api.github.com/users/intel-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/intel-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/intel-analytics/subscriptions","organizations_url":"https://api.github.com/users/intel-analytics/orgs","repos_url":"https://api.github.com/users/intel-analytics/repos","events_url":"https://api.github.com/users/intel-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/intel-analytics/received_events","type":"Organization","site_admin":false},"repo":{"id":66823715,"node_id":"MDEwOlJlcG9zaXRvcnk2NjgyMzcxNQ==","name":"BigDL","full_name":"intel-analytics/BigDL","owner":{"login":"intel-analytics","id":10941215,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwOTQxMjE1","avatar_url":"https://avatars3.githubusercontent.com/u/10941215?v=4","gravatar_id":"","url":"https://api.github.com/users/intel-analytics","html_url":"https://github.com/intel-analytics","followers_url":"https://api.github.com/users/intel-analytics/followers","following_url":"https://api.github.com/users/intel-analytics/following{/other_user}","gists_url":"https://api.github.com/users/intel-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/intel-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/intel-analytics/subscriptions","organizations_url":"https://api.github.com/users/intel-analytics/orgs","repos_url":"https://api.github.com/users/intel-analytics/repos","events_url":"https://api.github.com/users/intel-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/intel-analytics/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/intel-analytics/BigDL","description":"BigDL: Distributed Deep Learning Library for Apache Spark","fork":false,"url":"https://api.github.com/repos/intel-analytics/BigDL","forks_url":"https://api.github.com/repos/intel-analytics/BigDL/forks","keys_url":"https://api.github.com/repos/intel-analytics/BigDL/keys{/key_id}","collaborators_url":"https://api.github.com/repos/intel-analytics/BigDL/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/intel-analytics/BigDL/teams","hooks_url":"https://api.github.com/repos/intel-analytics/BigDL/hooks","issue_events_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/events{/number}","events_url":"https://api.github.com/repos/intel-analytics/BigDL/events","assignees_url":"https://api.github.com/repos/intel-analytics/BigDL/assignees{/user}","branches_url":"https://api.github.com/repos/intel-analytics/BigDL/branches{/branch}","tags_url":"https://api.github.com/repos/intel-analytics/BigDL/tags","blobs_url":"https://api.github.com/repos/intel-analytics/BigDL/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/intel-analytics/BigDL/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/intel-analytics/BigDL/git/refs{/sha}","trees_url":"https://api.github.com/repos/intel-analytics/BigDL/git/trees{/sha}","statuses_url":"https://api.github.com/repos/intel-analytics/BigDL/statuses/{sha}","languages_url":"https://api.github.com/repos/intel-analytics/BigDL/languages","stargazers_url":"https://api.github.com/repos/intel-analytics/BigDL/stargazers","contributors_url":"https://api.github.com/repos/intel-analytics/BigDL/contributors","subscribers_url":"https://api.github.com/repos/intel-analytics/BigDL/subscribers","subscription_url":"https://api.github.com/repos/intel-analytics/BigDL/subscription","commits_url":"https://api.github.com/repos/intel-analytics/BigDL/commits{/sha}","git_commits_url":"https://api.github.com/repos/intel-analytics/BigDL/git/commits{/sha}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/comments{/number}","issue_comment_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/comments{/number}","contents_url":"https://api.github.com/repos/intel-analytics/BigDL/contents/{+path}","compare_url":"https://api.github.com/repos/intel-analytics/BigDL/compare/{base}...{head}","merges_url":"https://api.github.com/repos/intel-analytics/BigDL/merges","archive_url":"https://api.github.com/repos/intel-analytics/BigDL/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/intel-analytics/BigDL/downloads","issues_url":"https://api.github.com/repos/intel-analytics/BigDL/issues{/number}","pulls_url":"https://api.github.com/repos/intel-analytics/BigDL/pulls{/number}","milestones_url":"https://api.github.com/repos/intel-analytics/BigDL/milestones{/number}","notifications_url":"https://api.github.com/repos/intel-analytics/BigDL/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/intel-analytics/BigDL/labels{/name}","releases_url":"https://api.github.com/repos/intel-analytics/BigDL/releases{/id}","deployments_url":"https://api.github.com/repos/intel-analytics/BigDL/deployments","created_at":"2016-08-29T07:59:50Z","updated_at":"2018-06-23T02:15:35Z","pushed_at":"2018-06-25T05:14:20Z","git_url":"git://github.com/intel-analytics/BigDL.git","ssh_url":"git@github.com:intel-analytics/BigDL.git","clone_url":"https://github.com/intel-analytics/BigDL.git","svn_url":"https://github.com/intel-analytics/BigDL","homepage":"https://bigdl-project.github.io/","size":21825,"stargazers_count":2523,"watchers_count":2523,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":572,"mirror_url":null,"archived":false,"open_issues_count":143,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":572,"open_issues":143,"watchers":2523,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557"},"html":{"href":"https://github.com/intel-analytics/BigDL/pull/2557"},"issue":{"href":"https://api.github.com/repos/intel-analytics/BigDL/issues/2557"},"comments":{"href":"https://api.github.com/repos/intel-analytics/BigDL/issues/2557/comments"},"review_comments":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557/comments"},"review_comment":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/intel-analytics/BigDL/pulls/2557/commits"},"statuses":{"href":"https://api.github.com/repos/intel-analytics/BigDL/statuses/c19a8ed843054e288d7e67ac80e0b9217ae091e3"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"wzhongyuan"}}