{"action":"created","comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/comments/172064374","pull_request_review_id":101019437,"id":172064374,"diff_hunk":"@@ -1386,8 +1386,37 @@ MXNET_DLL int MXSymbolInferType(SymbolHandle sym,\n                                 const int **aux_type_data,\n                                 int *complete);\n \n-\n-\n+/*!\n+ * \\brief Convert a symbol into a quantized symbol where FP32 operators are replaced with INT8\n+ * \\param sym_handle symbol to be converted\n+ * \\param ret_sym_handle quantized symbol result\n+ * \\param num_excluded_symbols number of layers excluded from being quantized in the input symbol\n+ * \\param excluded_symbols array of symbols to be excluded from being quantized\n+ * \\param num_offline number of parameters that are quantized offline\n+ * \\param offline_params array of c strings representing the names of params quantized offline\n+ */\n+MXNET_DLL int MXQuantizeSymbol(SymbolHandle sym_handle,\n+                               SymbolHandle *ret_sym_handle,\n+                               mx_uint num_excluded_symbols,\n+                               SymbolHandle *excluded_symbols,","path":"include/mxnet/c_api.h","position":18,"original_position":18,"commit_id":"662570bf1257616568caf828ad1513de8cd88d7b","original_commit_id":"662570bf1257616568caf828ad1513de8cd88d7b","user":{"login":"KellenSunderland","id":7443219,"avatar_url":"https://avatars0.githubusercontent.com/u/7443219?v=4","gravatar_id":"","url":"https://api.github.com/users/KellenSunderland","html_url":"https://github.com/KellenSunderland","followers_url":"https://api.github.com/users/KellenSunderland/followers","following_url":"https://api.github.com/users/KellenSunderland/following{/other_user}","gists_url":"https://api.github.com/users/KellenSunderland/gists{/gist_id}","starred_url":"https://api.github.com/users/KellenSunderland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KellenSunderland/subscriptions","organizations_url":"https://api.github.com/users/KellenSunderland/orgs","repos_url":"https://api.github.com/users/KellenSunderland/repos","events_url":"https://api.github.com/users/KellenSunderland/events{/privacy}","received_events_url":"https://api.github.com/users/KellenSunderland/received_events","type":"User","site_admin":false},"body":"Are excluded_symbols mutable?  Just wondering if they should be const here.","created_at":"2018-03-04T21:03:28Z","updated_at":"2018-03-04T21:03:29Z","html_url":"https://github.com/apache/incubator-mxnet/pull/9552#discussion_r172064374","pull_request_url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/comments/172064374"},"html":{"href":"https://github.com/apache/incubator-mxnet/pull/9552#discussion_r172064374"},"pull_request":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552"}}},"pull_request":{"url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552","id":165000783,"html_url":"https://github.com/apache/incubator-mxnet/pull/9552","diff_url":"https://github.com/apache/incubator-mxnet/pull/9552.diff","patch_url":"https://github.com/apache/incubator-mxnet/pull/9552.patch","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9552","number":9552,"state":"open","locked":false,"title":"[REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration","user":{"login":"reminisce","id":4978794,"avatar_url":"https://avatars1.githubusercontent.com/u/4978794?v=4","gravatar_id":"","url":"https://api.github.com/users/reminisce","html_url":"https://github.com/reminisce","followers_url":"https://api.github.com/users/reminisce/followers","following_url":"https://api.github.com/users/reminisce/following{/other_user}","gists_url":"https://api.github.com/users/reminisce/gists{/gist_id}","starred_url":"https://api.github.com/users/reminisce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reminisce/subscriptions","organizations_url":"https://api.github.com/users/reminisce/orgs","repos_url":"https://api.github.com/users/reminisce/repos","events_url":"https://api.github.com/users/reminisce/events{/privacy}","received_events_url":"https://api.github.com/users/reminisce/received_events","type":"User","site_admin":false},"body":"## Description ##\r\nThis PR implements model **quantization** by adopting the [TensorFlow approach](https://www.tensorflow.org/performance/quantization) with **calibration** by borrowing the idea from Nvidia's [TensorRT](http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf). The focus of this work is on keeping quantized models (ConvNets for now) inference accuracy loss under control when compared to their corresponding FP32 models. It also provides a framework in MXNet for easily plugging in high-performance operators for low-bit operations generated by using [TVM](https://github.com/dmlc/tvm).\r\n\r\nThis is a joint work of @ZihengJiang and @reminisce.\r\n- @ZihengJiang implemented the model quantization flow and quantized operators by calling [cuDNN APIs](http://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#api-introduction) for convolution, pooling, and fully-connected operators.\r\n- @reminisce implemented the calibration flow and refactored the operator implementation into using nnvm interfaces as well as wrote unit tests and examples, designed user-level API, conducted benchmarks, and fixed bugs to make the code mergeable to MXNet master branch.\r\n\r\n## Details ##\r\nPlease see the following slides for more details on implementation and benchmark results.\r\n[quantization_github.pptx](https://github.com/apache/incubator-mxnet/files/1662073/quantization_github.pptx)\r\n\r\n## Code Structure ##\r\n- Backend: `src/operator/quantization/` contains quantized operators, quantization and calibration flow, and quantization util functions.\r\n- Frontend: `python/mxnet/quantization.py` contains one user API for generating quantized models from FP32 models.\r\n- Examples: `example/quantization/` contains examples of generating quantized models and using quantized models for inference.\r\n- Unit tests: `tests/python/quantization/` contains unit tests for quantization.\r\n\r\n## Notes ##\r\n- Since we have used cuDNN for implementing the quantized operators, the quantized models generated in the examples of this PR can only run on the Nvidia GPUs supporting the [dp4a instruction](https://devblogs.nvidia.com/mixed-precision-programming-cuda-8/) for inference. We performed our benchmarks on [AWS P3 instances](https://aws.amazon.com/ec2/instance-types/p3/).\r\n- The inference speed of the quantized models is about 50% slower than FP32 models. This is majorly resulted from three transpose operations in the quantized convolution operator to transform data layouts between NCHW and NHWC in order to call [`cudnnConvolutionForward`](http://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnConvolutionForward). In addition, we have noticed that even without transposing data layouts, the INT8 convolution of NHWC is slower than FP32 of NCHW for big images such as `(64, 56, 56)`. In the future, we hope to leverage the strength of [TVM](https://github.com/dmlc/tvm) to generate high-performance INT8 operators to replace the current implementation of calling cuDNN for quantized convolution.\r\n- The unit tests of quantization are put under `tests/python/quantization` because it needs a P3 instance to run. @marcoabreu is working on setting up the testing environment. Once it's done, we will submit the unit tests under that folder to a different label from the commonly used one.\r\n\r\nWe would like to thank all the following people for discussion, suggestion, providing datasets, and guidance on configuring examples. @mli @piiswrong @zhreshold @astonzhang @szha @eric-haibin-lin @srochel @madjam @bhavinthaker @marcoabreu \r\n\r\nWe would appreciate everyone's efforts of reviewing this PR.\r\n\r\n@cjolivier01 @anirudh2290 @rahul003 ","created_at":"2018-01-25T00:31:15Z","updated_at":"2018-03-04T21:03:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"359cbcce879e34d9c92c8cf199e86a4dcad3bb2b","assignee":null,"assignees":[],"requested_reviewers":[{"login":"szha","id":2626883,"avatar_url":"https://avatars2.githubusercontent.com/u/2626883?v=4","gravatar_id":"","url":"https://api.github.com/users/szha","html_url":"https://github.com/szha","followers_url":"https://api.github.com/users/szha/followers","following_url":"https://api.github.com/users/szha/following{/other_user}","gists_url":"https://api.github.com/users/szha/gists{/gist_id}","starred_url":"https://api.github.com/users/szha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szha/subscriptions","organizations_url":"https://api.github.com/users/szha/orgs","repos_url":"https://api.github.com/users/szha/repos","events_url":"https://api.github.com/users/szha/events{/privacy}","received_events_url":"https://api.github.com/users/szha/received_events","type":"User","site_admin":false},{"login":"cjolivier01","id":11234557,"avatar_url":"https://avatars1.githubusercontent.com/u/11234557?v=4","gravatar_id":"","url":"https://api.github.com/users/cjolivier01","html_url":"https://github.com/cjolivier01","followers_url":"https://api.github.com/users/cjolivier01/followers","following_url":"https://api.github.com/users/cjolivier01/following{/other_user}","gists_url":"https://api.github.com/users/cjolivier01/gists{/gist_id}","starred_url":"https://api.github.com/users/cjolivier01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjolivier01/subscriptions","organizations_url":"https://api.github.com/users/cjolivier01/orgs","repos_url":"https://api.github.com/users/cjolivier01/repos","events_url":"https://api.github.com/users/cjolivier01/events{/privacy}","received_events_url":"https://api.github.com/users/cjolivier01/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552/commits","review_comments_url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552/comments","review_comment_url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9552/comments","statuses_url":"https://api.github.com/repos/apache/incubator-mxnet/statuses/662570bf1257616568caf828ad1513de8cd88d7b","head":{"label":"reminisce:merge_quantization_to_master","ref":"merge_quantization_to_master","sha":"662570bf1257616568caf828ad1513de8cd88d7b","user":{"login":"reminisce","id":4978794,"avatar_url":"https://avatars1.githubusercontent.com/u/4978794?v=4","gravatar_id":"","url":"https://api.github.com/users/reminisce","html_url":"https://github.com/reminisce","followers_url":"https://api.github.com/users/reminisce/followers","following_url":"https://api.github.com/users/reminisce/following{/other_user}","gists_url":"https://api.github.com/users/reminisce/gists{/gist_id}","starred_url":"https://api.github.com/users/reminisce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reminisce/subscriptions","organizations_url":"https://api.github.com/users/reminisce/orgs","repos_url":"https://api.github.com/users/reminisce/repos","events_url":"https://api.github.com/users/reminisce/events{/privacy}","received_events_url":"https://api.github.com/users/reminisce/received_events","type":"User","site_admin":false},"repo":{"id":79397572,"name":"mxnet","full_name":"reminisce/mxnet","owner":{"login":"reminisce","id":4978794,"avatar_url":"https://avatars1.githubusercontent.com/u/4978794?v=4","gravatar_id":"","url":"https://api.github.com/users/reminisce","html_url":"https://github.com/reminisce","followers_url":"https://api.github.com/users/reminisce/followers","following_url":"https://api.github.com/users/reminisce/following{/other_user}","gists_url":"https://api.github.com/users/reminisce/gists{/gist_id}","starred_url":"https://api.github.com/users/reminisce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reminisce/subscriptions","organizations_url":"https://api.github.com/users/reminisce/orgs","repos_url":"https://api.github.com/users/reminisce/repos","events_url":"https://api.github.com/users/reminisce/events{/privacy}","received_events_url":"https://api.github.com/users/reminisce/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/reminisce/mxnet","description":"Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more","fork":true,"url":"https://api.github.com/repos/reminisce/mxnet","forks_url":"https://api.github.com/repos/reminisce/mxnet/forks","keys_url":"https://api.github.com/repos/reminisce/mxnet/keys{/key_id}","collaborators_url":"https://api.github.com/repos/reminisce/mxnet/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/reminisce/mxnet/teams","hooks_url":"https://api.github.com/repos/reminisce/mxnet/hooks","issue_events_url":"https://api.github.com/repos/reminisce/mxnet/issues/events{/number}","events_url":"https://api.github.com/repos/reminisce/mxnet/events","assignees_url":"https://api.github.com/repos/reminisce/mxnet/assignees{/user}","branches_url":"https://api.github.com/repos/reminisce/mxnet/branches{/branch}","tags_url":"https://api.github.com/repos/reminisce/mxnet/tags","blobs_url":"https://api.github.com/repos/reminisce/mxnet/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/reminisce/mxnet/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/reminisce/mxnet/git/refs{/sha}","trees_url":"https://api.github.com/repos/reminisce/mxnet/git/trees{/sha}","statuses_url":"https://api.github.com/repos/reminisce/mxnet/statuses/{sha}","languages_url":"https://api.github.com/repos/reminisce/mxnet/languages","stargazers_url":"https://api.github.com/repos/reminisce/mxnet/stargazers","contributors_url":"https://api.github.com/repos/reminisce/mxnet/contributors","subscribers_url":"https://api.github.com/repos/reminisce/mxnet/subscribers","subscription_url":"https://api.github.com/repos/reminisce/mxnet/subscription","commits_url":"https://api.github.com/repos/reminisce/mxnet/commits{/sha}","git_commits_url":"https://api.github.com/repos/reminisce/mxnet/git/commits{/sha}","comments_url":"https://api.github.com/repos/reminisce/mxnet/comments{/number}","issue_comment_url":"https://api.github.com/repos/reminisce/mxnet/issues/comments{/number}","contents_url":"https://api.github.com/repos/reminisce/mxnet/contents/{+path}","compare_url":"https://api.github.com/repos/reminisce/mxnet/compare/{base}...{head}","merges_url":"https://api.github.com/repos/reminisce/mxnet/merges","archive_url":"https://api.github.com/repos/reminisce/mxnet/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/reminisce/mxnet/downloads","issues_url":"https://api.github.com/repos/reminisce/mxnet/issues{/number}","pulls_url":"https://api.github.com/repos/reminisce/mxnet/pulls{/number}","milestones_url":"https://api.github.com/repos/reminisce/mxnet/milestones{/number}","notifications_url":"https://api.github.com/repos/reminisce/mxnet/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/reminisce/mxnet/labels{/name}","releases_url":"https://api.github.com/repos/reminisce/mxnet/releases{/id}","deployments_url":"https://api.github.com/repos/reminisce/mxnet/deployments","created_at":"2017-01-19T00:07:24Z","updated_at":"2018-02-08T01:53:43Z","pushed_at":"2018-03-03T05:28:02Z","git_url":"git://github.com/reminisce/mxnet.git","ssh_url":"git@github.com:reminisce/mxnet.git","clone_url":"https://github.com/reminisce/mxnet.git","svn_url":"https://github.com/reminisce/mxnet","homepage":"http://mxnet.io","size":36155,"stargazers_count":1,"watchers_count":1,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":2,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":1,"open_issues":2,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"40da6ccd2775ebe08683518c866bcb2fec467347","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":34864402,"name":"incubator-mxnet","full_name":"apache/incubator-mxnet","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/incubator-mxnet","description":"Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more","fork":false,"url":"https://api.github.com/repos/apache/incubator-mxnet","forks_url":"https://api.github.com/repos/apache/incubator-mxnet/forks","keys_url":"https://api.github.com/repos/apache/incubator-mxnet/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/incubator-mxnet/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/incubator-mxnet/teams","hooks_url":"https://api.github.com/repos/apache/incubator-mxnet/hooks","issue_events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/events{/number}","events_url":"https://api.github.com/repos/apache/incubator-mxnet/events","assignees_url":"https://api.github.com/repos/apache/incubator-mxnet/assignees{/user}","branches_url":"https://api.github.com/repos/apache/incubator-mxnet/branches{/branch}","tags_url":"https://api.github.com/repos/apache/incubator-mxnet/tags","blobs_url":"https://api.github.com/repos/apache/incubator-mxnet/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/incubator-mxnet/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/incubator-mxnet/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/incubator-mxnet/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/incubator-mxnet/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/incubator-mxnet/languages","stargazers_url":"https://api.github.com/repos/apache/incubator-mxnet/stargazers","contributors_url":"https://api.github.com/repos/apache/incubator-mxnet/contributors","subscribers_url":"https://api.github.com/repos/apache/incubator-mxnet/subscribers","subscription_url":"https://api.github.com/repos/apache/incubator-mxnet/subscription","commits_url":"https://api.github.com/repos/apache/incubator-mxnet/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/incubator-mxnet/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/incubator-mxnet/contents/{+path}","compare_url":"https://api.github.com/repos/apache/incubator-mxnet/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/incubator-mxnet/merges","archive_url":"https://api.github.com/repos/apache/incubator-mxnet/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/incubator-mxnet/downloads","issues_url":"https://api.github.com/repos/apache/incubator-mxnet/issues{/number}","pulls_url":"https://api.github.com/repos/apache/incubator-mxnet/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/incubator-mxnet/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/incubator-mxnet/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/labels{/name}","releases_url":"https://api.github.com/repos/apache/incubator-mxnet/releases{/id}","deployments_url":"https://api.github.com/repos/apache/incubator-mxnet/deployments","created_at":"2015-04-30T16:21:15Z","updated_at":"2018-03-04T20:04:25Z","pushed_at":"2018-03-04T20:16:52Z","git_url":"git://github.com/apache/incubator-mxnet.git","ssh_url":"git@github.com:apache/incubator-mxnet.git","clone_url":"https://github.com/apache/incubator-mxnet.git","svn_url":"https://github.com/apache/incubator-mxnet","homepage":"http://mxnet.io","size":33619,"stargazers_count":13243,"watchers_count":13243,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":4867,"mirror_url":null,"archived":false,"open_issues_count":815,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":4867,"open_issues":815,"watchers":13243,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552"},"html":{"href":"https://github.com/apache/incubator-mxnet/pull/9552"},"issue":{"href":"https://api.github.com/repos/apache/incubator-mxnet/issues/9552"},"comments":{"href":"https://api.github.com/repos/apache/incubator-mxnet/issues/9552/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9552/commits"},"statuses":{"href":"https://api.github.com/repos/apache/incubator-mxnet/statuses/662570bf1257616568caf828ad1513de8cd88d7b"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"KellenSunderland"}}