{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/616","repository_url":"https://api.github.com/repos/NVIDIA/DIGITS","labels_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/616/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/616/comments","events_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/616/events","html_url":"https://github.com/NVIDIA/DIGITS/pull/616","id":138184284,"number":616,"title":"Progress notifications in infer_many [ DON'T MERGE]","user":{"login":"gheinrich","id":3889770,"avatar_url":"https://avatars.githubusercontent.com/u/3889770?v=3","gravatar_id":"","url":"https://api.github.com/users/gheinrich","html_url":"https://github.com/gheinrich","followers_url":"https://api.github.com/users/gheinrich/followers","following_url":"https://api.github.com/users/gheinrich/following{/other_user}","gists_url":"https://api.github.com/users/gheinrich/gists{/gist_id}","starred_url":"https://api.github.com/users/gheinrich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gheinrich/subscriptions","organizations_url":"https://api.github.com/users/gheinrich/orgs","repos_url":"https://api.github.com/users/gheinrich/repos","events_url":"https://api.github.com/users/gheinrich/events{/privacy}","received_events_url":"https://api.github.com/users/gheinrich/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NVIDIA/DIGITS/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/NVIDIA/DIGITS/labels/UI","name":"UI","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-03-03T13:58:21Z","updated_at":"2016-03-11T18:10:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/NVIDIA/DIGITS/pulls/616","html_url":"https://github.com/NVIDIA/DIGITS/pull/616","diff_url":"https://github.com/NVIDIA/DIGITS/pull/616.diff","patch_url":"https://github.com/NVIDIA/DIGITS/pull/616.patch"},"body":"*Close #611*\r\n\r\n### Summary ###\r\n\r\nThis change causes the inference job page to be displayed immediately while the job is running. Progress notifications are reflected on the job status bar to show the estimated time remaining. Results are not showed continuously however when the inference job completes the page is being reloaded to display all classifications and statistics.\r\n\r\n### Implementation details ###\r\n\r\nThe inference tool was updated to perform inference in batches of (by default) 1024. This allows progress messages to be sent back to the web app.\r\n\r\n~~The `classify_many` route now serves two purposes:~~\r\n~~- kick off a new inference job, when `job_id` is an instance of a classification model. In that case the `image_list` field is required;~~\r\n~~- check the current status of the inference job, when `job_id` is an instance of an inference job.~~\r\n\r\n~~In both cases the route returns a page that shows the current status of the inference job. If the job is done this route shows the classification results and statistics. If this sounds too convoluted I can add another route.~~\r\n\r\nOn second thought it feels like a separate route is cleaner. I moved the `classify_many` route from the `model` blueprint to `digits.inference.images.classification.views.classify_many`. When that route is being accessed required parameters are `job_id` (the model job ID) and `image_list`. The client is then redirected to `digits.inference.views.show`. This new route shows the current status of the inference process. When the route is being accessed before the job is done, progress notifications are shown in the job status bar. When the job is done a socketio message is sent to the client to reload the page (and show all results). This sounds slightly racey (the \"job done\" update might be sent before the client establishes the socket io connection) therefore a javascript timer was added to automatically refresh the page on client side if no notification is received within the first 10 seconds.\r\n\r\n> NOTE: the inference job is automatically removed when the `show` route is being accessed after the job is done but what do we need to do about \"unclaimed\" jobs?\r\n\r\nThe REST interface is updated in the same way. The only difference being that there is no way to send updates to the client through socketio. To address this issue, the inference job ID is returned in the initial JSON response. The user can then access the new `digits.inference/<job>.json` route to check the job status and progress. For example, the initial request might look like this (`20160117-142505-b706` is the model job id):\r\n```\r\n$ curl --form \"image_list=@/tmp/val.txt\" http://localhost:5000//inference/images/classification/classify_many.json?job_id=20160117-142505-b706\r\n{\r\n  \"id\": \"20160304-214034-d677\", \r\n  \"name\": \"Classify Many Images\", \r\n  \"progress\": 0.0, \r\n  \"status\": \"Initialized\"\r\n```\r\n\r\nThe client can now check the current status of the inference job:\r\n```\r\n$ curl http://localhost:5000/inference/20160304-214034-d677.json\r\n{\r\n  \"id\": \"20160304-214034-d677\", \r\n  \"name\": \"Classify Many Images\", \r\n  \"progress\": 0.6827577010268036, \r\n  \"status\": \"Running\"\r\n}\r\n```\r\n\r\nThe `progress ` field can be used by the client to poll \"intelligently\" i.e. according to the speed at which progress is being made.\r\n\r\nFinally, when the job is done, classification results are shown:\r\n```\r\n$ curl http://localhost:5000/inference/20160304-214034-d677.json\r\n{\r\n  \"classifications\": {\r\n    \"/home/greg/ws/datasets/mnist/train/0/00037.png\": [\r\n      [\r\n        \"0\", \r\n         100.0\r\n       ], \r\n...\r\n```\r\n\r\n### Progress ###\r\n\r\n- [x] batched inference in inference tool\r\n- [x] send progress notifications to server\r\n- [x] show job progress immediately\r\n- [x] reload page when inference job is done\r\n- [x] timer to detect socketio race between client and server\r\n- [x] update REST interface\r\n- [x] classify One\r\n- [x] classify Many\r\n- [x] top-N classification\r\n- [ ] generic inference\r\n- [x] update classification tests\r\n- [ ] update generic tests\r\n- [ ] find out what to do about unclaimed jobs\r\n\r\n### Future ###\r\n\r\n- keep model resident (avoid reloading model on every batch)\r\n- adaptive batch size (e.g. to aim for one update notification every x seconds instead of every x samples)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/comments/195482099","html_url":"https://github.com/NVIDIA/DIGITS/pull/616#issuecomment-195482099","issue_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/616","id":195482099,"user":{"login":"gheinrich","id":3889770,"avatar_url":"https://avatars.githubusercontent.com/u/3889770?v=3","gravatar_id":"","url":"https://api.github.com/users/gheinrich","html_url":"https://github.com/gheinrich","followers_url":"https://api.github.com/users/gheinrich/followers","following_url":"https://api.github.com/users/gheinrich/following{/other_user}","gists_url":"https://api.github.com/users/gheinrich/gists{/gist_id}","starred_url":"https://api.github.com/users/gheinrich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gheinrich/subscriptions","organizations_url":"https://api.github.com/users/gheinrich/orgs","repos_url":"https://api.github.com/users/gheinrich/repos","events_url":"https://api.github.com/users/gheinrich/events{/privacy}","received_events_url":"https://api.github.com/users/gheinrich/received_events","type":"User","site_admin":false},"created_at":"2016-03-11T18:10:19Z","updated_at":"2016-03-11T18:10:19Z","body":"> How much memory might these jobs potentially take up on the server? Just the network outputs? Maybe a 1hr timeout instead?\r\n\r\nYes, network outputs plus resized images and visualizations. The resized image is used during single image inference to show what the actual network output looks like and also during Top-N classification. Admittedly, we don't need it for multiple image classification, though I figured it is preferable to have a common set of data and let the `views` choose how to display results.\r\n\r\n> I would expect the server to save the first num_test_images of the input file to disk, and then let tools/inference.py read it in batches of 64 or 24 or whatever the network was designed for. What are you doing instead?\r\n\r\nThe script in `tools/inference.py` is using the framework's `infer_many()` method to perform inference. `infer_many()` can take any number of input samples however it needs these samples to be resized to the network input dimensions. Before this PR, this works as follows:\r\n- read and resized *all* images from the list up to `num_test_images`\r\n- provide the full set of resized images to `infer_many()` and have the framework do the inference.\r\n\r\nAs I mentioned in the previous comment there are a couple of issues associated with this scheme so I introduced the notion of a batch in the `tools/inference.py` script: instead of loading/resizing all images, I am loading batches of (by default 1024) images and I pass those batches to `infer_many()`. So indeed this isn't the same thing as the batch at GPU level since `infer_many()` further splits its input data into smaller network dependent batches. Now the question is: what batch size to use in `tools/inference.py`? If I use too small a batch size there will be a large overhead in `infer_many()` since it reloads the model every time. In the TODO list there is an item for keeping the model resident in memory to reduce the overhead. This sounds easy enough to implement in Caffe but it will be less obvious in Torch since inference isn't done in Python."}}