{"action":"created","issue":{"url":"https://api.github.com/repos/fastai/fastai/issues/1382","repository_url":"https://api.github.com/repos/fastai/fastai","labels_url":"https://api.github.com/repos/fastai/fastai/issues/1382/labels{/name}","comments_url":"https://api.github.com/repos/fastai/fastai/issues/1382/comments","events_url":"https://api.github.com/repos/fastai/fastai/issues/1382/events","html_url":"https://github.com/fastai/fastai/pull/1382","id":393723949,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQwNjY2MDE1","number":1382,"title":"ImageDownloader widget","user":{"login":"xnutsive","id":43633,"node_id":"MDQ6VXNlcjQzNjMz","avatar_url":"https://avatars1.githubusercontent.com/u/43633?v=4","gravatar_id":"","url":"https://api.github.com/users/xnutsive","html_url":"https://github.com/xnutsive","followers_url":"https://api.github.com/users/xnutsive/followers","following_url":"https://api.github.com/users/xnutsive/following{/other_user}","gists_url":"https://api.github.com/users/xnutsive/gists{/gist_id}","starred_url":"https://api.github.com/users/xnutsive/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xnutsive/subscriptions","organizations_url":"https://api.github.com/users/xnutsive/orgs","repos_url":"https://api.github.com/users/xnutsive/repos","events_url":"https://api.github.com/users/xnutsive/events{/privacy}","received_events_url":"https://api.github.com/users/xnutsive/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-12-23T05:01:50Z","updated_at":"2018-12-31T16:20:10Z","closed_at":"2018-12-30T10:29:27Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/fastai/fastai/pulls/1382","html_url":"https://github.com/fastai/fastai/pull/1382","diff_url":"https://github.com/fastai/fastai/pull/1382.diff","patch_url":"https://github.com/fastai/fastai/pull/1382.patch"},"body":"Proposing the `ImageDownloader` widget. Similarly to `ImageCleaner`, it's an ipywidget that works in Notebook or Lab, allows users to quickly research and download images from google search. \r\n\r\n[Here's the thread on fastai dev forums](https://forums.fast.ai/t/imagedownloader-widget/32776). \r\n\r\n**Why:** \r\n\r\nBuilding image classifiers in the first weeks of the MOOC is twice as fun with your own dataset, right? I noticed fastai v1 ships with a JS snippet to download images and that's mentioned in the new version of the MOOC's notebooks. \r\n\r\n`ImageDownloader` would allow people experiment with new datasets easily, we'll have more people implementing their own classifiers and apps based on them → more fun, and more inspiration for them and other students to learn and implement their ideas. \r\n\r\n**What works:**\r\n\r\n`ImageDownloader()` works in Jupyter Notebook and Lab: \r\n\r\n- [x] User types in the search term / query, number of images, and image resolution they want. \r\n- [x] Downloads images from google, stores them in a directory. The directory is created automatically if it doesn't exist. \r\n- [x] Previews a few images from the download. \r\n- [x] Allows downloading >100 images per label, but requires `selenium` and `chromedriver` to do that. Shows an error message explaining how to install those if the user requests more than a hundred images. \r\n- [x] Uses fastai's `download_url()` and `parallel()` to speed things up ;-) \r\n- [x] Integrates with `fastprogress` and shows progress bar when downloading. \r\n- [x] Calls `verify_images()` after the download to clean up the dataset. \r\n- [x] Directory structure compatible with datablock API, i.e. `from_folder()` “just works”.\r\n- [x] Also provides a function to download images in iPython / python scripts: `download_google_images()`, although it's in `fastai.widgets.image_downloader`. It maybe useful to move it to `fastai.visual.data`, but I wanted to start small and show @sgugger the diff first. \r\n\r\n**Also taken care of:** \r\n\r\n- [x] Docs: I created a section in `docs_src/widgets.ipynb`, documented both the widget, and the function, with examples, using `show_doc()`, and regenerated the html docs with `tools/build-docs`\r\n- [x] Added a test to make sure the widget init runs without errors. \r\n\r\n**Ideas to improve this:**\r\n\r\n- [ ] I'm working on a full example: fetch a dataset, train, save the model, and serve it as a REST API and a website, just to give other students some inspiration. Probably not for `fastai/examples`, I can post that outside of fastai if that'd be more appropriate.\r\n- [ ] It'd be good to add some tests to test the widget internal logic, it's decoupled into several helper functions, but they're not unit tested.\r\n- [ ] Since it's questionable whereas it's legal to scrape google search like that, maybe this can't be merged at all, or maybe it's a good idea to add a legal disclaimer to the docs? \r\n\r\n@fpingham, @sgugger, I'm not really a Python developer, and I'd be happy to improve the code, fix any smells, code styles, anything that comes up. I'd be happy to help in other projects as well, just wanted to start with something simple.\r\n\r\nThank you both very much for your work on fastai, for your comments on the forum thread, and for reviewing the PR. \r\n\r\nWhat do you think, @sgugger? \r\n"},"comment":{"url":"https://api.github.com/repos/fastai/fastai/issues/comments/450664336","html_url":"https://github.com/fastai/fastai/pull/1382#issuecomment-450664336","issue_url":"https://api.github.com/repos/fastai/fastai/issues/1382","id":450664336,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDY2NDMzNg==","user":{"login":"xnutsive","id":43633,"node_id":"MDQ6VXNlcjQzNjMz","avatar_url":"https://avatars1.githubusercontent.com/u/43633?v=4","gravatar_id":"","url":"https://api.github.com/users/xnutsive","html_url":"https://github.com/xnutsive","followers_url":"https://api.github.com/users/xnutsive/followers","following_url":"https://api.github.com/users/xnutsive/following{/other_user}","gists_url":"https://api.github.com/users/xnutsive/gists{/gist_id}","starred_url":"https://api.github.com/users/xnutsive/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xnutsive/subscriptions","organizations_url":"https://api.github.com/users/xnutsive/orgs","repos_url":"https://api.github.com/users/xnutsive/repos","events_url":"https://api.github.com/users/xnutsive/events{/privacy}","received_events_url":"https://api.github.com/users/xnutsive/received_events","type":"User","site_admin":false},"created_at":"2018-12-31T16:20:10Z","updated_at":"2018-12-31T16:20:10Z","author_association":"CONTRIBUTOR","body":"Sure, will do.\n\nOn Mon, Dec 31 2018 at 2:44 AM, < notifications@github.com > wrote:\n\n> \n> \n> \n> One last problem @xnutsive ( https://github.com/xnutsive ) : the doc\n> notebook widget.image_cleaner can't be executed automatically now, which\n> breaks our expanded test suite (all the doc notebooks are run when we\n> launch the tests before a release). I've fixed it by moving some cells in\n> markdown code, but you may want to have a second look.\n> \n> \n> \n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub (\n> https://github.com/fastai/fastai/pull/1382#issuecomment-450631557 ) , or mute\n> the thread (\n> https://github.com/notifications/unsubscribe-auth/AACqcf0kUv3kZzepMyRbabm6cG8XOA8oks5u-ep4gaJpZM4Zftpg\n> ).\n> \n> \n>"}}
{"actor":{"display_login":"xnutsive"}}