How would I download files (video) with Python using wget and save them locally? There will be a bunch of files, so how do I know that one file is downloaded so as to automatically start downloding another one?



Short answer (simplified). To get one file

 import urllib
 urllib.urlretrieve("http://google.com/index.html", filename="local/index.html")

You can figure out how to loop that if necessary.

| improve this answer | |

Don't do this. Use either urllib2 or urlgrabber instead.

| improve this answer | |
  • 12
    This answer needs to be expanded. Why shouldn't wget be used? – muhuk Mar 18 '10 at 10:26
  • 11
    Because it starts a whole new process just to do things that Python itself is capable of. – Ignacio Vazquez-Abrams Mar 18 '10 at 10:36
  • 6
    Because it undermines portability. – Ekevoo Apr 14 '11 at 0:51
  • 7
    Is it nontrivial to write wget -rl1 -I /stuff/i/want/ http://url/<incrementing number> with either of these libs? – Seppo Erviälä Jun 27 '12 at 8:59
  • 5
    wget works through a VPN client, whereas urllib gives me this error for a https: urlopen error Tunnel connection failed: 407 Proxy Authentication Required – tommy.carstensen Feb 8 '15 at 18:36

If you use os.system() to spawn a process for the wget, it will block until wget finishes the download (or quits with an error). So, just call os.system('wget blah') in a loop until you've downloaded all of your files.

Alternatively, you can use urllib2 or httplib. You'll have to write a non-trivial amount code, but you'll get better performance, since you can reuse a single HTTP connection to download many files, as opposed to opening a new connection for each file.

| improve this answer | |

No reason to use os.system. Avoid writing a shell script in Python and go with something like urllib.urlretrieve or an equivalent.

Edit... to answer the second part of your question, you can set up a thread pool using the standard library Queue class. Since you're doing a lot of downloading, the GIL shouldn't be a problem. Generate a list of the URLs you wish to download and feed them to your work queue. It will handle pushing requests to worker threads.

I'm waiting for a database update to complete, so I put this together real quick.


import sys
import threading
import urllib
from Queue import Queue
import logging

class Downloader(threading.Thread):
    def __init__(self, queue):
        super(Downloader, self).__init__()
        self.queue = queue

    def run(self):
        while True:
            download_url, save_as = queue.get()
            # sentinal
            if not download_url:
                urllib.urlretrieve(download_url, filename=save_as)
            except Exception, e:
                logging.warn("error downloading %s: %s" % (download_url, e))

if __name__ == '__main__':
    queue = Queue()
    threads = []
    for i in xrange(5):

    for line in sys.stdin:
        url = line.strip()
        filename = url.split('/')[-1]
        print "Download %s as %s" % (url, filename)
        queue.put((url, filename))

    # if we get here, stdin has gotten the ^D
    print "Finishing current downloads"
    for i in xrange(5):
        queue.put((None, None))

| improve this answer | |
  • 1
    there is a mistake in download_url, save_as = queue.get(). should be download_url, save_as = self.queue.get(). – disfated Nov 26 '11 at 2:08

Install wget via pypi http://pypi.python.org/pypi/wget/0.3

pip install wget

then run, just as documented

python -m wget <url>
| improve this answer | |
  • 14
    For anyone else who found this confusing, the linked library doesn't use wget. It uses urllib. And it currently doesn't support anything close to what wget ( gnu.org/software/wget ) does. – Rob Russell Dec 31 '13 at 16:58

No reason to use python. Avoid writing a shell script in Python and go with something like bash or an equivalent.

| improve this answer | |
  • 2
    Writing a shell script in Python is OK. If you want to get something done quickly but you hate the syntax of bash, just do it in Python. If you make a larger project, then yes, try to avoid these external calls. – Jabba Apr 11 '12 at 7:10
  • 3
    Python is a fine scripting language. – Mark Lakata Nov 15 '12 at 0:00
