Take the 2-minute tour ×

@slhck provided a useful answer to this similar question.

Downloading multiple files with wget and handling parameters

This code was provided to read urls from a text file filelist, then run a loop to download each.

    #!/usr/bin/env bash 
    while read line 
        wget -c --load-cookies cookies.txt $line -O ${line##*/} 
    done < filelist


As I'm not familiar with shell scripts, I am wondering if you could download multiple xml files by:

  • creating a simple comma seperated text file filelist
  • read from:


renamedfile.xml, url-01.php
renamedFileDifferent.xml, url-02.php
specificFileRename.xml, "url-03"
newFilename.xml, "url-04"

  • read through each line
  • split the line into newfile, remoteFile
  • And run:  

    wget -O newfile remoteFile >/dev/null 2>&1 


Is it possible to write this in a shell script? Can you provide an example?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

To get the text from before the comma you use ${line/,*}.

(What this actually does is replace ",*" or all the text from the comma onwards with empty string - leaving only the part of the string before the comma)

To get the text from after the comma you use ${line/*,}.

So the full command would be:

while read line
    wget -O ${line/,*} ${line/*,} >/dev/null 2>&1 
done < filelist

or, on one line:

while read line; do echo wget -O ${line/,*} ${line/*,} >/dev/null 2>&1; done < filelist

In Windows (assuming you've installed wget, from http://gnuwin32.sourceforge.net/packages/wget.htm, and set your path correctly), it would be:

for /f "delims=, tokens=1*" %a in (filelist) do wget -O %a %b
share|improve this answer
What directory is a good place to put the text file, and shell file typically? This will be run as a cron job. –  rrrfusco Sep 25 '12 at 0:59
Wherever you want to run it from (and save the files to). eg. if you put filelist and test.sh in sub directory of your home, and then run it from there, that's where it will save the files. –  Luke Sep 25 '12 at 1:03
I'm not sure it will run as expected if run as a cron job. You will want to specify absolute paths in the filelist (~\downloaddir\renamedfile.xml) and probably in the crontab as well (46 8 * * cd ~\downloaddir; test.sh or maybe just 46 8 * * ~\downloaddir\test.sh). Refer stackoverflow.com/questions/13204/… and unix.com/shell-programming-scripting/… –  Luke Sep 25 '12 at 11:09
another option is to harcode the paths into the script, eg. wget -O ~/downloaddir/${line/,*} ${line/*,} >/dev/null 2>&1 and done < ~/downloaddir/filelist –  Luke Sep 25 '12 at 11:12
add comment

Your Answer


