user618: I am looking for getting all games played in 2013 as quickly and painlessly as possible. A batch file to download many ziped pgn files would be non-ideal, but ok. I prefer the Week in Chess as a source due to their high DB consistency quality, but the ultimate goal is to compile a single high quality DB of all games played in the year 2013.

user2001: Write a script that downloads the files, one at the time. The filename structure will allow you to use a loop to capture all the files you wish to get. Then, write a script that unzips the files and concatenates them in a single large PGN file. Then, write a script that finds and discards duplicated games. Other strategies seem to include a Firefox add-on (https://addons.mozilla.org/en-US/firefox/addon/downthemall/) and in general, the topic has been discussed in other places (http://www.chess.com/forum/view/general/13-million-master-games), (http://www.chess.com/forum/view/chess-equipment/quotlost-issuesquot-of-twic), (http://chessdb.sourceforge.net/tutorial/twic-retriveal.php).

user7500: An easy solution in C++:
    #include <urlmon.h>

    #pragma comment(lib, "urlmon.lib")

    const int firstnumber = 920, lastnumber = 1104;
    const char * const Spath = "http://www.theweekinchess.com/zips/twic"; // source path
    const char * const Dpath = "C:\\Users\\<username>\\Downloads\\twic\\"; // destination path

    void main()
    {
        int x;

        for (x = firstnumber; x <= lastnumber; ++x)
            URLDownloadToFile(nullptr, (Spath + std::to_string(x) \
    + "g.zip").c_str(), (Dpath + std::to_string(x) + ".zip").c_str(), 0, nullptr);


    }

A powershell unzip script:
$shell = new-object -com shell.application

for ($x = 920; $x -le 1103; $x+=1) {

    $zip = $shell.NameSpace(“C:\Users\<username>\Downloads\twic\$x.zip”)

    foreach($item in $zip.items())
    {
        $shell.Namespace(“C:\Users\<username>\Downloads\twic”).copyhere($item)
    }

}

A simple command line pgn file merge: copy *.pgn twic.pgn

user17656: I was trying to solve the same problem but since I am not familiar with C++ I did it in Python instead. My program also unzips and extracts the PGN files from the downloaded zip files.
Code:
  import pycurl
  from io import BytesIO
  import os
  import zipfile
  from pathlib import Path

  c = pycurl.Curl()

  for i in range(start, end + 1):

    url = "http://theweekinchess.com/zips/twic" + str(i) + "g.zip"

    save_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic/twic" + str(i) + ".zip")
    unzipped_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic")

    with open(save_location_path, 'wb') as f:

        c.setopt(c.URL, url)
        c.setopt(c.WRITEDATA, f)
        c.perform()

    with zipfile.ZipFile(save_location_path, 'r') as myzip:
        myzip.extractall(unzipped_location_path)

    os.remove(save_location_path)

  c.close()

Notes:

Firstly to find the url I just put my mouse pointer on the download pgn button on the TWIC website. I noticed that the urls have a common structure and the only thing that changes are the numbers. However, that url appeared to be incorrect as I was getting a 301 (moved permanently) error. As it turned out it moved to the same url but without the 'www' at the start.
Adjust the range function depending on what files you want to download, so if you want to download from 1100 to 1250 then set start = 1100 and end = 1250
The unzipped_location_path should be the path to the folder where you want the pgn files to be saved to. The save_location_path will be a path to where the zip files will be downloaded to first (I chose the same folder for this). These also need a file name at the end of the path which needs to be different for each zip. Hence the str(i) + ".zip" ending.
It is not necessary to use Path() - it's use is that this script can now be run on another operating system which perhaps has the same path (Windows for instance uses backslashes instead and this path would not work)
The pycurl library downloads the files. Zipfile then reads these files and extracts them. Finally os.remove(save_location_path) removes the no longer useful zip files.
Finally, I originally downloaded CBV Chessbase files instead. However, I could not see an easy way to combine all those files together and I would have had to append to my database one file at a time. That is why I downloaded PGN instead as now I could use the cat command ($ cat * >twic1100-1250.pgn) to merge all the files into one.

Automating the process

Finally, I thought to myself: would it not be nice if the script was executed weekly and my database would be up to date from now on? To see how I adapted my code have a look on my GitHub: https://github.com/paluchasz/downloading_games_from_TWIC/blob/master/downloading_twic_games.py
I then set up a cron job in the command line to execute the script every Tuesday.


user1967: Here is your requested zip:  2013_TWIC.zip
I created a command script for this. See attached. I assumed you are running
Windows, but the script is easily adapted to other OSes. 
For downloading, it uses curl. You can get that at http://curl.haxx.se/download.html, and
for Windows, I recommend the msi installer.
For unzipping, I used Java's Jar tool, which is free. But use whatever you
prefer. 
When running the script, run dos shell as Administrator in Win8 or Win7. 
At the end of the script it puts all PGNs into one large PGN file called 2013_TWIC.pgn
HTH.
Click: Script to get 2013 TWIC PGN files 

618: <p>I am looking for getting all games played in 2013 as quickly and painlessly as possible. A batch file to download many ziped pgn files would be non-ideal, but ok. I prefer the Week in Chess as a source due to their high DB consistency quality, but the ultimate goal is to compile a single high quality DB of all games played in the year 2013.</p>
 2001: <p>Write a script that downloads the files, one at the time. The filename structure will allow you to use a loop to capture all the files you wish to get. Then, write a script that unzips the files and concatenates them in a single large PGN file. Then, write a script that finds and discards duplicated games. Other strategies seem to include a Firefox add-on (<a href="https://addons.mozilla.org/en-US/firefox/addon/downthemall/" rel="nofollow">https://addons.mozilla.org/en-US/firefox/addon/downthemall/</a>) and in general, the topic has been discussed in other places (<a href="http://www.chess.com/forum/view/general/13-million-master-games" rel="nofollow">http://www.chess.com/forum/view/general/13-million-master-games</a>), (<a href="http://www.chess.com/forum/view/chess-equipment/quotlost-issuesquot-of-twic" rel="nofollow">http://www.chess.com/forum/view/chess-equipment/quotlost-issuesquot-of-twic</a>), (<a href="http://chessdb.sourceforge.net/tutorial/twic-retriveal.php" rel="nofollow">http://chessdb.sourceforge.net/tutorial/twic-retriveal.php</a>).</p>
 7500: <p>An easy solution in C++:</p>

<pre><code>    #include &lt;urlmon.h&gt;

    #pragma comment(lib, "urlmon.lib")

    const int firstnumber = 920, lastnumber = 1104;
    const char * const Spath = "http://www.theweekinchess.com/zips/twic"; // source path
    const char * const Dpath = "C:\\Users\\&lt;username&gt;\\Downloads\\twic\\"; // destination path

    void main()
    {
        int x;

        for (x = firstnumber; x &lt;= lastnumber; ++x)
            URLDownloadToFile(nullptr, (Spath + std::to_string(x) \
    + "g.zip").c_str(), (Dpath + std::to_string(x) + ".zip").c_str(), 0, nullptr);


    }
</code></pre>

<p>A powershell unzip script:</p>

<pre><code>$shell = new-object -com shell.application

for ($x = 920; $x -le 1103; $x+=1) {

    $zip = $shell.NameSpace(“C:\Users\&lt;username&gt;\Downloads\twic\$x.zip”)

    foreach($item in $zip.items())
    {
        $shell.Namespace(“C:\Users\&lt;username&gt;\Downloads\twic”).copyhere($item)
    }

}
</code></pre>

<p>A simple command line pgn file merge: <code>copy *.pgn twic.pgn</code></p>
 17656: <p>I was trying to solve the same problem but since I am not familiar with C++ I did it in Python instead. My program also unzips and extracts the PGN files from the downloaded zip files.</p>

<h2>Code:</h2>

<pre><code>  import pycurl
  from io import BytesIO
  import os
  import zipfile
  from pathlib import Path

  c = pycurl.Curl()

  for i in range(start, end + 1):

    url = "http://theweekinchess.com/zips/twic" + str(i) + "g.zip"

    save_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic/twic" + str(i) + ".zip")
    unzipped_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic")

    with open(save_location_path, 'wb') as f:

        c.setopt(c.URL, url)
        c.setopt(c.WRITEDATA, f)
        c.perform()

    with zipfile.ZipFile(save_location_path, 'r') as myzip:
        myzip.extractall(unzipped_location_path)

    os.remove(save_location_path)

  c.close()
</code></pre>

<h2>Notes:</h2>

<ul>
<li>Firstly to find the url I just put my mouse pointer on the download pgn button on the TWIC website. I noticed that the urls have a common structure and the only thing that changes are the numbers. However, that url appeared to be incorrect as I was getting a 301 (moved permanently) error. As it turned out it moved to the same url but without the 'www' at the start.</li>
<li>Adjust the range function depending on what files you want to download, so if you want to download from 1100 to 1250 then set start = 1100 and end = 1250</li>
<li>The unzipped_location_path should be the path to the folder where you want the pgn files to be saved to. The save_location_path will be a path to where the zip files will be downloaded to first (I chose the same folder for this). These also need a file name at the end of the path which needs to be different for each zip. Hence the <em>str(i) + ".zip"</em> ending.</li>
<li>It is not necessary to use <em>Path()</em> - it's use is that this script can now be run on another operating system which perhaps has the same path (Windows for instance uses backslashes instead and this path would not work)</li>
<li>The pycurl library downloads the files. Zipfile then reads these files and extracts them. Finally <em>os.remove(save_location_path)</em> removes the no longer useful zip files.</li>
<li>Finally, I originally downloaded CBV Chessbase files instead. However, I could not see an easy way to combine all those files together and I would have had to append to my database one file at a time. That is why I downloaded PGN instead as now I could use the cat command ($ cat * >twic1100-1250.pgn) to merge all the files into one.</li>
</ul>

<h2>Automating the process</h2>

<ul>
<li>Finally, I thought to myself: would it not be nice if the script was executed weekly and my database would be up to date from now on? To see how I adapted my code have a look on my GitHub: <a href="https://github.com/paluchasz/downloading_games_from_TWIC/blob/master/downloading_twic_games.py" rel="nofollow noreferrer">https://github.com/paluchasz/downloading_games_from_TWIC/blob/master/downloading_twic_games.py</a></li>
<li>I then set up a cron job in the command line to execute the script every Tuesday.</li>
</ul>
 1967: <p>Here is your requested zip:  <a href="http://nlightening.com/test/2013_TWIC.zip" rel="noreferrer">2013_TWIC.zip</a></p>

<p>I created a command script for this. See attached. I assumed you are running
Windows, but the script is easily adapted to other OSes. </p>

<p>For downloading, it uses curl. You can get that at <a href="http://curl.haxx.se/download.html" rel="noreferrer">http://curl.haxx.se/download.html</a>, and
for Windows, I recommend the msi installer.</p>

<p>For unzipping, I used Java's Jar tool, which is free. But use whatever you
prefer. </p>

<p>When running the script, run dos shell as Administrator in Win8 or Win7. 
At the end of the script it puts all PGNs into one large PGN file called 2013_TWIC.pgn
HTH.</p>

<p>Click: <a href="http://nlightening.com/test/GetTwic2013Pgn.cmd" rel="noreferrer">Script to get 2013 TWIC PGN files</a> </p>
