Downloading webpages programmatically <p>I need to download certain web pages from a web site for my research. I tried to download them using Darcy Wripper, WinHTTrack and C# WebClient method. It happens so that some of the pages get downloaded while (majority) others throw error. For example I have copied this error from Darcy Wripper. </p> <blockquote> <p>27 <a href="http://www.dawn.com/news/515958/animadversion-only-the-globes-looked-golden" rel="nofollow">http://www.dawn.com/news/515958/animadversion-only-the-globes-looked-golden</a> Error N/A N/A 500 0 403 - Request Aborted 11 <a href="http://www.dawn.com/news/813115/flavours-of-culture" rel="nofollow">http://www.dawn.com/news/813115/flavours-of-culture</a> Error N/A N/A 500 0 403</p> </blockquote> <p>The same webpage opens in Firefox (sometimes there is an error that we are in maintenance, but next try opens the page). My question: Is there a method in C# to download such problematic pages? I also tried to look for a Firefox Wrapper for .Net with no luck. I was under the impression that maybe a I can open these pages in Firefox using an array of links one by one and save them from a command line program. Any help or insight in this regard will be highly appreciated. Regards</p>