Q:

A GUI-Less Browser to scrape with

I'm writing a tool to get info from several websites, all require me to log in. My normal approach is to follow the requests and responses in Fiddler or alike, and follow that direct path.
However, that feels a bit strict. A minimal change in the website could break my code. So I'm looking for something like a UI-less browser that I can use the following way:
Browser.Load("https://sourceforge.net/account/login.php");
Browser.Document.ElementById("form_loginname").Value = "login";
Browser.Document.ElementById("form_pw").Value = "password";
(Browser.Document.ElementById("login") As WebButton).Click(); // the login button is named "login".

After this code finishes I'd like to see the page I would get in a regular browser.
Does something like I described exist?
EDIT - C# support is preferred

A:

This is a lot like automated user testing for web applications, with the key difference being you don't own the tested application. 
Selenium is a popular library to automate driving a browser.  If you want to run a program using Selenium in a headless fashion, you can use a headless X server such as Xvfb.  Regarding not using Xvfb, there is a older question on alternatives.
Since Selenium can drive many browsers from a variety of programming languages, I encourage you to explore the Selenium tag on StackOverflow.
Of particular interest for you would be something like SimpleBrowser.WebDriver: Selenium bindings for an in-memory light weight browser for .Net.   Actually, SimpleBrowser directly might address your needs, without the added complexity of Selenium.

