XmlHttp, WinHttp, Cookie-based Auth and Too Much Coffee

I had a couple of cups of coffee with dinner last night, and ended up perched in front of my PC, unable to sleep, and unable to solve a problem involving XMLHTTP and (as it turns out) cookies.

As a long-time Bigpond Cable user at home, I was interested in Darryn's scripts for BPA Usage monitoring with MRTG. The version that was there last night used Lynx to download a blob of XML

“Surely this is what that XMLHttp thing I've heard of is for?”, I thought, while quietly shivering in the cold (or was it just the coffee?).

About an hour and three quarters after receiving my first “Access Denied” message when trying to connect to an HTTPS site (Basic auth) using XMLHttp, I worked out that there were probably cookies involved for authentication (or at least, the authentication led to the client being sent a cookie rather than being an end in itself) as well as the Basic credentials - I set up a quick local test site that was fine. Web searching proved fruitless, but I wasn't ready to give up.

I even tried a Webclient-based .Net application to see if it fared any better (even though the point was to try to use scriptable features and avoid an external exe) - but it had much the same issue.

The answer - which took about three minutes to implement - was to use the WinHTTP object instead, which handily resembles a client browser enough that it accepts and re-sends cookies on what I assume is a per-instance basis, whereas XMLHttp is less browser-like, and more about the methods and invocations (I found references to being able to look at the headers returned from the invocation by XmlHttp, but building a cookie engine just seemed like more work than was necessary - again, this is purely supposition and conjecture!).

So, the VBS code returning “Access Denied” looked like this:

site = "https://usage/theusagepage"
set xmlhttp = CreateObject("Microsoft.XMLHttp")
xmlhttp.Open "GET", site, False, Username, Password
' then do stuff with the response

And the solution turned out to be something like this instead, which sailed through without a problem:

site = "https://usage/theusagepage"
set winhttp = CreateObject("WinHttp.WinHttpRequest.5")
winhttp.Open "GET", site, False
winhttp.SetCredentials Username, Password, 0 ' for web server, not proxy
set xmlDoc = CreateObject("Microsoft.XMLDOM")

Thought I'd share it to hopefully save someone some time - most of the KB articles on the subject were fairly specific errors that didn't quite match up.