Take the 2-minute tour ×

I'm using Delphi and I'm trying to get the source from web pages. Here is my problem, I tried to get the source for a simple (well probably not) web page with Indy (idHttp) and Clever Components (clHttp). They both returned the same results. However, if I use IE and/or Google Chrome, they both give the same results but different as the Delphi components (Indy and Clever). The Delphi components returned source with many objects missing. However the browsers view source options returned all the code and objects I need. Finally my question is, is there any way I could retrieve a web page source with Delphi exactly the same way as the ones showing by the browsers ?

Thanks in advance


Edit: Looks like any of the delphi components can handle the span tags

share|improve this question
I'd like to make a suggestion about your question (have no answer for you on this, sorry). Unless there are people on SO who know exactly what you are asking about, it seems likely that you need to specify what particular parts IE and Google Chrome managed to show that were missing from the results returned by the Delphi components you tried. –  Andriy M Oct 19 '12 at 5:40
Also, mentioning that web page URL might help –  Jan Doggen Oct 19 '12 at 6:36
@mhoude: What do you mean by "... can handle the span tags". You just request a URL and get back a bunch of characters as response. The components do not handle any of the content of this response at all. –  NilsB Oct 19 '12 at 8:10
add comment

2 Answers

It's probably because the control is sending a User Agent string which is different than the ones used by IE or Chrome. So in other words, the SERVER is sending back a different source than it does for IE or Chrome.

For example, In TIdDHTTP, set:

Request.AcceptEncoding=gzip, deflate
Request.UserAgent=Mozilla/4.0 (compatible; MSIE 6.0; Win32)
Request.Host=(web site name)
share|improve this answer
Note that TIdHTTP manages the ContentType, AcceptEncoding, and Host values for you. The rest can be set manually. –  Remy Lebeau Oct 19 '12 at 20:48
add comment

Basically, requesting the page from the component is doing the same as the browser does. Only the browser might do several more requests and other activities eg. JavaScript that might change the DOM.

Try switching off JavaScript and compare again. If your familiar with chrome developer tools, check the raw input from the first HTTP get and compare that. If still different, then modify your request to make it identical to the chrome request.

share|improve this answer
add comment

Your Answer


