Difference in operation of HEAD and GET when object doesn't exist

Giganews Newsgroups
Subject: Difference in operation of HEAD and GET when object doesn't exist
Posted by:  Ross McMillan (nospam@gmail.com)
Date: Wed, 27 May 2009

I'm writing a thread-based task to freshen objects via HTTP.  I read
metadata from the web object with TidHTTP.Head and if the destination is
older, I use TidHTTP.Put to send the fresh copy up.

If  the object exists I get back a 200 response immediately from the Head
request.  If it doesn't exist Indy raises a read timeout exception after the
read timeout has expired.  If I bracket the call to the Head method with a
try..except and inspect the Response code after the exception has been
handled it is 404 as expected.

If I use a TidHTTP.Get instead and the object doesn't exist the behaviour is
the same, except that the exception occurs almost immediately.

Why the difference?  It seems like the server ignores a HEAD request for a
resource that doesn't exist, and the 404 is being returned locally by Indy
after the read timeout has expired, whereas for a GET request the server
sends back a 404 response as soon as it knows the resource is not present.

Where I have a lot of files to check for the existence of, it seems like
using GET would be more efficient (I could say only request a chunk of 1
byte) rather than HEAD which seems to waste a lot of time waiting around.

Or am I missing something?