Custom 404 Error Pages

I made an interesting discovery this morning. A few weeks ago I was doing a little SEO on The Wargame Company (Devon) and thought I would look into utilising Google SiteMaps. After creating the XML file with the correct format it's just a matter of having Google approve it. They do this by accessing a random page i.e. www.domain.com/GooglesWonderfulPageddmmyyyyhhmmssmmm (which clearly should return a 404) and check the response code -I guess to ensure that you're not trying to spoof the pages in some way.

"What's the problem? I've got custom 404 pages" I hear you cry! Well, if like us you've written some fancy page to handle the error and email you/log it to a database, it turns out that you're not returning a 404 error at all!

What I discovered was that if you configure IIS to handle 404 error pages with a URL you're actually returning a response code of 200. After a little thinking, the only conclusion we could come to was that when setting it as a URL in IIS you're actually redirecting the request which is either a 301 or perhaps a 307 (see http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html for more information on response codes) and then the final page the user hit's returns a 200 (Response Status "OK") rather than the desired 404 -clearly not what we want!

After a little more investigation we also found that the same thing happened when using ASP.Net's built in handlers and the same thing happens, the only time it doesn't is when you handle the 404 with a File in IIS rather than a URL.

"What can I do about it?" Well that's simple, if you're going to use a URL to handle your 404 errors, make sure you change the Response Status Codes to the correct code, i.e. 404, this is pretty simple to do:

ASP.Net 2.0: Page.Response.StatusCode = 404;

ASP.Net 1.1 (I think): Response.StatusCode = 404;

ASP: Response.Status = "404 You are Unauthorized"

I hope that helps someone out there!

Tim

Update: I've just run fiddler on The Wargame Company (Devon) and and can confirm you get a Response Status Code of 301 before the 200.