Does it make sense to return a
410 instead of
404 when some page has been permanently removed? In this video, Matt Cutts says that Google treats them the same, but are there other reasons to implement a
410 (except for technical correctness)?
On our website we have recently implemented 410 errors for pages that have been removed permanently. We have around 40 million pages in the Google index and get crawled with 2 million requests per day by the Googlebot.
After cleaning up our database we found a large number of 404 errors kept showing up in the Crawl Errors on Webmaster Tools. When we switched to 410 statuses instead of 404 the number of errors per day cut in half while the total number of requests done kept around the same. So that's more successfully crawled pages per day in the end. It looks like pages with a 404 status get crawled multiple times before being removed from the index fully.
So I would say, yes definitely worth using 410 where appropriate in order to maintain your Google Crawl budget.
There is more than Google in this world. A 410 unambiguously tells a bot that the file is gone. A 404 does not. A persistent bot might keep trying to find a 404 indefinitely whereas they might stop trying to find a 410 immediately which would make your server very happy.
Besides there being more search engines than Google out there, there's also no reason to assume that Google won't ever change the way they treat 410 responses. Indeed, it seems that's already happened: the information Matt Cutts quotes in the video is from 2007, whereas this post from 2009 by John Mu on Google's Webmaster Central forums says otherwise:
"I followed up on the 404 vs 410 thing with the team here. As mentioned by some others here & elsewhere, we have generally been treating them the same in the past.
However, after looking at how webmasters use them in practice we are now treating the 410 HTTP result code as a bit "more permanent" than a 404. So if you're absolutely sure that a page no longer exists and will never exist again, using a 410 would likely be a good thing."
So it looks like Google is now indeed treating 410 responses differently from 404s.
In my experience, Google doesn't understand 410 error codes.
Some time ago, I changed a full site so all the old URLs are "410 Gone". I can't use 301 redirects because there is no direct relationship to a new URL.
I used a
.htaccess to tell Google the old URLs are gone, but it keeps telling me there are a HUGE amount of 404 errors on my site. I checked the URLs in the crawl error report and they are all are 410, not 404 as it states.
There is more than Google in this world.
Quoted for truth. Given your requirement, to tell bots (and presumably humans as well) that a page has been permanently removed, I'd actually opt for a
301 redirect to a page explaining (if necessary) why the content was removed, or take the bot / user back to the index page.
The lesser-known HTTP error codes usually aren't as well supported, understood or implemented across the board, so I generally stick to the ones that are in order to ensure the expected and hopefully seamless browsing experience.
Choosing which option to go with is dependent on how many request are still coming through for the (now) missing content.