1
vote
2answers
107 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
2answers
669 views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
5
votes
1answer
120 views

How to fix thousands of 404's?

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
2
votes
1answer
276 views

Edit robots.txt on google sites

Getting many 404 errors on my website for the following URLs: /system/app/pages/admin/revisions* I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google ...
27
votes
3answers
2k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...