1

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www.

To clean up the search results, we've added an exclusion in robots.txt for the old hostname and a server redirect on the old homepage from www to www1: 

RedirectMatch 301 /$ http://www1.example.com/
RedirectMatch 301 /index.htm http://www1.example.com/

This has worked well for search results, removing many outdated pages from the index, but when searching for the main site we see the old www URL listed, followed by the message:

A description for this result is not available because of this site's robots.txt.

What we really want is www1 URLs in the search result (and no message about robots.txt). Any ideas?

Improve this question

1 Answer 1

Reset to default
3

If you have a robots.txt on the www server that prevents crawling there, then Googlebot will never see the 301 redirects that you put in place. It won't be able to crawl the URLs that are redirecting.

You want to remove the www site from robots.txt and let Google crawl it. Then they will know what the correct canonical URL for your website is now.

Improve this answer
4
  • That makes sense. We need to have the old site in robots.txt though as it was messing up the search results. So I guess we need robots.txt to disallow the site but allow the root page?
    – pelms
    Oct 7, 2013 at 8:42
  • If it is just your index page that is redirected and you don't want the rest of the site crawled, then you would need to craft a robotst.txt file that allowed the root page only. Since Google understands "Allow" directives, you could use something like Allow: /$ and Allow /index.htm$
    – Stephen Ostermiller ♦
    Oct 7, 2013 at 15:56
  • 1
    Tried Disallow: /*/* to just disallow subdirectories, with a .htaccess 301 redirect on the root and old homepage. This has got rid of the 'robots.txt' message but Google is still displaying the old URL. Will wait a few days and see if it updates.
    – pelms
    Oct 10, 2013 at 16:23
  • 1
    Checked again and the old URL has been replaced by the new one. Success :¬)
    – pelms
    Oct 25, 2013 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.