Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www.

To clean up the search results, we've added an exclusion in robots.txt for the old hostname and a server redirect on the old homepage from www to www1:

RedirectMatch 301 /$ http://www1.example.com/
RedirectMatch 301 /index.htm http://www1.example.com/

This has worked well for search results, removing many outdated pages from the index, but when searching for the main site we see the old www URL listed, followed by the message:

A description for this result is not available because of this site's robots.txt.

What we really want is www1 URLs in the search result (and no message about robots.txt). Any ideas?

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

If you have a robots.txt on the www server that prevents crawling there, then Googlebot will never see the 301 redirects that you put in place. It won't be able to crawl the URLs that are redirecting.

You want to remove the www site from robots.txt and let Google crawl it. Then they will know what the correct canonical URL for your website is now.

share|improve this answer
    
That makes sense. We need to have the old site in robots.txt though as it was messing up the search results. So I guess we need robots.txt to disallow the site but allow the root page? –  pelms Oct 7 '13 at 8:42
    
If it is just your index page that is redirected and you don't want the rest of the site crawled, then you would need to craft a robotst.txt file that allowed the root page only. Since Google understands "Allow" directives, you could use something like Allow: /$ and Allow /index.htm$ –  Stephen Ostermiller Oct 7 '13 at 15:56
1  
Tried Disallow: /*/* to just disallow subdirectories, with a .htaccess 301 redirect on the root and old homepage. This has got rid of the 'robots.txt' message but Google is still displaying the old URL. Will wait a few days and see if it updates. –  pelms Oct 10 '13 at 16:23
1  
Checked again and the old URL has been replaced by the new one. Success :¬) –  pelms Oct 25 '13 at 8:24
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.