We've recently moved much of our site to a Web CMS which published to www1
rather than the old www
.
To clean up the search results, we've added an exclusion in robots.txt for the old hostname and a server redirect on the old homepage from www
to www1
:
RedirectMatch 301 /$ http://www1.example.com/
RedirectMatch 301 /index.htm http://www1.example.com/
This has worked well for search results, removing many outdated pages from the index, but when searching for the main site we see the old www
URL listed, followed by the message:
A description for this result is not available because of this site's robots.txt.
What we really want is www1
URLs in the search result (and no message about robots.txt). Any ideas?