We've recently moved much of our site to a Web CMS which published to
www1 rather than the old
To clean up the search results, we've added an exclusion in robots.txt for the old hostname and a server redirect on the old homepage from
RedirectMatch 301 /$ http://www1.example.com/ RedirectMatch 301 /index.htm http://www1.example.com/
This has worked well for search results, removing many outdated pages from the index, but when searching for the main site we see the old
www URL listed, followed by the message:
A description for this result is not available because of this site's robots.txt.
What we really want is
www1 URLs in the search result (and no message about robots.txt). Any ideas?