Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a client that has 3 domain names. For the sake of this argument I will call them www.website1.com, www.website2.com and www.website3.com.

He used to have a website package with VistaPrint and www.website1.com was his main business website and website2 and website3 used to redirect to the main website.

However recently I created a new WordPress website for him and now his main domain name is www.website3.com and website1 and website2 simply redirect to the main website.

But when I go on Google and search for the domain name (any of the three) it does not show me the description of the website and it says: A description for this result is not available because of this site's robots.txt – learn more.

I used to have a robots.txt file but have deleted it since the migration to the client's host however Google seems to have indexed it in the meanwhile. I got in touch with the hosting company and they said that it is a virtual file and that it doesn't physically exist - and that the issue with the description in Google is happening because it has indexed the website while the robots.txt was there.

I want all references to website1 and website2 to vanish from Google and I only want it to index www.website3.com, which is the main business website - with the correct description of the site

What are my options? I logged into Google Webmaster Tools and it still says that the file is there.

I hope you understand what I want to achieve but feel free to ask for clarification

Any advice will be greatly appreciated as I want to solve this issue ASAP before it starts damaging the client's online presence and reputation

share|improve this question
    
just checked webmaster tools and this is what it says: Googlebot is blocked from http://www.mydomain.com/ –  Bo5ao Jun 10 '13 at 13:17

1 Answer 1

up vote 2 down vote accepted

Rather than delete your robots.txt file, you should upload one that explicitly allows all crawling.

User-agent: *
Disallow:

When you delete a robots.txt file, Google often uses the old one for some time period (maybe even weeks) to make sure that you didn't delete it by accident.

share|improve this answer
    
you are not going to believe it! I seem to have left an option ticked that says something like "Discourage crawlers to index this site" - so naturally I unticked it and what Wordpress did was modify the content of my htaccess file and the website was all of a sudden not available :error 500. Never thought Wordpress actually modifies the htaccess file –  Bo5ao Jun 10 '13 at 14:24
    
we learn something new everyday - I now know that messing with that WP option will update my htaccess file!! –  Bo5ao Jun 10 '13 at 14:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.