Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has been deleted but it still has not been crawled and shows that robots are disallowed access. EVen in the absence of a robots.txt file. The site is a wordpress based website - everything seems to suggest that there should be no block for any crawlers.

Running a search for site:claimsadvicecentre.co.uk should bring up atleast 5 pages however its only listing the main page.

What could be wrong here?

share|improve this question
2  
Google webmaster tools is good for giving feedback of problems with crawling your site: google.com/webmasters –  John C Feb 5 '12 at 9:18
    
What's happen in the .htaccess file - your robots file looks ok. –  toomanyairmiles Feb 5 '12 at 9:44
    
There shouldn't be a robots file to begin with. –  Ali Feb 5 '12 at 10:06
1  
its there. just delete it from the root of your site, you could just leave a blank one in place then it won't 404 claimsadvicecentre.co.uk/robots.txt –  toomanyairmiles Feb 5 '12 at 10:10
    
OMG! I swore I deleted it from my FTP... –  Ali Feb 5 '12 at 10:10
show 1 more comment

3 Answers 3

up vote 3 down vote accepted

So far, nothing seems immediately wrong. Instead, some of your assumptions appear to be wrong.

First, questions.
When you say: "but it [] shows that robots are disallowed" what is the "it" being referred to?

How long has it been since you deleted the robots file? That's not going to make any difference until you get crawled again.

Now, your search example suggests that you have been indexed, at least to some degree. But that doesn't mean the engines will decide all your pages will be worth actually returning as search results.

Beyond that, searches with the site: operator do not necessarily return everything indexed for a site, but only a selection. If you need to actually know how well your site's been crawled, you get that information from Webmaster Tools.

share|improve this answer
    
I guess its been almost like 5 days now since it has been deleted. When I mean it I'm referring to the results of the search for site:claimsadvicecentre.co.uk . –  Ali Feb 5 '12 at 10:07
    
robots.txt is appears to still be in place claimsadvicecentre.co.uk/robots.txt –  toomanyairmiles Feb 5 '12 at 10:11
    
I just reuploaded a different robots.txt - can't for the life of me understand how come the file would still be there even after being deleted –  Ali Feb 5 '12 at 10:26
add comment

So the robots file still appears to be present but the code has changed since i looked at it first thing it now shows:-

User-agent: *
Allow: /

But to allow all, it should be (what it was this morning)

User-agent: *
Disallow:

You can find some more examples in the wiki article on Robots Exclusion Standard.

Could it be that one of your wordpress plugins is auto generating a robots file?

share|improve this answer
add comment

To combine everything into one big answer, here's what you should do...

  1. Make sure your robots.txt is correct. Here's what it should look like if you want the crawlers to index everything on your site:

    User-agent: *
    Disallow:

    Please note that the Allow field is not officially supported by all crawlers (Disallow is the universally accepted field).

  2. Create an XML Sitemap that lists the pages on your site. You can do this manually, or you can use an automatic generator.
  3. Register your site with Google Webmaster Tools.
  4. Submit your XML Sitemap to Google Webmaster Tools.

Once you've completed these steps, your site will be well on its way to being indexed.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.