I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has been deleted but it still has not been crawled and shows that robots are disallowed access. EVen in the absence of a robots.txt file. The site is a wordpress based website - everything seems to suggest that there should be no block for any crawlers.

Running a search for site:claimsadvicecentre.co.uk should bring up atleast 5 pages however its only listing the main page.

What could be wrong here?

  • 2
    Google webmaster tools is good for giving feedback of problems with crawling your site: google.com/webmasters
    – John C
    Commented Feb 5, 2012 at 9:18
  • What's happen in the .htaccess file - your robots file looks ok. Commented Feb 5, 2012 at 9:44
  • There shouldn't be a robots file to begin with.
    – Ali
    Commented Feb 5, 2012 at 10:06
  • 1
    its there. just delete it from the root of your site, you could just leave a blank one in place then it won't 404 claimsadvicecentre.co.uk/robots.txt Commented Feb 5, 2012 at 10:10
  • OMG! I swore I deleted it from my FTP...
    – Ali
    Commented Feb 5, 2012 at 10:10

3 Answers 3


So far, nothing seems immediately wrong. Instead, some of your assumptions appear to be wrong.

First, questions.
When you say: "but it [] shows that robots are disallowed" what is the "it" being referred to?

How long has it been since you deleted the robots file? That's not going to make any difference until you get crawled again.

Now, your search example suggests that you have been indexed, at least to some degree. But that doesn't mean the engines will decide all your pages will be worth actually returning as search results.

Beyond that, searches with the site: operator do not necessarily return everything indexed for a site, but only a selection. If you need to actually know how well your site's been crawled, you get that information from Webmaster Tools.

  • I guess its been almost like 5 days now since it has been deleted. When I mean it I'm referring to the results of the search for site:claimsadvicecentre.co.uk .
    – Ali
    Commented Feb 5, 2012 at 10:07
  • robots.txt is appears to still be in place claimsadvicecentre.co.uk/robots.txt Commented Feb 5, 2012 at 10:11
  • I just reuploaded a different robots.txt - can't for the life of me understand how come the file would still be there even after being deleted
    – Ali
    Commented Feb 5, 2012 at 10:26

So the robots file still appears to be present but the code has changed since i looked at it first thing it now shows:-

User-agent: *
Allow: /

But to allow all, it should be (what it was this morning)

User-agent: *

You can find some more examples in the wiki article on Robots Exclusion Standard.

Could it be that one of your wordpress plugins is auto generating a robots file?


To combine everything into one big answer, here's what you should do...

  1. Make sure your robots.txt is correct. Here's what it should look like if you want the crawlers to index everything on your site:

    User-agent: *

    Please note that the Allow field is not officially supported by all crawlers (Disallow is the universally accepted field).

  2. Create an XML Sitemap that lists the pages on your site. You can do this manually, or you can use an automatic generator.
  3. Register your site with Google Webmaster Tools.
  4. Submit your XML Sitemap to Google Webmaster Tools.

Once you've completed these steps, your site will be well on its way to being indexed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.