Robots are still disallowed even without a robots.txt file

Question

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has been deleted but it still has not been crawled and shows that robots are disallowed access. EVen in the absence of a robots.txt file. The site is a wordpress based website - everything seems to suggest that there should be no block for any crawlers.

Running a search for site:claimsadvicecentre.co.uk should bring up atleast 5 pages however its only listing the main page.

What could be wrong here?

Answer 1

So far, nothing seems immediately wrong. Instead, some of your assumptions appear to be wrong.

First, questions.
When you say: "but it [] shows that robots are disallowed" what is the "it" being referred to?

How long has it been since you deleted the robots file? That's not going to make any difference until you get crawled again.

Now, your search example suggests that you have been indexed, at least to some degree. But that doesn't mean the engines will decide all your pages will be worth actually returning as search results.

Beyond that, searches with the site: operator do not necessarily return everything indexed for a site, but only a selection. If you need to actually know how well your site's been crawled, you get that information from Webmaster Tools.

Answer 2

So the robots file still appears to be present but the code has changed since i looked at it first thing it now shows:-

User-agent: *
Allow: /

But to allow all, it should be (what it was this morning)

User-agent: *
Disallow:

You can find some more examples in the wiki article on Robots Exclusion Standard.

Could it be that one of your wordpress plugins is auto generating a robots file?

Answer 3

To combine everything into one big answer, here's what you should do...

1. Make sure your robots.txt is correct. Here's what it should look like if you want the crawlers to index everything on your site:

   User-agent: *
   Disallow:

   Please note that the Allow field is not officially supported by all crawlers (Disallow is the universally accepted field).

2. Create an XML Sitemap that lists the pages on your site. You can do this manually, or you can use an automatic generator.
3. Register your site with Google Webmaster Tools.
4. Submit your XML Sitemap to Google Webmaster Tools.

Once you've completed these steps, your site will be well on its way to being indexed.