Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message:

URL and linked pages submitted to index.

The problem is that my site is not indexed yet although Google bot has started crawling it. When I search Google for my domain name I get:

A description for this result is not available because of this site's robots.txt

I have removed the disallow option and in fact robots.txt is as follows for the last 2 days:

User-agent: Googlebot-Image
Disallow: / 

In fact, when I test the robots.txt file via the Webmaster Tools I get:

Detected as a directory; specific files may have different restrictions

Google is crawling it at a slow pace (~400 pages per day).

How can I can tell Google to index my site before finishing with the crawling process? If this is not possible, can I speed up the crawl?

share|improve this question
Too soon to say that Google is not indexing your site. The crawl is a separate process from data analysis and SERP display. A couple of days ago is followed by 1-4 weeks. After that point, you can start saying that Google isn't indexing anything. – Fiasco Labs Dec 7 '13 at 18:53
up vote 5 down vote accepted

I have removed the disallow option and in fact robots.txt is empty for the last 2 days.

If your robots.txt files goes away or becomes empty, Google sometimes assumes that this is an error and won't honor the change for some time. Instead of an empty robots.txt file, you should make one that explicitly allows everything:

User-agent: *

After you have changed your robots.txt file, Google usually takes at least a week and sometimes as much as a month to index a website. You'll just have to wait it out.

If it hasn't crawled the entire website, it will index the pages that it has crawled before completing a crawl the entire site.

The only way to speed up crawl process is to increase the reputation of the site with more high quality inbound links. The number of pages that Googlebot crawls and the rate it which it returns to re-crawl them are governed mainly by Pagerank.

share|improve this answer
Thanks for the answer. I have tested my robots.txt through Webmaster tools and I got: Allowed Detected as a directory; specific files may have different restrictions. Is it possible to just index the homepage? – Nick Dec 7 '13 at 13:32
If Googlbot is actively crawling your site then there isn't a problem with the robots.txt file. Its just a matter of waiting until the content gets indexed. – Stephen Ostermiller Dec 7 '13 at 14:27
Thanks for your answer Stephen. If this is the case then it can take months! Can I at least index the homepage? – Nick Dec 7 '13 at 18:29
You can't speed up Google in any way about this. It shouldn't take months. Maybe a month for some of the less popular pages on your site. I'd be surprised if it took more than a week until the home page gets indexed. – Stephen Ostermiller Dec 7 '13 at 19:48
Google finally indexed the homepage after 5 days! It seems I just have to wait! Thanks for your help! – Nick Dec 10 '13 at 11:09

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.