3

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message:

URL and linked pages submitted to index.

The problem is that my site is not indexed yet although Google bot has started crawling it. When I search Google for my domain name I get:

A description for this result is not available because of this site's robots.txt

I have removed the disallow option and in fact robots.txt is as follows for the last 2 days:

User-agent: Googlebot-Image
Disallow: / 

In fact, when I test the robots.txt file via the Webmaster Tools I get:

Allowed
Detected as a directory; specific files may have different restrictions

Google is crawling it at a slow pace (~400 pages per day).

How can I can tell Google to index my site before finishing with the crawling process? If this is not possible, can I speed up the crawl?

  • Too soon to say that Google is not indexing your site. The crawl is a separate process from data analysis and SERP display. A couple of days ago is followed by 1-4 weeks. After that point, you can start saying that Google isn't indexing anything. – Fiasco Labs Dec 7 '13 at 18:53
6

I have removed the disallow option and in fact robots.txt is empty for the last 2 days.

If your robots.txt files goes away or becomes empty, Google sometimes assumes that this is an error and won't honor the change for some time. Instead of an empty robots.txt file, you should make one that explicitly allows everything:

User-agent: *
Disallow: 

After you have changed your robots.txt file, Google usually takes at least a week and sometimes as much as a month to index a website. You'll just have to wait it out.

If it hasn't crawled the entire website, it will index the pages that it has crawled before completing a crawl the entire site.

The only way to speed up crawl process is to increase the reputation of the site with more high quality inbound links. The number of pages that Googlebot crawls and the rate it which it returns to re-crawl them are governed mainly by Pagerank.

  • Thanks for the answer. I have tested my robots.txt through Webmaster tools and I got: Allowed Detected as a directory; specific files may have different restrictions. Is it possible to just index the homepage? – Nick Dec 7 '13 at 13:32
  • 1
    If Googlbot is actively crawling your site then there isn't a problem with the robots.txt file. Its just a matter of waiting until the content gets indexed. – Stephen Ostermiller Dec 7 '13 at 14:27
  • Thanks for your answer Stephen. If this is the case then it can take months! Can I at least index the homepage? – Nick Dec 7 '13 at 18:29
  • 1
    You can't speed up Google in any way about this. It shouldn't take months. Maybe a month for some of the less popular pages on your site. I'd be surprised if it took more than a week until the home page gets indexed. – Stephen Ostermiller Dec 7 '13 at 19:48
  • Google finally indexed the homepage after 5 days! It seems I just have to wait! Thanks for your help! – Nick Dec 10 '13 at 11:09
-2

Put the following into your robots.txt to tell Googlebot to crawl everything.

User-agent: *
Allow: /
  • This doesn't answer the question. In this case Googlebot is already crawling everything. Your robots.txt suggestion would be better with a blank disallow line (Disallow: ) rather than an Allow: / line. Most robots don't understand disallow. – Stephen Ostermiller Mar 30 '18 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.