recently it takes 5 to 8 days for my site (www.StudyHorror.com) to be indexed. I'm updating the website everyday.

What are the possible causes for this problem?

  • For Example, this url link hasn't been indexed on Google for 6 days. Any ideas? – studyhorror Aug 15 '11 at 1:23

Here is your reason


User-agent: * Disallow:

Remove that !

| improve this answer | |
  • That should definately do it. – chrisjlee Aug 15 '11 at 17:01
  • ??? OMG! why? I thought every site must have one robots.txt. Can you please explain for me? – studyhorror Aug 16 '11 at 4:25
  • @studyhorror this robotstxt.org/robotstxt.html and this en.wikipedia.org/wiki/Robots_exclusion_standard should help you – HackToHell Aug 16 '11 at 13:29
  • @HackToHell, Chris J. Lee: Thanks you guys for your help! All the best to you! Kind Regards! – studyhorror Aug 17 '11 at 0:05
  • The robots.txt appears to be blank now but having Disallow: with nothing after it doesn't block anything, so this definitely isn't the problem. – DisgruntledGoat Oct 24 '11 at 18:35

I would like to add into your comment the option to adjust google crawl rate.

I think it's just a suggestion to Goolge how frequently we are confident for Google to update the content. If it's too high, it can consume heaps of our bandwidth.

Therefore, it's not possible to adjust Google crawl rate by setting that up. :)

| improve this answer | |

If your site crawl slowly and you want fast crawling then you have to update your site's crawl rate in google webmaster tool.

here it is:

To change the crawl rate:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Under Site configuration, click Settings.
  3. In the Crawl rate section, select the option you want.
| improve this answer | |

I'd follow the recommendations people gave and if you want even faster Index make sure you have a sitemap and update it frequently or when adding fresh content.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.