recently it takes 5 to 8 days for my site (www.StudyHorror.com) to be indexed. I'm updating the website everyday.

What are the possible causes for this problem?

  • For Example, this url link hasn't been indexed on Google for 6 days. Any ideas? Aug 15, 2011 at 1:23

4 Answers 4


Here is your reason


User-agent: * Disallow:

Remove that !

  • That should definately do it.
    – chrisjlee
    Aug 15, 2011 at 17:01
  • ??? OMG! why? I thought every site must have one robots.txt. Can you please explain for me? Aug 16, 2011 at 4:25
  • @studyhorror this robotstxt.org/robotstxt.html and this en.wikipedia.org/wiki/Robots_exclusion_standard should help you
    – HackToHell
    Aug 16, 2011 at 13:29
  • @HackToHell, Chris J. Lee: Thanks you guys for your help! All the best to you! Kind Regards! Aug 17, 2011 at 0:05
  • The robots.txt appears to be blank now but having Disallow: with nothing after it doesn't block anything, so this definitely isn't the problem. Oct 24, 2011 at 18:35

I would like to add into your comment the option to adjust google crawl rate.

I think it's just a suggestion to Goolge how frequently we are confident for Google to update the content. If it's too high, it can consume heaps of our bandwidth.

Therefore, it's not possible to adjust Google crawl rate by setting that up. :)


If your site crawl slowly and you want fast crawling then you have to update your site's crawl rate in google webmaster tool.

here it is:

To change the crawl rate:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Under Site configuration, click Settings.
  3. In the Crawl rate section, select the option you want.

I'd follow the recommendations people gave and if you want even faster Index make sure you have a sitemap and update it frequently or when adding fresh content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.