Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

recently it takes 5 to 8 days for my site (www.StudyHorror.com) to be indexed. I'm updating the website everyday.

What are the possible causes for this problem?

share|improve this question
    
For Example, this url link hasn't been indexed on Google for 6 days. Any ideas? –  studyhorror Aug 15 '11 at 1:23
add comment

3 Answers

Here is your reason

http://www.studyhorror.com/robots.txt

User-agent: * Disallow:

Remove that !

share|improve this answer
    
That should definately do it. –  chrisjlee Aug 15 '11 at 17:01
    
??? OMG! why? I thought every site must have one robots.txt. Can you please explain for me? –  studyhorror Aug 16 '11 at 4:25
    
@studyhorror this robotstxt.org/robotstxt.html and this en.wikipedia.org/wiki/Robots_exclusion_standard should help you –  HackToHell Aug 16 '11 at 13:29
    
@HackToHell, Chris J. Lee: Thanks you guys for your help! All the best to you! Kind Regards! –  studyhorror Aug 17 '11 at 0:05
    
The robots.txt appears to be blank now but having Disallow: with nothing after it doesn't block anything, so this definitely isn't the problem. –  DisgruntledGoat Oct 24 '11 at 18:35
add comment

I would like to add into your comment the option to adjust google crawl rate.

I think it's just a suggestion to Goolge how frequently we are confident for Google to update the content. If it's too high, it can consume heaps of our bandwidth.

Therefore, it's not possible to adjust Google crawl rate by setting that up. :)

share|improve this answer
add comment

If your site crawl slowly and you want fast crawling then you have to update your site's crawl rate in google webmaster tool.

here it is:

To change the crawl rate:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Under Site configuration, click Settings.
  3. In the Crawl rate section, select the option you want.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.