1

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my robots.txt file:

User-agent: *
Disallow:

Sitemap: http://www.sonjalimone.com/sitemap_index.xml

I have waited over 24 hours for Google to recrawl my site so I must have something wrong in the robots.txt file. It is a WordPress site if that makes any difference, though I don't see why it would.

Does anyone know what else might cause this issue or is there something wrong with the above?

Improve this question
  • 3
    Did you check your logs for googlebot hits? Crawling and inclusion in the search index is very different. It can takes days or weeks to be listed in the search engine results. – jeffatrackaid Jun 25 '14 at 15:30
2

Google will fetch the robots.txt file itself from your site every 24 hours. If you make changes to your robots.txt file, you must wait a day to ensure that Googlebot picks up your changes.

After it has the correct robots.txt file, Googlebot will start crawling and indexing your entire site properly. As a general rule, I expect to see changes to the documents that Google indexes in about two weeks. If you have a large site, the deeper pages may take as much as a month or two to get recrawled.

Use the fetch as Google feature from "Crawl" -> "Fetch as Google" in Google Webmaster Tools to ensure that Googlebot is able to download the pages that you expect.

You can also use the Blocked URLs tool under "Crawl" -> "Blocked URLs" in Google Webmaster Tools to ensure that Google is seeing the correct version of your robots.txt file and that it can crawl any URL that you specify in that tool.

Improve this answer
2

24 hours is not enough time to wait. Google will almost certainly not be crawling your site that frequently. It could take days or weeks before they re-index your robots.txt and crawl your site again.

And to answer the question you will have next, no, there is nothing you can do to speed that up.

Improve this answer
0

If you do not want to block bots, then I recommend you remove robots.txt. In my experience it often causes more harm than good due to errors. In the absence of robots.txt, most search engines will crawl your entire site.

Then use Google's webmaster tools to submit a sitemap.

Improve this answer
  • The version of robots.txt in the question looks fine and should allow the entire site to crawled. I've never had a problem with a robots.txt file like this that allows all crawling. It is very similar to one that I use successfully on one of my sites – Stephen Ostermiller♦ Jun 25 '14 at 16:29
  • No it should not cause issues, but years ago JohnMu, a Google employee, recommended not using robots.txt if you plan to allow everything anyway. goo.gl/PEQKWi – jeffatrackaid Jun 25 '14 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.