0

I implemented the unavailable_after tag in my pages like the below and set to unavailable after 3 days but in Google Webmaster Tools I got 404 for that page.

<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 05-May-2014 15:00:00 EST">

How do I know Googlebot visit(crawl) my pages again to read this tag (or) how long Googlebot will take to crawl the pages again ? please give me a suggestion!.

Note: I set unavailable_after tag for certain content types(not whole website) with highest priority(1.0) in XML sitemap.

Improve this question

1 Answer 1

Reset to default
2

Googlebot recrawls pages with a frequency based on how popular they are and how often they have changed in the past. CNN's home page gets crawled every few minutes. A PageRank 0 page near the back of an unpopular blog may only get crawled monthly.

As a general rule of thumb, I allow two to three weeks for all the pages on my site to get recrawled when I make changes.

I don't believe that Googlebot actually uses the crawl priority or change date in XML sitemaps to trigger recrawls. The only way that I know to force Google to crawl a page is to use the "Fetch as Google" feature in Webmaster Tools. That tool has a limit of 1000 pages that have to be submitted manually one by one.

See: How to request Google to re-crawl my website?

Improve this answer
3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.