Search Console is reporting 404s for URLs that don't exist on my site. However, GoogleBot seems to be crawling a URL by guessing a page based off of relative paths in my links. For example:

I have a page http://www.example.com/joe/product/cool-product

It contains a relative link: <a href="/art/modern">modern art</a>

I seem to be getting a lot of 404s in Search Console for http://www.example.com/joe/product/art/modern

Which doesn't exist and I can only assume the GoogleBot is building this URL from the current page URL and the path in a strange way.

Does this affect my Crawl Budget?

  • 1
    Googlebot is crawling them. Everything Googlebot crawls is part of your sites "crawl budget". Commented Feb 20, 2017 at 12:14
  • Googlebot has a big budget. Most sites don't have to worry about having more pages than Google is willing to crawl. Commented Mar 30, 2017 at 9:29

1 Answer 1

  • Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server.

  • Crawl demand is how much Google wants to crawl your pages. This is
    based on how popular your pages are and how stale the content is in
    the Google index.

  • Crawl budget is “taking crawl rate and crawl demand together.” Google defines crawl budget as “the number of URLs Googlebot can and wants
    to crawl.”

For more read: here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.