I assume that search engine robots crawl the whole page and then the canonical tag because pages that are self canonicalized still get indexed and their contents are still parsed.

However, does a self canonical link cause an infinite loop wasting crawl budget?

  • 1
    Short answer? No. These are not links per se' so they do not get followed. Just noted. Cheers!! – closetnoc Feb 16 '17 at 17:42
  • Why would a bot crawl the very same URL again immediately after? Do you assume that bots blindly follow all links on a page? – unor Feb 16 '17 at 21:17
  • @unor I did assume that bots blindly followed all links and that's were the confusion came from regarding these canonical tags. Now I know better thanks to closetnoc – GrapeSoda Feb 17 '17 at 12:48

Bots don’t¹ blindly follow all links on a page.

A bot remembers the URLs it has crawled and the URL it is currently crawling. If it finds a link to one of those URLs again (something which happens all the time -- think of the site navigation, for example), there’s typically no need to crawl it again for some time.

¹ Well, technically they could, of course. And there are likely some peculiar/bugged bots out there that do this. But it would not only waste your resources, also the bot’s. So bot providers have a reason to make their bots "smart": saving costs.

  • Ok, this makes logical sense but I feel I don't fully understand how link equity works. If the bots don't follow a link it has crawled before, how does it pass equity? I thought that the more a link appears in a site the more important it is deemed as it will be getting crawled more thus more juice. I guess I am wrong. – GrapeSoda Feb 17 '17 at 20:37
  • 1
    @GrapeSoda: No matter if there is something like link equity, or how it would work: How would crawling the URL again help here? Unless it has changed in the meantime, it is the very same document, and the bot already knows it (which includes knowing which links go out). In other words: Why crawl again (expensive) instead of looking it up in its own cache/index (not so expensive)? – unor Feb 17 '17 at 21:32
  • I haven't thought about it like that before and it makes complete sense. – GrapeSoda Feb 26 '17 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.