5

In my Search Console I had a sudden increase in 404's. They were linked from a webpage that listed files from the site in a table but none of these URLs were nested inside an <a> tag.

This was the only instance of the URLs I could find on the page which led me to question, does Google still follow a url string even if it's not in an <a> tag?

9

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/

Google crawled stuff like

<script>var test="www.example.net/page.html";</script>

or simply

www.example.net/page.html

So i think they only look for a URL patterns.

  • 2
    It is worth adding that it is unclear if Google assigns value (link juice or Pagerank) to the non-anchor links. Also, 404 errors listed in search console won't hurt your site as long as you don't expect to have content on those URLs. There can be a million 404 URLs caused by this problem and Google won't penalize your site in any way. – Stephen Ostermiller Aug 30 '16 at 16:15
  • Good effort! However, I don't see an example of your first one (JS comment) in your list of tests? In fact, you later state, "Comments and CDATA blocks Google seems to ignore completely." (Google translated) Also, all the "successfully crawled" URLs in your test appear to be full absolute URLs. Whereas the examples listed above (www.example.net) are technically relative. (?) – MrWhite Aug 30 '16 at 19:37
  • @w3dk you are right, i wrote it out of memory, i edited the answer – Milo Tischler Aug 30 '16 at 20:18
  • ...and what about the relative/absolute difference? – MrWhite Aug 30 '16 at 21:24
  • In my experience, Googlebot will try to crawl any string with a slash embedded in the middle, or any string that ends in ".html". So Googlebot crawls some relative URLs that it finds in JavaScript or page text. – Stephen Ostermiller Aug 30 '16 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.