5

In my Search Console I had a sudden increase in 404's. They were linked from a webpage that listed files from the site in a table but none of these URLs were nested inside an <a> tag.

This was the only instance of the URLs I could find on the page which led me to question, does Google still follow a url string even if it's not in an <a> tag?

Improve this question

1 Answer 1

Reset to default
9

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/

Google crawled stuff like

<script>var test="www.example.net/page.html";</script>

or simply

www.example.net/page.html

So i think they only look for a URL patterns.

Improve this answer
5
  • 2
    It is worth adding that it is unclear if Google assigns value (link juice or Pagerank) to the non-anchor links. Also, 404 errors listed in search console won't hurt your site as long as you don't expect to have content on those URLs. There can be a million 404 URLs caused by this problem and Google won't penalize your site in any way.
    – Stephen Ostermiller ♦
    Commented Aug 30, 2016 at 16:15
  • Good effort! However, I don't see an example of your first one (JS comment) in your list of tests? In fact, you later state, "Comments and CDATA blocks Google seems to ignore completely." (Google translated) Also, all the "successfully crawled" URLs in your test appear to be full absolute URLs. Whereas the examples listed above (www.example.net) are technically relative. (?)
    – MrWhite
    Commented Aug 30, 2016 at 19:37
  • @w3dk you are right, i wrote it out of memory, i edited the answer
    – Milo Tischler
    Commented Aug 30, 2016 at 20:18
  • ...and what about the relative/absolute difference?
    – MrWhite
    Commented Aug 30, 2016 at 21:24
  • In my experience, Googlebot will try to crawl any string with a slash embedded in the middle, or any string that ends in ".html". So Googlebot crawls some relative URLs that it finds in JavaScript or page text.
    – Stephen Ostermiller ♦
    Commented Aug 30, 2016 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.