Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot?

For example, I have some code like this:

var url = "/some/object/--ct_id--/".replace('--ct_id--', content_id);

Regular users end up going to the correct URLs because the --ct_id-- is replaced with an integer. Google seems to be trying to spider "/some/object/--ct_id--/" which isn't a valid link.

Does anyone know what triggers the URL detection? Should I change the variable name? Should I escape the slashes with unicode notation?

  • 1
    Just because Google tries to crawl it, there isn't a problem. It doesn't count against your site in any way. It just makes the report in Google Search Console noisy. – Stephen Ostermiller Oct 14 '16 at 14:50
  • We get asked about 404 errors in search console a lot here. I'm not sure that these questions have the exact same issues, but the answers are helpful: Is it possible for a script tag to be perceived as an inbound link for SEO? and Google follows JavaScript string as relative path - produces 404 error – Stephen Ostermiller Oct 14 '16 at 14:55
  • @StephenOstermiller - Well in the example given, the backend is expecting an integer but gets "--ct_id--" instead and throws an error and I get an email about it. So, it's generating other noise I'd like to cut back on too. – Tim Tisdall Oct 14 '16 at 15:08
  • The heuristic that Googlebot uses is triggering based on the slashes in the string literal. "|some|object|--ct_id--|".replace('|','/').replace('--ct_id--', content_id) would probably do better. However, now that Google knows about that "URL" it may continue to crawl it periodically even if the page doesn't link to it. You may just have to configure your server not to send out email for that case. – Stephen Ostermiller Oct 14 '16 at 15:11
  • I have my server configured to email me about 404 errors only when the referrer is from my own site . This allows me to catch broken internal links but doesn't get very noisy from crawler errors. Crawlers almost never send referrers. – Stephen Ostermiller Oct 14 '16 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.