4

My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data.

A) In one scenario, these URLs are part of a JS object I'm storing in a variable

<script nonce="abc123">
  const foo = { badUrl: "http://www.example.com/{these aren't the droids you're looking for" };
</script>

B) There's another scenario in which it's rendered like this:

<script id="my-data" type="application/json">
{ "badUrl": "http://www.example.com/{these aren't the droids you're looking for" };
</script>

  1. Which one is Googlebot likely trying to crawl (if you had to guess)? Is it both?
  2. Does Googlebot actually try to index URLs in random JavaScript objects or JSON blobs?
  3. How do I stop it from being so silly?

1 Answer 1

4

Googlebot is likely to discover and crawl URLs found either in JavaScript or JSON. Googlebot has been doing this for many years, well before it could execute JavaScript. It appears it uses heuristics to pick literals that look like URLs out of page source code.

Google can index URLs that it finds this way if it crawls them and finds a page with content there. In your case, I would expect Googlebot to crawl error pages which it wouldn't try to index.

You can prevent this problem by making your literals look less like URLs by building them from pieces. For example you might change it to:

<script nonce="abc123">
  const foo = { badUrl: "http://www.example.com/"+"{these aren't the droids you're looking for" };
</script>

That way Googlebot's heuristics would pick out http://www.example.com/ which is a valid URL and {these aren't the droids you're looking for wouldn't be interpreted as a URL at all.

The other option is to have this data in JavaScript or JSON files that are blocked by robots.txt. If Googlebot can't load the files, it won't try to find links in them. For that solution to work, the data has to able to be generated in a separate file rather than inline. Google also likes to be able to execute all your JavaScript to see how your page looks. You'd have to make sure that your page still renders fine with only very minor changes when that JavaScript can't be loaded or it could hurt your SEO.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.