2

Problem

I have a website with documentation that includes examples with plain text URLs. The other day, I noticed that Google Webmaster was telling me that one of those URLs generated a Page Not Found error.

Question

What is the best way to prevent Google from scraping such plain text URLs? (other than using example.com because I am using my domain name in those sample URLs, which I think makes more sense.)

Hide directory solution

Note that I found out that I could add a folder, in my case /api, to the robots.txt and at least all of those URLs were ignored.

User-agent: *
Disallow: /api

However, all the URLs in my documentations are not just about the REST API and I still have the problem with some other pages that I just cannot add to robots.txt (at least, to me that would not make sense, adding each page individually to robots.txt, when those pages do not exist in the first place?!)

That being said, I'm not so sure that this is a good solution as far as SEO is concerned since in effect those pages still generate an Internal Link 404 error (or maybe it's considered to be a 403?).

  • Google may find those URLs, attempt to crawl them, and then report them as errors in Search Console. However, they are not going to hurt your site in any way. Google doesn't penalize sites with 404 errors. In fact, it expects properly working sites to have some crawl errors. Google's John Mueller has said so here: plus.google.com/+JohnMueller/posts/RMjFPCSs5fm – Stephen Ostermiller Sep 11 '17 at 15:21
2

Generate the text with a simple javascript function, so the literal you want will appear when read, but not when scraped.

in the header something like

<script>function hide(str)
{   
    document.write('http://example.com'+str);
}
</script>

In line something like

<script>hide("/foo/bar.html")</script>
  • Isn't Google capable of running your JavaScripts? – Alexis Wilke Sep 3 '17 at 23:54
  • Theoretically, I supppose they could; but in practice no. You can never know if a script will run forever, and scrapers can't afford to wait. The same basic method can be used to hide email addresses from being harvested, or to prevent naughty robots from following links to places they shouldn't go. – ddyer Sep 4 '17 at 4:34
  • 1
    Google also has a heuristic to find things that look like URLs in JavaScript code. This approach won't solve the problem. Google will just find the same URLs from the JavaScript strings. You might be able to get this to work with additional obfuscation. Google likes to try to crawl string with slashes ('/'), ending with .html or starting with http. Somehow transform the strings to remove those and have the code add them back in. – Stephen Ostermiller Sep 11 '17 at 15:22
0

(you might want to confirm if the pages are html or text - I'm guessing html from your response to the JavaScript query)

You can modify the links with ref="nofollow" as a parameter. As per https://support.google.com/webmasters/answer/96569?hl=en this is something Google advocates.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.