3

I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.

Google search console keeps trying to discover/crawl certain URLs on my site that it shouldn't be doing. It's not a security hazard, but essentially what it's crawling are completely redundant URLs that don't exist in actuality. I have an idea of why it's happening, but I don't want to try and restructure my pages to accommodate Google in this regard.

These unnecessary page crawls always contain a certain phrase in the URL. Suppose this phrase is "unnecessaryPage", as in info-unnecessaryPage.htm is a URL that Google attempts to crawl, and I don't want it to.

How can I create a "disallow" line to do this? For example, if I wanted to disallow an entire directory, I'd put this:

Disallow: /DirectoryA

But I want to disallow everything that includes the phrase "unnecessaryPage" anywhere in the filename, directory, or URL. How would I make that Disallow line?

3
  • Are these links originating from outside? Or is it possible that there is a bug in your code and some of your pages are generating this url as a link? There are many website checking sites that will rule-out/confirm the latter. One such one is ahrefs.com. I am not affiliated with it, but I do use it. Sep 10 at 0:16
  • These links are actually in my code now that I've looked at them again, but they're paramater style links with "?"and parameters passed after the question mark. So, it's not a bug, but I don't want google trying to comb thousands of copies of the same page.
    – ravl13
    Sep 14 at 20:19
  • Adding a query string to the URL makes a new URL. Google has to crawl all the different query strings it finds because it doesn't know whether or not there's going to be different content when the query string is different. Sep 14 at 21:58

2 Answers 2

4

The directive would be something like this:

user-agent: *
disallow: /*unnecessaryPage

Thus, anything that comes before the phrase "unnecessaryPage" will be included via wildcard. No trailing wildcard: it is optional because it is ignored. Further examples can be found in Google's documentation on the matter.

2
  • 2
    I'd add the caveat at any time you use wild cards in a disallow rule, that rule will only be understood by major search engine bots including googlebot. Most bots are not advanced enough to understand the wild card syntax. Sep 13 at 21:45
  • 1
    Thanks, this matches up with an article I found over the weekend: geoffkenyon.com/how-to-use-wildcards-robots-txt Down at the bottom they also link to a useful syntax tester tool from google.
    – ravl13
    Sep 14 at 20:18
2

In my view the accepted answer is insufficient. The robots rule will stop Google from crawling these URLs. But it will not stop it from indexing them.

To stop Google from indexing them,

  1. they should either produce a 404, or
  2. they should not exist in any other page or
  3. the links must have noindex.

Even then, if an external site links to them, then they may still get indexed

4
  • Thanks for the clarification.
    – ravl13
    Sep 14 at 21:03
  • 1
    Google won't usually index URLs that are blocked in robots.txt. It sometimes does if there are enough external links pointing to them. If you're using query parameters for tracking there can be enough variation that having them all crawled could be problematic for crawl budget. If there are a small number of these URLs and the goal is to not have them indexed, then I agree with you that allowing them to be crawled but then using a no index tag would be better. Sep 14 at 22:03
  • @StephenOstermiller - I didnt intend to imply that robots entry was unnecessary. I meant it was insufficient. In this case, Google has already indexed them. Sep 14 at 22:10
  • 1
    If you use a noindex tag, then you actually need to remove the disallow from robots.txt or Googlebot will never crawl the URLs and see the tag. It is an either/or proposition. Using both is the same as just using robots.txt. Sep 14 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.