I've seen basic articles online about allowing/disallowing certain subdirectories in
robots.txt, but I have a more detailed need than that.
Google search console keeps trying to discover/crawl certain URLs on my site that it shouldn't be doing. It's not a security hazard, but essentially what it's crawling are completely redundant URLs that don't exist in actuality. I have an idea of why it's happening, but I don't want to try and restructure my pages to accommodate Google in this regard.
These unnecessary page crawls always contain a certain phrase in the URL. Suppose this phrase is "unnecessaryPage", as in
info-unnecessaryPage.htm is a URL that Google attempts to crawl, and I don't want it to.
How can I create a "disallow" line to do this? For example, if I wanted to disallow an entire directory, I'd put this:
But I want to disallow everything that includes the phrase "unnecessaryPage" anywhere in the filename, directory, or URL. How would I make that Disallow line?