I have a few dynamic generated URLs.
Can I use regex to block these URLs in a robots.txt file?
Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching.
Say if you wanted to block all URLs that have a
example any where in the URL, you can use a wild card entry *
User-agent: * Disallow: /*example
You can also use the dollar sign $ to specify that the URLs must end that way. So if you wanted to block all URLs that end with
example, but not URLs that had a
example elsewhere in the URL you could use:
User-agent: * Disallow: /*example$