28

I have a few dynamic generated URLs.

Can I use regex to block these URLs in a robots.txt file?

3
  • It would also be worth setting robots meta tags on the pages you don't want crawled/indexed. Dec 8, 2014 at 10:59
  • @AndrewLott In my case I have more then 500 page, so I thought use regex in robots.txt.. Dec 8, 2014 at 12:57
  • Then a rule in your site code is probably more helpful. Dec 8, 2014 at 13:32

2 Answers 2

29

Regular Expressions are not valid in robots.txt, but Google, Bing and some other bots do recognise some pattern matching.

Say if you wanted to block all URLs that have a example any where in the URL, you can use a wild card entry *

User-agent: *
Disallow: /*example

You can also use the dollar sign $ to specify that the URLs must end that way. So if you wanted to block all URLs that end with example, but not URLs that had aexample elsewhere in the URL you could use:

User-agent: *
Disallow: /*example$

More in-depth info for Google can be found here: Robots.txt Specifications, Bing here: How to Create a Robots.txt file and there is a an interactive guide on Moz here

4
  • Perfect, * is working fine, Tested in web master tool.. Thanks Max...:) Dec 8, 2014 at 5:39
  • I'd caution from using too fancy directives in your robots.txt file; these are really, really hard to debug later on. Try to keep things as simple as possible. Also remember that robots.txt is case-sensitive, so you may need to add alternate versions of the directives depending on your site. Dec 10, 2014 at 12:01
  • 3
    would be nice if regex was supported Apr 14, 2016 at 12:55
  • I tested your example and googlebot doesnt recognize that,i made example.php and put some code in it,also wrote regex as in your example and when i tested url in GSC report dont tell me than url is blocked by robots,txt. Jul 22, 2022 at 16:54
3

For blocking urls by regular expressions not from crawling (downloading) but from indexing in search engines you can use

https://developers.google.com/search/reference/robots_meta_tag

dynamic adding of HTTP header X-Robots-Tag into mod_rewrite rules of Apache (or nginx) that widely supports regular expressions.

1
  • Isnt usefull when we have spam url-s on hacked website. Jul 22, 2022 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.