I have a website that whereby the first 185 pages are sample profiles for demonstration purpose:


I want to block these pages from Google as they are somewhat similar in content to avoid penalty for being labelled as duplicate content. Is there a better way to do it than listing them out in robots.txt like so:

User-agent: *
Disallow: /profile/1
Disallow: /profile/2
Disallow: /profile/3
  • 1
    There isn't necessarily a "penalty" for duplicate content, it's just that Google is likely to only return at most 1 page (at its discretion) from the pages that it thinks are duplicate.
    – MrWhite
    Jul 14, 2014 at 9:27
  • I think any SEO penalty will come from poorly named URL's. Jul 14, 2014 at 11:43
  • Depending on your knowledge you could create a PHP script to output a text file with all your disallow rules.
    – TMH
    Jul 14, 2014 at 12:39
  • 1
    The file has to be named robots.txt, not robot.txt.
    – unor
    Jul 15, 2014 at 11:58
  • 2
    Google recommends not using the robots.txt file as a way of handling duplicate content, so using the robots meta tag (or x-robots-tag HTTP header) is preferred. Jul 16, 2014 at 1:19

3 Answers 3


You could put the robots meta tag in all of those pages: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

  • 2
    Why "nofollow"? If all relevant pages have "noindex" you shouldn't prevent crawling of all links on those pages. Jul 14, 2014 at 13:40
  • Not exactly what you proposed, I am using noindex on HTTP headers instead. Jul 15, 2014 at 1:05

It is not possible to use robots.txt (as defined by the original specification) in your case. A line like Disallow: /profile/1 will block all URLs whose paths start with /profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended).

Workaround: Add a character as suffix, for example a /. So your profile URLs would look like profile/1/, /profile/2/, …. Then you could specify Disallow: /profile/1/ etc.

That said, some robots.txt parsers support additional features which are not included in the original robots.txt specification. As you say you want to block the pages for Google, Google gives special meaning to the $ character:

To specify matching the end of a URL, use $

So for Google, you could write Disallow: /profile/1$. But other parsers that don’t support this feature will then index your profiles 1-185 as they only look for URL paths literally starting with /profile/1$.

So when you don’t want to add a suffix (and list all Disallow lines explicitly), or if you don’t want a Google-only solution (without suffix, but still listing Disallow lines explicitly), robots.txt is no solution for you.

Instead, you could use:

  • on the HTTP level: the HTTP header X-Robots-Tag

    X-Robots-Tag: noindex
  • on the HTML level: meta element with the robots name

    <meta name="robots" content="noindex" />

Both ways are supported by Google.


You are creating a file to be read by a robot, so create it with a robot:

<?php ob_start(); ?>

User-agent: *

  $limit = 185;

  for($i = 1; $i < $limit ; $i++) 
    echo "Disallow: /profile/$i\n";
 # rest of robots.txt here

Or if you are using leading zeros (better sorting) replace the echo line with:

  printf("Disallow: /profile/%03d\n", $i);

Of course, robots.php doesn't work, but that's what mod_rewrite is for: In .htaccess:

RewriteRule robots.txt robots.php [L]
  • Thanks, but that's not what I am asking and neither is it a better solution than what I have originally. Jul 15, 2014 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.