Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a website that whereby the first 185 pages are sample profiles for demonstration purpose:


I want to block these pages from Google as they are somewhat similar in content to avoid penalty for being labelled as duplicate content. Is there a better way to do it than listing them out in robots.txt like so:

User-agent: *
Disallow: /profile/1
Disallow: /profile/2
Disallow: /profile/3
share|improve this question
There isn't necessarily a "penalty" for duplicate content, it's just that Google is likely to only return at most 1 page (at its discretion) from the pages that it thinks are duplicate. – w3dk Jul 14 '14 at 9:27
I think any SEO penalty will come from poorly named URL's. – DarrylGodden Jul 14 '14 at 11:43
Depending on your knowledge you could create a PHP script to output a text file with all your disallow rules. – Tom Hart Jul 14 '14 at 12:39
The file has to be named robots.txt, not robot.txt. – unor Jul 15 '14 at 11:58
Google recommends not using the robots.txt file as a way of handling duplicate content, so using the robots meta tag (or x-robots-tag HTTP header) is preferred. – John Mueller Jul 16 '14 at 1:19
up vote 3 down vote accepted

It is not possible to use robots.txt (as defined by the original specification) in your case. A line like Disallow: /profile/1 will block all URLs whose paths start with /profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended).

Workaround: Add a character as suffix, for example a /. So your profile URLs would look like profile/1/, /profile/2/, …. Then you could specify Disallow: /profile/1/ etc.

That said, some robots.txt parsers support additional features which are not included in the original robots.txt specification. As you say you want to block the pages for Google, Google gives special meaning to the $ character:

To specify matching the end of a URL, use $

So for Google, you could write Disallow: /profile/1$. But other parsers that don’t support this feature will then index your profiles 1-185 as they only look for URL paths literally starting with /profile/1$.

So when you don’t want to add a suffix (and list all Disallow lines explicitly), or if you don’t want a Google-only solution (without suffix, but still listing Disallow lines explicitly), robots.txt is no solution for you.

Instead, you could use:

  • on the HTTP level: the HTTP header X-Robots-Tag

    X-Robots-Tag: noindex
  • on the HTML level: meta element with the robots name

    <meta name="robots" content="noindex" />

Both ways are supported by Google.

share|improve this answer

You could put the robots meta tag in all of those pages: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

share|improve this answer
Why "nofollow"? If all relevant pages have "noindex" you shouldn't prevent crawling of all links on those pages. – DisgruntledGoat Jul 14 '14 at 13:40
Not exactly what you proposed, I am using noindex on HTTP headers instead. – Question Overflow Jul 15 '14 at 1:05

You are creating a file to be read by a robot, so create it with a robot:

<?php ob_start(); ?>

User-agent: *

  $limit = 185;

  for($i = 1; $i < $limit ; $i++) 
    echo "Disallow: /profile/$i\n";
 # rest of robots.txt here

Or if you are using leading zeros (better sorting) replace the echo line with:

  printf("Disallow: /profile/%03d\n", $i);

Of course, robots.php doesn't work, but that's what mod_rewrite is for: In .htaccess:

RewriteRule robots.txt robots.php [L]
share|improve this answer
Thanks, but that's not what I am asking and neither is it a better solution than what I have originally. – Question Overflow Jul 15 '14 at 1:06

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.