I have a website that whereby the first 185 pages are sample profiles for demonstration purpose:


I want to block these pages from Google as they are somewhat similar in content to avoid penalty for being labelled as duplicate content. Is there a better way to do it than listing them out in robots.txt like so:

User-agent: *
Disallow: /profile/1
Disallow: /profile/2
Disallow: /profile/3
  • 1
    There isn't necessarily a "penalty" for duplicate content, it's just that Google is likely to only return at most 1 page (at its discretion) from the pages that it thinks are duplicate. – MrWhite Jul 14 '14 at 9:27
  • I think any SEO penalty will come from poorly named URL's. – DarrylGodden Jul 14 '14 at 11:43
  • Depending on your knowledge you could create a PHP script to output a text file with all your disallow rules. – TMH Jul 14 '14 at 12:39
  • 1
    The file has to be named robots.txt, not robot.txt. – unor Jul 15 '14 at 11:58
  • 2
    Google recommends not using the robots.txt file as a way of handling duplicate content, so using the robots meta tag (or x-robots-tag HTTP header) is preferred. – John Mueller Jul 16 '14 at 1:19

It is not possible to use robots.txt (as defined by the original specification) in your case. A line like Disallow: /profile/1 will block all URLs whose paths start with /profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended).

Workaround: Add a character as suffix, for example a /. So your profile URLs would look like profile/1/, /profile/2/, …. Then you could specify Disallow: /profile/1/ etc.

That said, some robots.txt parsers support additional features which are not included in the original robots.txt specification. As you say you want to block the pages for Google, Google gives special meaning to the $ character:

To specify matching the end of a URL, use $

So for Google, you could write Disallow: /profile/1$. But other parsers that don’t support this feature will then index your profiles 1-185 as they only look for URL paths literally starting with /profile/1$.

So when you don’t want to add a suffix (and list all Disallow lines explicitly), or if you don’t want a Google-only solution (without suffix, but still listing Disallow lines explicitly), robots.txt is no solution for you.

Instead, you could use:

  • on the HTTP level: the HTTP header X-Robots-Tag

    X-Robots-Tag: noindex
  • on the HTML level: meta element with the robots name

    <meta name="robots" content="noindex" />

Both ways are supported by Google.

| improve this answer | |

You could put the robots meta tag in all of those pages: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

| improve this answer | |
  • 2
    Why "nofollow"? If all relevant pages have "noindex" you shouldn't prevent crawling of all links on those pages. – DisgruntledGoat Jul 14 '14 at 13:40
  • Not exactly what you proposed, I am using noindex on HTTP headers instead. – Question Overflow Jul 15 '14 at 1:05

You are creating a file to be read by a robot, so create it with a robot:

<?php ob_start(); ?>

User-agent: *

  $limit = 185;

  for($i = 1; $i < $limit ; $i++) 
    echo "Disallow: /profile/$i\n";
 # rest of robots.txt here

Or if you are using leading zeros (better sorting) replace the echo line with:

  printf("Disallow: /profile/%03d\n", $i);

Of course, robots.php doesn't work, but that's what mod_rewrite is for: In .htaccess:

RewriteRule robots.txt robots.php [L]
| improve this answer | |
  • Thanks, but that's not what I am asking and neither is it a better solution than what I have originally. – Question Overflow Jul 15 '14 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.