It is not possible to use robots.txt (as defined by the original specification) in your case. A line like
Disallow: /profile/1 will block all URLs whose paths start with
/profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended).
Workaround: Add a character as suffix, for example a
/. So your profile URLs would look like
/profile/2/, …. Then you could specify
Disallow: /profile/1/ etc.
That said, some robots.txt parsers support additional features which are not included in the original robots.txt specification. As you say you want to block the pages for Google, Google gives special meaning to the
To specify matching the end of a URL, use
So for Google, you could write
Disallow: /profile/1$. But other parsers that don’t support this feature will then index your profiles 1-185 as they only look for URL paths literally starting with
So when you don’t want to add a suffix (and list all
Disallow lines explicitly), or if you don’t want a Google-only solution (without suffix, but still listing
Disallow lines explicitly), robots.txt is no solution for you.
Instead, you could use:
on the HTTP level: the HTTP header
on the HTML level:
meta element with the
<meta name="robots" content="noindex" />
Both ways are supported by Google.