2

I have a directory that I don't want Google to index at the top level. I currently do the following:

Disallow: /profiles/

This stops Google from indexing https://example.com/profiles/, but it also stops it from indexing https://example.com/profiles/exampleuser. Because new profiles are created every day, it is not feasible to simply add an allow statement for every individual subdirectory of /profiles/

How can I disallow the profiles directory itself while also allowing any subdirectory of the profiles directory?

Improve this question

2 Answers 2

Reset to default
1

Edit: If you are trying to prevent Google from indexing a page, you should not be using robots.txt at all. In that case, see Stephen's answer.

I will leave my original answer below, which shows how to set up your robots.txt file in the way you're asking for. This will prevent crawling of your desired URL pattern, not necessarily indexing.

You could use the $ operator, which matches URLs that end in a specific string:

Disallow: /profiles/$

You might also consider blocking versions of the page that come with parameters:

Disallow: /profiles/?

Consider using Google's robots.txt testing tool to test your setup, available here:
https://www.google.com/webmasters/tools/robots-testing-tool

Improve this answer
1

Robots.txt does not prevent indexing, it only prevents crawling. Google may decide to index a URL that is blocked in robots.txt when there are enough links to it, especially external links.

Robots.txt is the right tool when there are too many URLs to crawl. It is not the right tool for when you want to prevent a single URL from getting indexed.

I would suggest using a <meta name=robots content=noindex> tag in the <head> of https://example.com/profiles/ without blocking the page from getting crawled. That way Googlebot will be able to crawl the page and see that it shouldn't be indexed.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.