3

When I'm checking the Google results for our site's site:example.com, we get lots of index results like

  • xyz.example.com
  • xyz.example.com/abe
  • xyz.example.com/abcds

How do I disallow this type URL, so they don't show up in the search results?

Improve this question

6 Answers 6

Reset to default
2

Just include the following in the robots.txt and upload the file in the root folder of your subdomain you want to block from Search Engines. Verify using robot tester in the Search console tool. 

User-agent: * 
Disallow: /
Improve this answer
1

This is definitely the case when you should use meta tag "robots" and "noindex" the subdomain or what ever pages you want to hide from serps. After that, you could consider using url removal tool from google and getting rid of pages you don't want to see in serps, which will speed up the process a bit.

Do not go with robots.txt and blocking content of this type.

Improve this answer
1

The correct answer to your question is to put a robots.txt file at the root folder of your subdomain or put robots meta tags on your subdomain's index page to block them from crawling and indexing.

Improve this answer
0

It has very simple solution. You need to put the Robots.txt file in the root folder of your sudomain (i.e xyz.example.com).

After that you can use Robots meta tag to disallow SE from crawling your site (i.e xyz.example.com/abe, xyz.example.com/abcds).

Improve this answer
1
  • The Robots meta tag is different from robots.txt. It doesn't disallow crawling, only indexing.
    – Maximillian Laumeister ♦
    Jun 15, 2020 at 17:40
-3

Try:

User-agent: * 
Disallow: xyz.example.com

Or

Disallow: xyz.example.com/abc
Disallow: xyz.example.com/abcd
Disallow: xyz.example.com/abcde
Improve this answer
3
  • 1
    That isn't correct. Robots.txt never has host names in it.
    – Stephen Ostermiller ♦
    Jul 24, 2019 at 0:21
  • I'm not talking about robots.txt for exemple.com u need to add this in robots.txt xyz.exemple. com directory
    – ben teber
    Jul 24, 2019 at 20:26
  • 1
    Yes, you need to add the robots.txt in the directory root for the subdomain. You should edit your answer to say that. You should also edit out the "xyz.example.com" because those are not correct.
    – Stephen Ostermiller ♦
    Jul 24, 2019 at 22:19
-3

When you have to execute your file, you allow it and you do not have to.

User-agent: * 
Disallow: xyz.example.com
Disallow: /wp-admin/
Improve this answer
2
  • What is Disallow: xyz.example.com supposed to do? You can't have host names in a robots.txt file. If you are intending that to disallow a directory, that won't work either because directories start with a slash.
    – Stephen Ostermiller ♦
    Jul 29, 2019 at 14:45
  • This answer is very incorrect.
    – Maximillian Laumeister ♦
    Jun 15, 2020 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.