2

Using robots.txt I am attempting to stop all crawling of search URLs

Disallow: /rest_of_url/search&tour*

Above is what I am using. Our URL looks like the following for all search results. However, everything after search&tour can be different, for example:

https://www.example.com.au/rest_of_url/search&tour-sdfs=the-palce+lcation+&tour-duration=1/

Will the Disallow code above stop robots from crawling all of my search result pages?

Improve this question

2 Answers 2

Reset to default
3

Will the Disallow code above stop robots from crawling all of my search result pages?

Yes, it will stop the (good) bots that obey the robots.txt "standard".

However, you don't need the trailing *. robots.txt is prefix matching, so the "wildcard" * at the end can simply be omitted. (Wildcard type matches are an extension of the original standard anyway.)

And you obviously need the User-agent directive that precedes this rule, if you haven't got it already:

User-agent: *
Disallow: /rest_of_url/search&tour
Improve this answer
1
  • I've applied this method because of the structure of my CMS. Thank you - your information was really helpful
    – Trent
    Commented Nov 19, 2015 at 0:12
1

Disallow sets the files or folders that are not allowed to be crawled.

In addition, you can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code. When Googlebot next crawls that page, Googlebot will see the noindex meta tag and will drop that page entirely from Google Search results, regardless of whether other sites link to it.

Improve this answer
1
  • Thank you for your response - this would of been perfect but the structure of my CMS would mean it would effect other pages by adding this to the template
    – Trent
    Commented Nov 19, 2015 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.