3

I'm creating a robots.txt file for my website which now has a lot of pages.

I'm disallowing all the pages that I do not require to crawl by so:

Disallow: /folder/file.aspx

I've quite a lot of pages in my website. And there are some that uses query strings with them.

How can I specify optional existence of query string to disallow the robot from crawling?

I've tried this

Disallow: /folder/file.aspx?*

This will disallow file.aspx with any query string parameters. But will it disallow file_with_no_query_string.aspx if there's a file which is not expected to have query string parameters and I write

 Disallow: /folder/file_with_no_query_string.aspx?*

Cutting down short: Specifying "?*" , will it always want a query string?

0

As long as some of the query strings don't need to be crawled you can simply disallow access to the entire file using

Disallow: /folder/file.aspx*

This will disallow crawling both on the file on its own as well as all query strings appended to the end of the file.

  • No need for the (non-standard) * here. – unor Apr 15 '16 at 16:23
1

Disallow: /folder/file.aspx

What you had in the beginning is all that's required - in order to block both /folder/file.apsx and /folder/file.aspx?foo=bar. If in doubt check it with the robots.txt testing tool in Google Search Console.

robots.txt is prefix matching, so there is never any need to include the wildcard (*) at the end of the URL path.

The wildcard char (*) is also an extension to the original "standard", so for maximum compatibility it should be avoided anyway.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.