I'm creating a robots.txt file for my website which now has a lot of pages.
I'm disallowing all the pages that I do not require to crawl by so:
I've quite a lot of pages in my website. And there are some that uses query strings with them.
How can I specify optional existence of query string to disallow the robot from crawling?
I've tried this
This will disallow file.aspx with any query string parameters. But will it disallow file_with_no_query_string.aspx if there's a file which is not expected to have query string parameters and I write
Cutting down short: Specifying "?*" , will it always want a query string?