Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string?

Question

I know there are a lot of similar questions and I searched around but I never completely found an answer, so here is my question:

Do search engine robots crawl my dynamically generated URLs? With this I mean html pages generated by php based upon GET variables in the url. The links would look like this:

www.mywebsite.com/view.php?name=something 
www.mywebsite.com/view.php?name=somethingelse 
www.mywebsite.com/view.php?name=something

I have tried crawling my website with a test crawler found here: http://robhammond.co/tools/seo-crawler but it only visits my view page once, with just one variable in the header.

Most of the content on my website is generated by these GET variables from the database so I would really like the search engines to crawl those pages.

Answer 1

Yes. Search engines and any crawler that is coded properly can crawl pages with parameters and index them as separate pages. The fact that the crawler you referenced/used cannot handle this is an indication that I would not use this tool again.

I often recommend using http://www.screamingfrog.co.uk/seo-spider/ if you can access your site directly. This tool allows 500 pages free and is rather affordable if you wish to pay for it. ScreamingFrog is very good at reporting a long list of issues that any webmaster might be concerned with including most if not all of what the link you provided supplies.

If you do use this tool or any tool like this, I suggest turning of AdSense ads or any other ads you are using so that you do not get into trouble.

Answer 2

Googlebot and other search engine robots will crawl URLs with a variety of query strings by default. All they need is a link to each, or a sitemap that contains each. Google considers URLs different even when they differ only by the query string.

Google Webmaster Tools allows you to customize this behavior for your site. If you have a query parameter that doesn't actually change the page, you can tell Google about it in the site settings. Then Google may crawl only one representative URL rather than each value of the parameter.