Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of query variables in a long url), and then index that URL?

share|improve this question
add comment

1 Answer 1

They can follow this type of filters but not if they are forms. As for AJAX, they are mostly not followed but it is possible though rarely by accident!

However if those are links, it can generate an excessive and nearly infinite number of URLs which may confuse crawlers and prevent important pages from being indexed because crawlers would see more URLs to crawl.

Google Webmaster Tools lets you specify how each parameter they encountered is used. This is a good place to see which ones they already crawled. Look under Configuration -> URL Parameters.

The option of interest to you is: Yes: Changes, reorders or narrows page content which you can specify for each of your filter parameters. Then you can tell Google to decide what to do, to index very variation, only one or none. For example, you can let them index only one sort order.

share|improve this answer
    
Thanks alot. Looking at the webmaster parameters is actually what got me interested in this, since it showed some params in urls that i didnt explicitly link to, so I'm trying to figure out how it was indexed –  user21100 May 29 '13 at 19:39
    
@user21100 - Indeed and even after 8 years of building websites and 2 years working for Google, I still have no idea how so many URLs not linked from anywhere appear so often! –  Itai May 30 '13 at 12:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.