Should I differentiate parameters in my URLs?

Question

I think I read somewhere that it was a good practice to have certain "totally dynamic and irrelevant" parameters excluded from the nice URL rewriting techniques. Like here on the StackOverflow galaxy, when you login the URL is /users/login?returnurl=parameter.

The reason would also have been that it kind of "prevents" search engines to follow those links, and could be a good idea for "transition pages" that change often and should not be indexed as is, but still followed, like /news?page=5 where this page would be an index of the latest news, and by nature changes over time as old news are pushed down.

Is there any sense to that? Would you agree, and if so maybe point to some resources? If not, what do you think of it, and what should be done with those pages that keep changing?

Thanks for any thought!

Answer 1

I'd say stop reading wherever you read that drivel. Dynamic pages are a natural part of the web, very common, and you should definitely want them to be indexed by the search engines. The search engine crawlers regularly crawl pages looking for changed content and, if they detect any, update their indices. That's good for you because it means new content that you have will be available for others to find which is what you want.

And search engines can crawl virtually any URL regardless of the structure of that URL. So if that site is implying query strings are not search engine friendly they are greatly mistaken.

Answer 2

Google and Bing both crawl URLs with parameters in them. Using parameters as a way to confuse Google out of following the links will only cause you SEO issues.

If you want full information on it, check out Google Search Engine Optimization Guide.. It will educate you on what Google does and doesn't do and what they suggest you do.

Finally, if you don't want specific pages to be crawled or given PR by Google or Bing just put the files/directories into a robots.txt file like @Lese suggested or put a rel="nofollow" on your links. Those are the only ways to 100% guarantee you will not give PR to the pages you don't want to have PR.

Answer 3

There is really only one rule: If the page content doesn't change when the parameter changes, then tell Google to ignore it.

Your returnurl example is a good one - I assume the page doesn't change at all, so tell Googlebot to ignore that parameters in Webmaster Tools. (Actually Google is pretty good at detecting these anyway.)

But for page - definitely do not ignore this! You need the search engines to parse all your pages to find the links to the content.

Answer 4

Not using clean URLs does not exclude a page from being indexed. To do that, you need to use robots.txt.

And I don't know what you mean by "should not be indexed as is, but followed", but a page that changes often (and has lots of backlinks and can be easily gotten to by Google's crawlers) will automatically be crawled more often by Google, keeping the indexed content fresh. You can also adjust the crawl rate in the Google Webmasters tools.

Answer 5

There are some statements attached to this question that claim "search engines can crawl virtually any URL regardless of the structure of that URL".

Taken from the Google guidelines:

If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.

Here is the official link for further review: Google Web Master Guidelines

I strongly suggest you take this advice into consideration.