Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I think I read somewhere that it was a good practice to have certain "totally dynamic and irrelevant" parameters excluded from the nice URL rewriting techniques. Like here on the StackOverflow galaxy, when you login the URL is /users/login?returnurl=parameter.

The reason would also have been that it kind of "prevents" search engines to follow those links, and could be a good idea for "transition pages" that change often and should not be indexed as is, but still followed, like /news?page=5 where this page would be an index of the latest news, and by nature changes over time as old news are pushed down.

Is there any sense to that? Would you agree, and if so maybe point to some resources? If not, what do you think of it, and what should be done with those pages that keep changing?

Thanks for any thought!

share|improve this question
add comment

5 Answers

up vote 2 down vote accepted

There is really only one rule: If the page content doesn't change when the parameter changes, then tell Google to ignore it.

Your returnurl example is a good one - I assume the page doesn't change at all, so tell Googlebot to ignore that parameters in Webmaster Tools. (Actually Google is pretty good at detecting these anyway.)

But for page - definitely do not ignore this! You need the search engines to parse all your pages to find the links to the content.

share|improve this answer
    
Thx! I did not think of the link between parameter and Webmaster Tools. Now it all makes sense. –  Damien Oct 15 '10 at 9:21
add comment

I'd say stop reading wherever you read that drivel. Dynamic pages are a natural part of the web, very common, and you should definitely want them to be indexed by the search engines. The search engine crawlers regularly crawl pages looking for changed content and, if they detect any, update their indices. That's good for you because it means new content that you have will be available for others to find which is what you want.

And search engines can crawl virtually any URL regardless of the structure of that URL. So if that site is implying query strings are not search engine friendly they are greatly mistaken.

share|improve this answer
add comment

Google and Bing both crawl URLs with parameters in them. Using parameters as a way to confuse Google out of following the links will only cause you SEO issues.

If you want full information on it, check out Google Search Engine Optimization Guide.. It will educate you on what Google does and doesn't do and what they suggest you do.

Finally, if you don't want specific pages to be crawled or given PR by Google or Bing just put the files/directories into a robots.txt file like @Lese suggested or put a rel="nofollow" on your links. Those are the only ways to 100% guarantee you will not give PR to the pages you don't want to have PR.

share|improve this answer
add comment

Not using clean URLs does not exclude a page from being indexed. To do that, you need to use robots.txt.

And I don't know what you mean by "should not be indexed as is, but followed", but a page that changes often (and has lots of backlinks and can be easily gotten to by Google's crawlers) will automatically be crawled more often by Google, keeping the indexed content fresh. You can also adjust the crawl rate in the Google Webmasters tools.

share|improve this answer
    
It's just that it often happens to me that I see a result on google, lets say from an index of a forum, and then I can't find what I am looking for, either coz the page changed, or is packed with links, whereas what I wish I had gotten was the actual "topic" or page behind the link. So google's index is not "always" up to date and there ain't nothing you can "really" do about it... –  Damien Oct 15 '10 at 9:24
    
@Damien: That's why it's important to have lots of backlinks and a high PageRank. If google crawls your site every 2 months, then your longest "stale" time will be 2 months. But if they crawl your site every 2 days, then it will only very rarely be out of date. And forums are a worst case scenario since 99% of forum posts aren't worth indexing, so naturally Google isn't going to crawl the average forum on a daily basis. But even if you exclude the forum index from Google's index, that won't increase the chances of your threads being indexed. –  Lèse majesté Oct 15 '10 at 13:49
add comment

There are some statements attached to this question that claim "search engines can crawl virtually any URL regardless of the structure of that URL".

Taken from the Google guidelines:

If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.

Here is the official link for further review: Google Web Master Guidelines

I strongly suggest you take this advice into consideration.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.