Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. Google currently has all clean URLs indexed, along with a few unclean URLs too.

So the site still has linked everywhere the old URLs (ideally this wouldn't be the case but this is how it is ATM).

I would like to block the unclean URLs with robots.txt.

The question: if I block these unclean URLs with the robots.txt, when the entire website is linked with them (but they all redirect to the clean version), will this affect the indexing status at all?

share|improve this question

3 Answers 3

If you disallow the unclean URLs in robots.txt, polite bots will no longer visit these URLs. So they will never notice that you 301-redirect them to other URLs that they’d be allowed to crawl. Bots that don’t know your clean URLs yet would only be able to visit these pages when they are directly linked to with the clean variant (not the blocked unclean variant).

So you should not block them in robots.txt.

As you are 301-redirecting your unclean URLs to corresponding clean URLs, you don’t need to do anything. Bots will know what to do. If some search engines still have some unclean URLs indexed, it should only be a matter of time until they update their index.

share|improve this answer

There's no to need to block anything in robots.txt these days, simple use rel="canonical" on your pages and you never risk duplicate pages regardless if its accessible via unclean or clean because the canonical will tell Google the preferred.

In regards of the indexed non-clean URLs if you use canonical's these will deindex/update themselves however if you don't have page replacements and you just want to remove them then using both noindex in your head HTML and disallow: /unclean-url/ in your robots.txt (Google recommends both robots.txt and noindex usage).

Also for SEO purposes you should correct those URLs as soon as possible as your losing page juice due through the 301 redirect.

share|improve this answer
    
Ok thanks, but do you think it will affect the site or not? I'm working with a Budget and there are limits to what I can do ATM. Thanks. –  Edgar Quintero Oct 19 '13 at 0:38
1  
How would the Googlebot stumble upon noindex, when it isn’t allowed to crawl the page in the first place? –  unor Oct 19 '13 at 9:44

The redirection won't work on those blocked URLs. After adding 301 redirection you will not need to apply canonical as well. This won't effect the websites I have seen big brands changing the URL and make millions of redirection just like the SEOmoz did after changing its name to Moz.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.