So I have a website with URL's like this:


I've now cleaned them up with mod_rewrite into this:


I've fetched and re-indexed my website after the change.

What is the best way to make the old dirty ones disappear from search results and keep the clean ones? Is blocking profile.php with robots.txt enough?

2 Answers 2


Don't block the old URLs, or they won't work for people or engines that try to access them.

If you can, get the old URLs to return a 301 Redirect to the new URLs. A redirect will this ensure that any existing bookmarks or links to the old URLs will continue to work, including search results. More importantly, a 301 redirect specifically tells search engines understand that the old URLs have been permanently replaced with the new ones, and should also help SEO by transferring most of the "linkjuice" the old pages had.

Regardless, it will take time for search engines to recrawl your site. I would expect at least a month before the old ones disappear from search results, possibly more. However, if you have a redirect, it won't matter that much while you're waiting, and if you send a 301 redirect it should help things update faster.

However you can try to get Google to recrawl your site faster by going to Health > Fetch as Google and inputting a URL, then clicking "Submit to Index" followed by "URL and all linked pages," which presents the following message:

Select if your site has changed significantly. Google will use this URL as a starting point in indexing your site content. Google doesn't guarantee to index all pages on your site.

You should also specify canonical URLs on the new pages to further enforce their new status and reduce any potential duplication issues.

  • Thank you. I created a function that checks if $_SERVER['REQUEST_URI'] contains profile.php, if it does it will do a 301 redirect to the new, clean URL.
    – artsi
    Jun 1, 2012 at 19:07

Blocking profile.php with robots.txt would be bad for several reasons. You should setup 301 redirects - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93633

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.