We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php.

When you search the company on engines such as Google, you get the site and a listing of its sub pages which now lead to outdated pages.

Our plan doesn't allow us to edit the .htaccess and the old pages are .html docs so I cannot use PHP redirects either.

So if I put in place a robots.txt telling the engines to not crawl beyond index.php, how effective will this be in preventing/removing crawled sub pages.

And rough guess, how long before the search engines would update?

2 Answers 2


Having a robots.txt to tell the search engine not to crawl certain pages should work but there are no guarantees because search engines do what they want. You could also try creating a removal request in Webmaster Tools. The delay for search engine updates vary from a day to months and this also depends on the search engine (I have heard Bing is the slowest).


robot.txt is text file which just includes allow and disallow rules for any particular page of website. If you need to remove all pages from Search engine then you need to use Google web master tool. There is not particular time period need to remove website(updation process of search engine) from Google but approx 5 to 15 days needs to remove it from Google.

The important information about you website is that if you have removed website from Google then you can recover it in 30 days.

  • I definitely will do this, but doesn't Google Webmaster Tools only affect Google itself, or am I mistaken?
    – Stefan
    Jul 10, 2012 at 14:02
  • 1
    Yes, Google Webmaster Tools only affects Google's index, not Bing, etc.
    – mgkrebbs
    Jul 10, 2012 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.