5

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch?

Context: After replacing a website done in CMS-x with a new one done in CMS-y, 99% of pages/links/resources will be gone or moved to different locations, and even though there are proper 404/410 redirects in place, it still would be better if any bots indexing the website would not try to access old stuff.

Basically this: How to tell google a blog article has been updated? but site wide

Improve this question
5
  • 2
    I'd also recommend you read support.google.com/webmasters/answer/1663419?hl=en. It explains "If you recently changed your site and now have some outdated URLs in the index, Google's crawlers will see this as we recrawl your URLs, and those pages will naturally drop out of our search results. There's no need to request an urgent update."
    – Trebor
    Commented Jan 22, 2019 at 16:39
  • 1
    404/410 are not "redirects." They are errors, which will grossly inconvenience your users.
    – Kevin
    Commented Jan 22, 2019 at 22:00
  • @Kevin: You mean you haven't put a redirect on a 404? It has the amusing property of inconveniencing the bots but users don't notice. (In the old days a 404 error redirected to the 404 page; so redirects on 404 were honored.)
    – Joshua
    Commented Jan 22, 2019 at 22:04
  • @Joshua My 404 Redirect to a proper 404 page with a human-readable explanation and a search function
    – Pit
    Commented Jan 23, 2019 at 6:34
  • @Trebor unfortunately there are more bots. Google alone would not be a problem, I already submitted the new sitemap to them
    – Pit
    Commented Jan 23, 2019 at 6:36

1 Answer 1

Reset to default
6

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.

Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects

If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. Using "410 Gone" instead gets Google to drop the URLs from the index as soon as they are crawled without the 24 hour grace period that Google uses for "404 Not Found."

There is no directive to tell bots to forget about an old site either in search console or robots.txt.

What if you don't redirect?

It may be too much work to redirect, or your new CMS may not make the redirect implementation easy.

If you choose not to implement the redirects it will be something like starting over. Google will see that your old URLs return 404 status and it will remove them from the search index.

Your new URLs will eventually get indexed, but it may take a while. Changing all your URLs without redirects is a big sign that your site isn't stable and can't be trusted. All your rankings will be lost and your site will start over.

Googlebot will continue to crawl the old URLs for years. For it, hope springs eternal that you may someday put those pages back up.

If you do redirect, all your inbound links, users' bookmarks, and most of your current rankings will be preserved.

Why?

So why don't search engines have a "reset" button? Because there are almost always better options. In your case it is much better to redirect.

In the case when a site is penalized, Google doesn't provide a reset button because that might remove all penalties.

How?

So how do you implement the redirects? You need a list of your old URLs. You may have a sitemap from your old site that you can start with. You can also get the list from your server logs, Google Analytics, or even from Google Search console.

If you planned ahead, your URLs in your new CMS will be similar and you can implement a rewrite rule to handle them. If there is a pattern between the old and new URLs, it can be a one liner in a .htaccess file to issue the redirects for the entire site.

If you have to manually find the new URLs and map thousands of them one by one, you could look into RewriteMap functionality.

Improve this answer
2
  • 1
    Thanks. As it is a complete redesign, with most of the old stuff gone an only a few old pages being somehow represented in new one, I will redirect those, and for the rest have to rely on google doing it's business …
    – Pit
    Commented Jan 22, 2019 at 16:29
  • 1
    While the codes don't work well for Google, search and SEO, there may be cases where you want to use "303 See Other" and "300 Multiple Choice" redirects.
    – Stephen Ostermiller ♦
    Commented Jan 22, 2019 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.