2

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to Google and Bing and went on with life.

I have noticed, however BingBot (which I confirmed through the IP) has been slamming my server trying to continue to crawl the old ASP and HTML URLs. My server appears to respond with a 404 (checked via wget, direct web page hit, and through Bing itself) but Bing keeps trying to crawl it. I cannot figure out how to remove the pages from the Index Explorer. I can see all the "bad" ones there under various folders (e.g. /html, /v2, /g...whatever).

How can I get rid of all of these old pages permanently? None of these are in my sitemap (which I did submit) and none of these have been on my server for six months now.

1 Answer 1

1

You can't make search engines forget about old URLs on your site. Once search engines find content at a URL, they will continue to periodically crawl that URL indefinitely, even after you remove the content change the URL to show a 404 error.

Instead of returning a 404 error you could change to a more relevant status:

  • 301 Permanent Redirect - For the case in which there is a new page that replaces the old page.
  • 410 Gone - For the case that the removed page has no replacement.

Using those statuses can make search engine crawlers back off some, especially over time. However I have old URLs on my site that have been redirecting for 20 years, and Googlebot still comes and checks on them occasionally.

You say your server is getting "slammed" by these requests because they have to go through WordPress. You can add rules to .htaccess to short circuit these URLs and serve them much faster. You can usually serve tens or hundreds of 404 requests for the same cost as one real page this way. These rules should go at the top of your .htaccess file, before any rules for WordPress.

RewriteRule ^/?html/ - [G]
RewriteRule ^/?v2/ - [G]
RewriteRule ^/?g/ - [G]
RewriteRule \.asp$ - [G]
RewriteRule \.html$ - [G]

The first three rules would show a 410 Gone for everything in the folders you mention. The last two rules would showe 410 Gone by extension on the URL. You should only use them if there are no more .asp or .html URLs left on your site.

If you can't get the rare case that it might be harming your server, you could use block those URLs in robots.txt.

Disallow: /html
Disallow: /v2
Disallow: /g
Disallow: *.asp
Disallow: *.html

Both Googlebot and Bingbot understand and honor wildcard rules, but most bots don't. The last two rules will only help for specific bots that support advanced rules.

I would only block the URLs with robots.txt if the requests cause performance problems even after trying the rewrite rules. If you disallow the URLs, search engines may still index the URLs and send some visitors to them. See Why do Google search results include pages disallowed in robots.txt? If you allow the URLs to be crawled, the bots will continue to hit the URLs, but search engines will not include them in their search indexes.

3
  • Thank you for the info. The 404 "weight" seems to be a common thing among WordPress sites. I don't know if the stack to get the 404 back is too large or what. But, when many requests are sent in a row where a 404 is returned, it seems to really bog things down. Everything is fast otherwise. (There is a CDN in front of all this, but there are thousands of pages being crawled very fast.) One thing you said is you can't make search engines forget about old pages. But, Google Search Console seems to allow that. I have had no problems there.
    – JasCav
    Mar 30, 2020 at 21:51
  • I'm not what you did in Google Search Console to make Googlebot forget about those URLs. I'm not aware of any features there designed to do that. Mar 30, 2020 at 22:10
  • I added some rewrite rules for you to add that will bypass WordPress. They should make showing errors for those requests very fast. Mar 30, 2020 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.