Remove large amounts of pages from Bing Index Explorer

Question

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to Google and Bing and went on with life.

I have noticed, however BingBot (which I confirmed through the IP) has been slamming my server trying to continue to crawl the old ASP and HTML URLs. My server appears to respond with a 404 (checked via wget, direct web page hit, and through Bing itself) but Bing keeps trying to crawl it. I cannot figure out how to remove the pages from the Index Explorer. I can see all the "bad" ones there under various folders (e.g. /html, /v2, /g...whatever).

How can I get rid of all of these old pages permanently? None of these are in my sitemap (which I did submit) and none of these have been on my server for six months now.

Answer 1

You can't make search engines forget about old URLs on your site. Once search engines find content at a URL, they will continue to periodically crawl that URL indefinitely, even after you remove the content change the URL to show a 404 error.

Instead of returning a 404 error you could change to a more relevant status:

• 301 Permanent Redirect - For the case in which there is a new page that replaces the old page.
• 410 Gone - For the case that the removed page has no replacement.

Using those statuses can make search engine crawlers back off some, especially over time. However I have old URLs on my site that have been redirecting for 20 years, and Googlebot still comes and checks on them occasionally.

You say your server is getting "slammed" by these requests because they have to go through WordPress. You can add rules to .htaccess to short circuit these URLs and serve them much faster. You can usually serve tens or hundreds of 404 requests for the same cost as one real page this way. These rules should go at the top of your .htaccess file, before any rules for WordPress.

RewriteRule ^/?html/ - [G]
RewriteRule ^/?v2/ - [G]
RewriteRule ^/?g/ - [G]
RewriteRule \.asp$ - [G]
RewriteRule \.html$ - [G]

The first three rules would show a 410 Gone for everything in the folders you mention. The last two rules would showe 410 Gone by extension on the URL. You should only use them if there are no more .asp or .html URLs left on your site.

If you can't get the rare case that it might be harming your server, you could use block those URLs in robots.txt.

Disallow: /html
Disallow: /v2
Disallow: /g
Disallow: *.asp
Disallow: *.html

Both Googlebot and Bingbot understand and honor wildcard rules, but most bots don't. The last two rules will only help for specific bots that support advanced rules.

I would only block the URLs with robots.txt if the requests cause performance problems even after trying the rewrite rules. If you disallow the URLs, search engines may still index the URLs and send some visitors to them. See Why do Google search results include pages disallowed in robots.txt? If you allow the URLs to be crawled, the bots will continue to hit the URLs, but search engines will not include them in their search indexes.