1

A few years ago I had couple of sites. Then I closed them. Back then, the search engines had added those sites of mine into their databases. Specifically Google and Chinese Baidu know about them. Now their bots query my DNS server for non-existent sites.

The complication is that now I've got a working website. Let's call it example.com. Those non-existent sites are subdomains like forum.example.com.

The traffic is not significant. But why to have it on a daily basis?

As far as I see it, there are two ways to possibly deal with it:

  1. An administrative way, ie., to contact somehow the search engines and ask them nicely to remove those sites from their databases
  2. A technical way, ie., to deal with it myself with some BIND9 configuration.

But I just don't know how to approach the issue, frankly.

migrated from serverfault.com Feb 4 '16 at 3:59

This question came from our site for system and network administrators.

  • 2
    If the sites in question are sub-domains that are no longer used, just simply delete them from your DNS if you have not already. Your DNS will be queried from time to time, however, your web server will not be. In time, it will all stop. It is nearly impossible to make contact with a search engine that will get you anywhere. – closetnoc Feb 4 '16 at 4:42
2

You could set up a single page on the subdomain with a message to users that the subdomain is no longer active, and they should access www.example.com instead. Provide a link for users to click through to the main site, and include <meta name="robots" content="noindex"> in the header for search engines to know that it's no longer available.

This option is the most helpful to both users and search engines.

2

If you have access to those sites, add a hard 404-header. It won't get more obvious than that. The meta's (noindex and nofollow) still indicate that some content exists, you just don;t want it indexed (right now as far as they care).

If those sites dont exists, use a htaccess and some CNAMES to do that same trick. (CNAME forum.site.com as site, and htaccess it to 404).

  • It sounds like the problem isn't that the site is indexed, search engine bots still try to visit it occasionally. – Stephen Ostermiller Feb 5 '16 at 15:09
1

Google never forgets a site that it had once indexed. Googlebot will continue to try to visit your old sites for years. There is no way that I know of to get the bots to totally stop trying to visit the old URLs.

If the sites are truly down, Google won't include them in the search index. You won't get real users visiting those sites from search engines. You will only see occasional hits from the search engine bots trying to see if your site is back up yet.

It sounds like you have token the DNS entries for the sub-domains away. The only activity you will see is occasional DNS queries for the subdomains. This is the least amount of activity from search engine bots that you can hope for.

If you want to serve an error page, use the "410 Gone" HTTP status for it. That status makes it explicit that you removed the content. With that status, search engine bots will come back less often than with other statuses such as "404 Not Found." (They will still come check occasionally, though.)

0

If you still have control on your websites before, just add this on the head

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

That will instruct the Search Engine spiders to stop indexing your website and stop following links. Those entries will be flushed from their database. Google on the other hand uses more than 1 database and it may take around 7 days before all copies of the database gets purged.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.