Search engines crawling non-existent subdomains

Question

A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev added multiple subdomains to the link farm (e.g., foo.example.com, bar.example.com, foobar.example.com) Before this developer left, he claims to have removed all our links from the link farm. However, to this day, we still get search engines crawling sub-domains that do not exists. I should add that the developer added many, many, subdomain links.

What can I do to prevent these subdomains from getting crawled?

Answer 1

If the links still exist on other sites then Google will continue to look for them on your site, there isn't much you can do to stop it if you can't remove the links.

The only thing that you can do is add the code below to your htaccess file, this 410 gone message will let Google know the content is gone for good. Google may not stop looking but at least it will stop generating 404's each time it does look.

#Stuff to 410

Redirect gone /path/to/page.html
Redirect gone /directory-path/
Redirect gone foo.domain.com