2

When building websites I always do this on a test domain. This domain was subdomain of the main one. Now I have an issue. I forgot to make sure that bots wouldn't crawl the web page while developing.

So after finding out, I removed the page and the entire domain. Note that this was months ago. But now when I go on google and enter site:example.com I still get instances of the removed subdomain. This is despite that the website no longer exists. The domain doesn't exist either.

How is it still being found and how long does google take to realize the website doesnt exist anymore?

| improve this question | |
  • What type of error do you get when you try to load pages from that domain? Is it a DNS resolution error, or a 404 page not found errer, or something else? – Stephen Ostermiller♦ Sep 3 '19 at 9:58
  • I get a Web Server's Default Page This page is generated by Plesk, the leading hosting automation software. You see this page because there is no Web site at this address. @StephenOstermiller – Kevin.a Sep 3 '19 at 10:17
  • Have you removed sub-domain.domain.com from your DNS? If you are hosting your domain names on your registrars site, this is where you want to delete it. If sub-domain.domain.com cannot resolve, this problem should end in time. Otherwise, not knowing Plesk at all, if you were using Apache, I would have you put something in the .htaccess file on the first site created to create a 410 error. It may be the Plesk page is keeping things alive - though I would expect that Google would eventually try and reindex the pages and the problem would be somewhat solved. It can take a long time. – closetnoc Sep 9 '19 at 20:05
1

Google should be removing the development site from its index. I'm surprised that is taking months. The only ways that it should take that long are if:

  • You have thousands of pages. Google does have to crawl each page to realize it is gone. If you have a large number of pages, it may take Googlebot months to get back to each and every page.
  • /robots.txt on that subdomain blocks all crawling, so Googlebot can't return and see that site is gone.
  • The replacement default Plesk page has a 200 OK status rather than a 404 Not Found status. Although, even if it were 200 OK, I would expect Googlebot to identify it as a "Soft 404" page and remove the URLs from the search index.

To help the situation you could use one or two of:

  1. Ensure robots.txt isn't preventing Googlebot from crawling the URLs for the old development subdomain.

  2. Configure your server to issue 301 Permanent redirects to your public site from the old development site. This is probably my favorite solution, because Google will remove the redirect URLs from the index eventually and in the meantime even if users found the dev site in search results, they would get to the right place.

  3. Configure the server to respond with "410 Gone" errors instead of the default Plesk page. This is usually as simple as creating a .htaccess file in the document root for the development subdomain with:

    Redirect 410 /

  4. Remove the subdomain from your DNS entirely. Make sure there is no A, CNAME for the subdomain and make sure there is no wildcard record for all subdomains either. This is probably the quickest route to getting the entire subdomain de-indexed because Google should be able to remove the entire subdomain once the DNS fails, rather than having to crawln each page.

  5. Use the URL Removal Tool in Google Search Console. You can remove the tool to "Temporarily remove" any "URL that starts with a prefix" of the subdomain. That will de-index the URLs for six months. In the meantime Googlebot should have time to see that they don't exist anymore and they won't come back.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.