1

I have the following website:

enter image description here

However somewhere along the way Google seems to have decided to index my old development URL which doesn't exist anymore debtfreeme.223.165.77.73.sth.nz and I cannot for the life of me workout how to get Google to remove it from their search results. Any tips?

  • 4
    Sign into Google Search Console and request for the URL to be removed using their remove outdated content tool. – dan May 31 '17 at 4:46
  • Okay. Much clearer. Can you put a noindex tag on the blank page? I am concerned that Google will just reindex the page again later. Cheers!! – closetnoc May 31 '17 at 6:21
  • @dan the tool you have mention is perfect answer for OP question. – Goyllo May 31 '17 at 8:09
  • "old development URL which doesn't exist anymore" - Whilst that URL returns an empty response body, it's still returning a 200 OK status, rather than a 404 (or preferably 410) - so in some ways that URL does still exist. – MrWhite May 31 '17 at 13:36
  • Google sees empty response bodies as a "soft 404". An empty response body with 200 status isn't ideal, but it should be good enough to get Google to drop it. – Stephen Ostermiller May 31 '17 at 19:55
1

Yes blocking any URL with robots.txt is a way, but you can also create a Google Search Console property for development copy of your website and remove the URL permanently.

|improve this answer|||||
  • 1
    There are those that argue that the URL removal request is not permanent at all and that if the page is not otherwise restricted from search through use of robots.txt, noindx, or anyother recognized mechanism, then the page can be rediscovered and indexed again. This makes sense. And I have seen this happen. Removing a URL is a perfectly valid step providing ordinary means to restrict indexing has already occurred. – closetnoc May 31 '17 at 15:56
2

You can ask Google not to crawl in couple ways:

Method 1: Using robots.txt file. Create a robots.txt with code below and place in your dev server root folder.

User-agent: *

Disallow: /

Method 2:

The second method, just restrict the access using username and password. If you are using cPanel hosting, you can easily do this, and same in other hosting too. Meaning when you will fetch your dev server it should ask you to login.

|improve this answer|||||
  • One problem. This is likely a correct URL in one manner of speaking where the domain name either uses a CNAME or an A record to access. This URL comes from the host. Blocking access to this URL the way you suggest will also block access via the domain name. Instead, using .Htaccess to redirect any access via the incorrect URL to the correct URL should do the trick. Cheers!! – closetnoc May 31 '17 at 3:11
  • Sorry, I realised I could have been more specific. That development URL doesn't have a website attached to it- meaning no place for an .htaccess file. I just want the dev URL removed from google- and quite frankly, im suprised its even showing up given that browsing to the URL gives a completely blank page. – warm__tape May 31 '17 at 3:13
  • Blocking with robots.txt or using authentication are methods that should have been adopted at the beginning, they aren't necessarily the best or quickest methods for getting a URL removed from the SERPs. – MrWhite May 31 '17 at 13:27
  • @user82217 yeah i'm well aware of the process- ive got ~ 15 other dev sites set up exactly the same with no issues. Just strange that this one got there in the first place, and refuses to drop. – warm__tape May 31 '17 at 21:00
1

I've seen this happen many times in the past. Based on the limited information provided, it seems as though your issue is not with the dev url, but rather with your old dev hosting account. You see - google would not be indexing the pages if they did not exist on the server.

  1. You could always delete them, but that would prevent future use of your dev server if you want to rework some functionality & then go live.

  2. This could have been avoided altogether by using a noindex,nofollow meta tag so that google would have never indexed your dev site in the first place. The tag is a meta tag on your root url, along with every other page on your site. This setup appears as follows:

However, it is of the utmost importance that you remember to remove the noindex,nofollow tag prior to going live with your dev site - otherwise you will risk deindexing your main url in Google Search, which is not a fun experience.

lastly, I suggest trying to locate the server, and using an .htaccess (if on apache) to prevent indexation and crawling of the dev website. The code would look as follows, and you can simply copy/paste it into the .htaccess (if you don't have an .htaccess, then make one):

Header set X-Robots-Tag "noindex, nofollow"

With the above code implemented, the dev pages will no longer appear in search results, and your problem, from my limited understanding, will be solved.

|improve this answer|||||
  • Thanks for your input Josh- however please understand that I have already deleted the entire dev website as I no longer need it. I'm well aware of noindex/nofollow, and had this set for the dev site while it was up, adding to the mystery of why it is showing in search results at all. – warm__tape May 31 '17 at 20:53
  • Are you using Google Chrome as your browser? I think that I can help you If I have a little more info. – Josh Salganik Jun 2 '17 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.