Remove dev URL of website from Google Search

Question

I have the following website:

However somewhere along the way Google seems to have decided to index my old development URL which doesn't exist anymore debtfreeme.223.165.77.73.sth.nz and I cannot for the life of me workout how to get Google to remove it from their search results. Any tips?

Answer 1

You can ask Google not to crawl in couple ways:

Method 1: Using robots.txt file. Create a robots.txt with code below and place in your dev server root folder.

User-agent: *

Disallow: /

Method 2:

The second method, just restrict the access using username and password. If you are using cPanel hosting, you can easily do this, and same in other hosting too. Meaning when you will fetch your dev server it should ask you to login.

Answer 2

Yes blocking any URL with robots.txt is a way, but you can also create a Google Search Console property for development copy of your website and remove the URL permanently.

Answer 3

I've seen this happen many times in the past. Based on the limited information provided, it seems as though your issue is not with the dev url, but rather with your old dev hosting account. You see - google would not be indexing the pages if they did not exist on the server.

1. You could always delete them, but that would prevent future use of your dev server if you want to rework some functionality & then go live.

2. This could have been avoided altogether by using a noindex,nofollow meta tag so that google would have never indexed your dev site in the first place. The tag is a meta tag on your root url, along with every other page on your site. This setup appears as follows:

However, it is of the utmost importance that you remember to remove the noindex,nofollow tag prior to going live with your dev site - otherwise you will risk deindexing your main url in Google Search, which is not a fun experience.

lastly, I suggest trying to locate the server, and using an .htaccess (if on apache) to prevent indexation and crawling of the dev website. The code would look as follows, and you can simply copy/paste it into the .htaccess (if you don't have an .htaccess, then make one):

Header set X-Robots-Tag "noindex, nofollow"

With the above code implemented, the dev pages will no longer appear in search results, and your problem, from my limited understanding, will be solved.

Answer 4

According to google, you have a few options:

Quoting the content from google

Return 404 or 401

Remove or update the content on your site (images, pages, directories) and make sure that your web server returns either a 404 (Not Found) or 410 (Gone) HTTP status code. Non-HTML files (like PDFs) should be completely removed from your server.

Block access to the content

for example by requiring a password.

Indicate that the page should not be indexed

using the noindex meta tag. This is less secure than the other methods.

Do not use robots.txt as a blocking mechanism.