3

There are two subdomains for my website. The subdomain for the production site is www.example.com. The subdomain for the test site is test.example.com.

test.example.com is showing up in Google search results and needs to be removed. I have added a robot.txt file and set up basic authentication for test.example.com to keep Google from crawling the site in the future.

Is it possible to remove URLS that contain test.example.com without removing www.example.com? Google search console states "All URL variations (www/non-www and http/https) will be affected". Does this mean if I remove https://test.example.com, https://test.example.com/page1.html, https://test.example.com/page2.html then https://www.example.com, https://www.example.com/page1.html, https://www.example.com/page2.html will also be removed?

UPDATE: I decided not to add basic authentication. I'm redirecting any test pages in the Google search to the production site. robot.txt contains the following:

User-agent: * Disallow: /

It is my understanding that this can take weeks to remove the sight from Google search. Is there a way to make this happen sooner without affecting SEO on the production site?

Improve this question
4
  • Is the robots.txt behind the basic authentication? If so, it isn't doing anything (which is probably a good thing.) If not, it is preventing Googlebot from crawling the URLs on the test subdomain and seeing that they require authentication now.
    – Stephen Ostermiller ♦
    Commented May 17, 2023 at 18:51
  • 1
    I decided not top add basic authentication. I'm redirecting any test pages in the google search to the production site. Robot.txt contains the following: I decided not top add basic authentication. I'm redirecting any test pages in the google search to the production site. Robot.txt contains the following: User-agent: * Disallow: / It is my understanding that this can take weeks to remove the sight from google search. Is there a way to make this happen sooner without effecting SEO on the production site?
    – Anthony
    Commented May 18, 2023 at 14:15
  • 2
    If you are redirecting, users who click from search will get to the right place, so I wouldn't worry too much about it. Again, your robots.txt file is preventing Googlebot from seeing your redirects, so it should probably be removed.
    – Stephen Ostermiller ♦
    Commented May 18, 2023 at 15:10
  • 2
    For your comment question "Is there a way to make this happen sooner". I'm new to using GSC, but have you tried the "Removals" feature (in the "Indexing" group)? This is what I did recently for subdomains. I'm still wondering how the subdomains got indexed in the first place.
    – Tim R
    Commented May 19, 2023 at 4:18

2 Answers 2

Reset to default
0

Suggestions

  • Use <meta name='robots' content='noindex'> on all pages of your test site
  • Remove any links from your production site to the test site
  • Create a login page, for example test.example.com/login and dont have a link to it anywhere
  • On your login page, add a token (or more) in the SESSION variable
  • On all other pages, if the token is not there, redirect to the main site
  • Remove your pages at Remove Outdated Content
Improve this answer
0

hey!

Based on the update, it looks like you’ve already resolved the situation and did so in an elegant way.

However, there’s an important nuance. Blocking through robots.txt should work, but remember, robots.txt is more of a suggestion for search engines. To completely hide the subdomain, it’s better to block Google’s bots in the .htaccess file—this will guarantee the issue is resolved.

The effect of the disallow directive in robots.txt might take 1-3 months to reflect in Search Console, so be patient. :)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.