1

I support a website, www.questionexample.com, and I also have a test site setup for staging, staging.questionexample.com, on another server. They are essentially the same site and are not new.

For some reason, staging.questionexample.com is not listed or indexed in search engines and I'm not sure why? I don't want staging.questionexample.com to be listed, but I'm being asked why this is so? Our marketing team is worried about duplicate content. The production site shows up fine in search.

I'm sure staging is not listed because I search on Google and Bing for site:staging.questionexample.com and nothing returns, when I search for site:www.questionexample.com then all the pages are returned.

So far on staging.questionexample.com I've checked the following:

  1. robots.txt does not differ from the production site.
  2. There are no (meta name="robots" content="noindex") metatags on the staging pages.
  3. There is no "X-Robots-Tag = no-index" entry in HTTP Response Headers in IIS.
  4. There's no password to visit the staging site, anyone can visit it.

I am stumped as to why the staging site is not listed in search engines, even though that's what we want. However, I'd like to know why or where else to look to find out why it's completely not listed.

Improve this question
4
  • It simply has not bee discovered yet. If you do not want your sub-domain indexed by search engines, I suggest immediately updating your robots.txt file to prevent this. Cheers!!
    – closetnoc
    Commented Sep 7, 2016 at 18:06
  • It's been up for over a year though
    – Andy Jr.
    Commented Sep 7, 2016 at 19:50
  • They are essentially the same site and are not new - see duplicate content. Search engines try not to index content that's already been indexed.
    – dan ♦
    Commented Sep 8, 2016 at 3:56
  • 1
    See also: Why isn't my website in Google search results?
    – Stephen Ostermiller ♦
    Commented Sep 8, 2016 at 18:52

4 Answers 4

Reset to default
0

You need to go sign up with Google's Webmaster Tools. Google Webmaster Tools

Once signed up, click add property. You will need to confirm ownership in 1 of 2 ways. File upload to root folder of website, or hosting login via Webmaster Tools.

Once confirmed. Add property, then go to the crawl tab on the left and select fetch and render. Once fetched (usually within minutes) You will see a Submit to Google Index. Click submit.

I would also suggest submitting a sitemap when possible.

It will message you with any/all errors.

It is essential to use for all developers.

Text tutorial can be read here: Siteground

Improve this answer
4
  • This answer doesn't address why the staging site isn't indexed when the main site is indexed.
    – Stephen Ostermiller ♦
    Commented Sep 8, 2016 at 18:53
  • Actually it does. Because Google at that point will tell you the cause of any crawl and index errors in the webmaster message console. Any other answer would only be a guess.
    – norcal johnny
    Commented Sep 8, 2016 at 22:38
  • +1 for a potentially useful answer then. Search Console doesn't actually report what on your site isn't indexed though. Fetch as Google will report crawl errors, but not reasons for no index inclusion. If you upload a sitemap it will tell you how many of the pages are indexed, but it won't tell you why the others are not.
    – Stephen Ostermiller ♦
    Commented Sep 8, 2016 at 22:44
  • Yep, it seems the site was never discovered and never submitted, that's the only reason it wasn't listed and indexed. I've updated the robots.txt and added a noindex meta tag within the <head> of the master files to ensure this never happens on the staging site. Thanks!
    – Andy Jr.
    Commented Sep 9, 2016 at 15:13
-1

You must add your site to google, bing etc. Search for Google Webmaster tool or Bing webmaster tool. you can check your site is crawl by google bot or not by searching in google

site:yoursite.com

If google bot crawl your site then you will find only your site page link there.

Improve this answer
1
  • Ultimately, this was the answer. The test site seems to have just never been discovered by any searches, due to never having been submitted. I've updated the robots.txt and metadata to ensure the staging site is never listed.
    – Andy Jr.
    Commented Sep 9, 2016 at 15:11
-2

It depends on how long your website has been up for.

If you want to be indexed by Google, Bing, yahoo, ECT. Yes they do have bots to crawl the web but they still need people to accept them.

Also Google has made it where websites with huge pop-ups rank lower.

W3 verification also helps on that.

TL;DR

Just give it time.

What's the site?

Improve this answer
2
  • It's been up for over a year though, going on two years
    – Andy Jr.
    Commented Sep 7, 2016 at 19:50
  • This answer doesn't address why the staging site isn't indexed when the main site is indexed.
    – Stephen Ostermiller ♦
    Commented Sep 8, 2016 at 18:53
-2

While going through your website www.questionexample.com i can see its not live right now. questionexample.com’s server DNS address could not be found. That's why no bot can able to crawl.

Improve this answer
1
  • 2
    I believe this was just an example URL for the purpose of asking a question here.
    – dan ♦
    Commented Sep 8, 2016 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.