I have a website on which I use a redirect only for Googlebot to a HTML file. It only contains metadata that I want to show in the search results. I have removed everything else from the HTML:

<!DOCTYPE html>
<html data-n-head="">
  <meta data-n-head="true" charset="utf-8">
  <meta data-n-head="true" content="width=device-width,initial-scale=1" name="viewport">
  <meta data-n-head="true" content="MyWebsite" name="description">
  <meta data-n-head="true" content="MyWebsite" property="og:description">
  <title data-n-head="true">MyWebsite</title>

Why can't I find the site on Google anymore when I search for the domain? Is Google not indexing it on purpose because it does not contain anything in the body?

I currently use the following rule in my web.config that redirects the bot to the HTML with the domain specific meta tags:

            <rule name="Redirect Bots" stopProcessing="true">
              <match url="^$"/>
              <conditions logicalGrouping="MatchAny">
                <add input="{HTTP_USER_AGENT}" pattern="(Googlebot)|(Bingbot)|(MSNBot)|(Facebot)|(facebookexternalhit)|(Twitterbot)|(Pinterest)" />
              <action type="Redirect" url="https://{HTTP_HOST}/robots/{HTTP_HOST}/index.html"/>
  • In other words you are cloaking and showing Googlebot only pages with no content that it will treat as soft 404 error pages. Why are you doing this? Are you trying to prevent the rest of the text on your pages from getting indexed? – Stephen Ostermiller Feb 20 '18 at 17:21
  • My website changes dynamically based on the domain that is used to access it. I.e. domainA.com and domainB.com both point to the same server. Javascript then places the appropriate content in the metatags, pagetitle etc. Unfortunately this does not get picked up by Google and I have therefore create separate pages for the each domain that are served only to the bot – Chris Feb 20 '18 at 17:25
  • You can have multiple domains pointing to the same server in ways that Googlebot can access. Have you looking into creating virtual hosts under your web server? – Stephen Ostermiller Feb 20 '18 at 17:26
  • Regular users should not be redirected. They should all point to the same index.html, no matter what domain they are using. Only the bots should be redirected to another html with the domain specific metadata. I will add some more info to the question – Chris Feb 20 '18 at 17:32
  • Google requires that you treat bots and users the same. You need to find a solution that doesn't involve cloaking. – Stephen Ostermiller Feb 20 '18 at 17:33

You have to show Googlebot what the page will look like when it sends a visitor to it. Google will not send humans to pages that it doesn't know what is on those pages. When you redirect Googlebot to a file that just displays the meta description, Google thinks that these pages will only have meta content and therefore that's not very useful to visitors. Or worse, Google will know that you're showing it only meta content but that you're showing visitors more content than that.

You cannot hide the content from Google that you are going to show its users that it sends to your site. This will lead to automatic deindexing and traffic loss.

I understand that you have some sort of dynamically generating content on your pages that you're trying to show visitors based on where they are from, but you still have to give Google a better understanding of what the page will look like when they send a visitor there.

I'm not sure if experiments have been made about dynamically generated content targeting ip addresses of countries regarding SEO performance, but if that is what you are going for then you should do more research on this and whether that is going to work with Google.

The less content that you can show Google, the worse your SEO performance is going to be.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.