2

We are working on a new version of a website. The current version is indexed by google, everything looks fine. The new version will have more or less the same content & links, but we are changing the design slightly & implement a new architecture (eg. using AngularJS etc). We want to test the new version which is still under development to see how google will fetch it. We added a robots.txt file to 'disallow /' because we don't want google to index our development server.

Is it safe or recommended to use the "fetch as google" tool before the release of the new version?

PS: I think this post suggests to wait until the release: Manual "Fetch as Google" or better wait? But then, how could we test that google will see all the content correctly?

Improve this question
4
  • You should always test your site for your users, not for search bots. If the site is compatible with most major browsers backing date several years its extremely like the site will work with Google. Your tests should include all browsers with several versions and on multiple platforms, and disabling JS is always a plus when testing and using .no-js.
    – Simon Hayter
    Sep 10 '15 at 16:46
  • I'm not by no way suggesting you shouldn't test Google but ultimately, Google is not your priority, your users are... and if you make sure you users can view your site on a vast wide of devices and platforms then Google will be able too.
    – Simon Hayter
    Sep 10 '15 at 16:53
  • We are doing a full range of tests for usability, security, regression and automated tests etc. Fetching as google is one of the steps to make sure google will be able to fetch our AngularJS application so that our pages will stay indexed. Thank you for your advice though.
    – Adi Fatol
    Sep 11 '15 at 13:25
  • I'm running into the same problem. Parts of our site do not render when being "fetched as Google". The end user does see those contents, anyway. Did you find a solution for this, yet?
    – René Schubert
    Dec 3 '15 at 8:49
1

There are two possibilities, use the noindex meta or use the X-Robots-Tag header. We used both.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?hl=en

Our solution was to change the robots.txt and allow only google to crawl our pages:

eg.

User-agent: * 
Disallow: / 
User-agent: Googlebot 
Allow: /

but disallow it to index them by using the header X-Robots-Tag: noindex and also adding the meta eg. <meta name="googlebot" content="noindex" />

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.