2

Is there a way to "ask" google to add a query string paramater to the url when crawling my website? (maybe in robots.txt?) so that when it crawls example.com, it would add something like ?iam=google . Because we use Cloudfront to serve our website, we need a way to detect search engines on the cloudfront and forward them to origin server.

migrated from stackoverflow.com Jun 27 '14 at 13:46

This question came from our site for professional and enthusiast programmers.

1

There is no way to get Googlebot to add parameters. Google does not support such functionality.

Googlebot does send a User-Agent header:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

However, you should not being doing anything differently for Googlebot than you do for users. Google calls that practice cloaking. Google will penalize sites that treat Googlebot specially.

There should be no reason that your Cloudfront servers can't serve the same cached content to Googlebot that they serve to users.

If your pages are AJAX and you need Google to be able to access the content, you can implement the hash bang AJAX crawling scheme.

To implement it, you need to have use #! in your URL where you would normalling use a # sign. Then Googlebot will fetch a URL with a parameter _escaped_fragment=everything+after+the+hash+sign. Your server needs to be configured to return the content that would be shown to the user at that point.

If you use pushstate to change your URLs (and they don't have a hash), but the page still uses JavaScript to load the content (rather than the first page having all the data served by the server), you can still use a similar technique. You can include a meta tag to tell Googlebot to fetch the _escaped_fragment version of the URL:

 <meta name="fragment" content="!">

See point #4 from "Transformation of the URL" in Google's documentation

  • Hey thanks for the answer. We're not trying to cloak. The thing is the website uses javascript heavily, all the content is loaded via ajax so I was thinking I would make static version of the page for robots. Cloudfront doesn't forward the user agent header unfortunately. – Moosh Jun 25 '14 at 14:29
  • I added a section to my answer about how to deal with crawlable AJAX in a situation like this. – Stephen Ostermiller Jun 25 '14 at 15:30
  • Thanks for the tips. I actually am aware of this technique but it won't be of much use for this situation as we use history api not hash. So lets say if example.com/page1/ is cached on cloudfront, the crawler would crawl the cached "javascript" version of the page, that's why my question is how to detect the crawler on cloudfront somehow. – Moosh Jun 25 '14 at 17:30
  • Google even has a mechanism to deal with that. I've added information about the meta fragment tag to the answer. – Stephen Ostermiller Jun 25 '14 at 18:25
  • This is exactly what I needed. Thank you very much Stephen! – Moosh Jun 26 '14 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.