0

I am maintaining a website with several million pages of user generated content.

On each page, there is a calculator implemented allowing people to press a [+] or [-] button to increase or decrease a value. This calculator will show the result in real time using javascript.

Now we are reworking our AMP pages and try to keep them as close to the non-AMP versions as possible. However, we currently cannot implement this javascript calculator on AMP for several reasons.

Therefore, we are thinking about including the [+] and [-] buttons on the AMP page as links to the non-AMP version with a GET param in order to show the calculation result on the non-AMP page.

We are afraid though that this will affect our crawl budget since Googlebot will most likely follow these two buttons for each of the millions of AMP pages.

Is this fear justified and if so, how to handle this situation? Thank you very much!

We have considered the following options:

  • use a simple POST form with a submit button for each [+]/[-] button - but according to several sources, Googlebot will often follow simple POST forms as well, so this appears not to be a solution
  • use a hash tag instead of a GET param (and handle it with javascript) - but Googlebot seems to index links with hash tags as well - therefore no solution
  • use a GET param and tell google to ignore it in the search console - but it seems like this feature is not reliable and outdated anyway

1 Answer 1

1

We are thinking about including the [+] and [-] buttons on the AMP page as links to the non-AMP version with a GET param in order to show the calculation result on the non-AMP page.

These GET parameters create an "infinite space" (there are in practice an "infinite" number of pages, one for every number in the GET param). So you would be correct to worry about Google wasting your crawl budget in this space.

The best way to deal with this is to block crawling of this infinite space in your robots.txt file. So for instance, to block crawling of pages like this:

http://example.com/calculator/?num=1

http://example.com/calculator/?num=2

You would add this into your robots.txt:

User-agent: *
Disallow: /calculator/?

This would block crawling of every url with a query string, but allow /calculator/ itself (no query string) to be indexed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.