1

How do I tell search engine crawlers not to check a checkbox when indexing my site? I want to do something like this:

<input type="checkbox" rel="nofollow" />

, but the rel attribute is not listed in the list of attributes here: The Input (Form Input) element, which makes sense, because this isn't a link. But I am not sure how to tell search engines that they shouldn't check this when filling out the form. If I include this rel="nofollow" attribute here, will search engines comply anyway, even though it is not valid?

3
  • Is this a form a GET or POST ? Commented Feb 1, 2023 at 4:29
  • It's a GET form.
    – kloddant
    Commented Feb 1, 2023 at 14:02
  • What would make you think that (any) SEO crawler would arbitrarily change the checkbox input status in your code?
    – elbrant
    Commented Feb 3, 2023 at 2:17

2 Answers 2

1

It seems that Google only follows GET forms. So make it a POST.

That means that if a search form is forbidden in robots.txt, we won't crawl any of the URLs that a form would generate. Similarly, we only retrieve GET forms and avoid forms that require any kind of user information. For example, we omit any forms that have a password input or that use terms commonly associated with personal information such as logins, userids, contacts, etc. We are also mindful of the impact we can have on web sites and limit ourselves to a very small number of fetches for a given site.

From developers.google

For indexing different search results, I suggest the following approaches

  1. Use different params or urls for different search results and put them in the sitemap and refer to them on a sacrificial or site-map page.
  2. Detect that a robot is crawling and make some inputs read-only and autofill others to achieve different search results.
2
  • Thank you for your response. It's a GET form because it's a search form, and GET allows users to bookmark and link the search results, and it allows search engines to index the search results. As you mention, POST does not allow search engines to index the results at all, which is not what I want; I am looking for a way to tell it to index only certain results and not others. This form also does not modify any information on the server, so POST is not conceptually appropriate, because POST is intended for situations when you are modifying information on the server.
    – kloddant
    Commented Feb 1, 2023 at 14:27
  • That wasn't obvious, so have updated my answer. Commented Feb 2, 2023 at 12:46
0

You can block Googlebot from crawling your forms by excluding them in your robots.txt file.

3
  • I want it to crawl the form, but I only want it to pay attention to certain inputs in the form and not others.
    – kloddant
    Commented Jan 31, 2023 at 15:11
  • You don't want Google to crawl and index site search results. It is bad user experience for users to search on Google only to click to another set of similar looking search results. Google has dropped entire sites from their search index because those sites allow their search results to be crawled and indexed. See Search results in search results Commented Feb 1, 2023 at 16:48
  • It is also possible to use robots.txt to disallow certain parameters in URLs using wildcards, so even if you are dead set on trying to get your search results indexed, this answer is the approach I would recommend. Commented Feb 1, 2023 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.