10

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form?

For example, if I were to have 4 translations of one page, and the language selection were a POST form to the same URL, which then redirected to the language-specific URL, would Googlebot be able to crawl the "other" languages?

9

The Googlebot can submit forms, but it generally doesn't unless it can detect a reason to do so.

So from the links, if your translations were AJAX'd and built properly, Googlebot may very well submit the form to see what the results would be. However, this behavior (especially on POST), is not guaranteed and you should probably use GET to make things more accessible to the robots.

  • 1
    @DisgruntledGoat Are you sure? Do you have a source? The linked article states that "For text boxes, our computers automatically choose words from the site that has the form". The article was written in 2008 though, so maybe things have changed? – s427 Aug 14 '13 at 10:50
  • 2
    @s427 Yes you are correct, I have deleted my comment. For GET forms (such as a search box) Googlebot may generate text and submit it. There are some caveats listed there - they do not submit POST forms, nor any forms with password fields. – DisgruntledGoat Aug 14 '13 at 11:41
4

Bing absolutely does. Before I told it not to it would continually submit both a contact form and a survey form. I knew it was Bingbot because I included $_SERVER['HTTP_USER_AGENT'] in the $_POST information. No other spider ever submitted those forms.

  • How did you tell Bing to stop submitting? Thanks! – Thomas Landauer Feb 25 '16 at 11:39
  • I checked the HTTP_USER_AGENT for bingbot and disabled the submit button. – nemmy May 2 '18 at 0:38
3

They can and do. I've got a simple email collection form on the front of a site I'm about ready to launch. I just received a submission from that form from IP 66.249.73.214, which is a GoogleBot IP.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.