1

I know internal search result pages can be indexed sometimes but what I don't get is how it is crawled? Crawler only crawls links. Does that mean that someone would have to link to an internal search result page to get it indexed?

I suppose the algorithm doesn't randomly try to insert strings into the search box, but I also don't think someone would link to these pages.

  • It shouldn't matter if Google discovers your site search, you shouldn't allow Googlebot to crawl site search results pages. If your site search gets indexed, it is bad for user experience and Google could penalize your entire site. See: Matt Cutts: Search results in search results – Stephen Ostermiller May 16 at 18:13
  • @StephenOstermiller Yes, I am pretty sure that in most cases It doesn't make sense to let GoogleBot crawl these pages. I am still not sure how the bot discovers them however... – Tomáš Nosek May 17 at 0:02
2

I suppose the algorithm doesn't randomly try to insert strings into the search box.

Actually, Googlebot has done so and may still be doing it as I've not head to the contrary.

https://www.sistrix.com/ask-sistrix/google-index-google-bot-crawler/can-the-google-bot-fill-out-and-crawl-forms/

but I also don't think someone would link to these pages.

Who knows what people really do... there are weirdos out there! :-)

But could there be other reasons?

Yes. Actually Googlebot is not alone. There are thousands of spiders checking out websites and once in a while one of these may index your pages somewhere which is found by Googlebot and then checked out... and then indexed.

Also, Google themselves have had an experimental bot which would add sub-folders on one of my websites. It would end up with something like:

https://www.example.com/foo/foo/foo/foo/foo/foo/foo/foo/...

Because my CMS shows /foo and ignores the children, it would actually return a page. The good thing, though, is that the CMS would properly define the canonical URL so I was never penalized because of that bug. But I had many like that for a while... Hackers can also end up generating some mostly random URLs so it could look like a search even if it isn't a real search at all (i.e. the Google searches use a GET so whether it came from their form or not is not relevant to the shown results.)

As mentioned by Trebor, a good idea is to put all the results under a folder such as /search and have that path forbidden in your robots.txt.

Disallow: /search/

That will not prevent those hackers from making your search pages public, but it will not impact your search results since Googlebot will ignore those links.

1

No, it doesn't insert text into the search box. I'm assuming the results of your search page point to other pages that exist on your site. If these pages are navigable via your sitemap or other links on your site, the crawler will find them that way.

However, if the content your search page returns is hidden from them, then they normally won't be found. The crawler is not trying to use your search page to find additional content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.