1

Users have the ability to search on my site. This function renders a search results page that has occasionally been indexed by google and served in SERPs. Normally this is not an issue, but when documents are updated users need to see the most recent information and sometimes stale content/files are index higher than the current information.

The solution would be: <meta name="robots" content="noindex" /> in the head of my search results page. My site is templated, including a base template that has head in it, so the same head gets served for every page. I can't put a noindex in there without some sort of conditional logic that checks if the URL is for a search page, which seems real hacky.

I could create the noindex meta element dynamically with javascript inside my search page template, but someone has brought up that this is pointless because crawlers will crawl only the pre-processed markup. Is this the case?

Alternatively, I could add a disallow in my robots.txt, but Google has said this isn't a great method (2nd paragraph, bold). Any insight on this?

Am I stuck using conditional logic in the head of my base template?

Improve this question
6
  • 1
    Why would the noindex tag have to be created dynamically with JavaScript? Couldn't you just edit the search page template to put it in directly?
    – Stephen Ostermiller ♦
    Jul 19, 2017 at 14:42
  • Disallow in robots.txt mostly stops indexing. If Google can't crawl it, Google won't usually index it. The only exceptions are when there are a bunch of external links to it.
    – Stephen Ostermiller ♦
    Jul 19, 2017 at 14:49
  • 1
    According to MOZ, Google should see your meta robots insert using JS just fine. As well, the second Google link bold statement does not make sense even for Google. This is precisely what the robots.txt file is for! Google is referring to the idiot policy of listing linked to pages (dangling links) for a period until the page is fetched or the link is found to be broken. Being blocked via robots should remove the SERP link in time. You can use your search template and simply place your meta robots tag there. I would do that and use robots.txt. Cheers!!
    – closetnoc
    Jul 19, 2017 at 16:38
  • @StephenOstermiller I was under the impression meta tags are only allowed inside the head element. A cursory search says that's not entirely true, but it still seems hacky, i.e. broken markup.
    – smilebomb
    Jul 19, 2017 at 17:18
  • Why couldn't you edit the meta tag into the head element of your search page?
    – Stephen Ostermiller ♦
    Jul 19, 2017 at 17:26

1 Answer 1

Reset to default
2

Yes, this could very well be the case regarding the pre-processed markup. I'm not aware of any ironclad evidence on this, but I would consider it risky to use the JS solution.

In this specific case disallowing via robots.txt is probably your best solution as it's the solution that Google themselves suggest. Keep in mind that I wasn't able to find that bit of quoted text on the quoted URL though.

"Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines." (http://searchengineland.com/google-warning-against-letting-your-search-results-get-indexed-10709).

Improve this answer
1
  • 1
    The actually quote from Google about robots.txt on that page is "You should not use robots.txt as a means to hide your web pages from Google Search results."
    – Stephen Ostermiller ♦
    Jul 19, 2017 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.