Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Should I disallow the age gate page in robots.txt?

I need to make sure the crawlers can get through the gate.

share|improve this question
I found this site to be pretty thorough when I was initially researching robots.txt robotstxt.org –  davidisawesome Jun 5 '12 at 19:44
didn't have what i was looking for, just need to know if i should disallow the age gate page?? –  Jordan Reiser Jun 5 '12 at 19:58
personally I don't see any reason you should have to. The age gate just has the user confirm they are at least x-years-old, then they continue to the home page, right? It sounds like a personal preference really –  davidisawesome Jun 5 '12 at 20:27
right, but i read that sometimes the bots have trouble getting through to the home page because of the age gate –  Jordan Reiser Jun 5 '12 at 20:27
Do you really want crawlers to potentially show age sensitive information to people out of that age range? –  Rowland Shaw Jun 28 '12 at 7:43
show 3 more comments

migrated from stackoverflow.com Jun 29 '12 at 12:48

This question came from our site for professional and enthusiast programmers.

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.