Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Should I disallow the age gate page in robots.txt?

I need to make sure the crawlers can get through the gate.

share|improve this question

migrated from stackoverflow.com Jun 29 '12 at 12:48

This question came from our site for professional and enthusiast programmers.

    
I found this site to be pretty thorough when I was initially researching robots.txt robotstxt.org –  davidisawesome Jun 5 '12 at 19:44
    
didn't have what i was looking for, just need to know if i should disallow the age gate page?? –  Jordan Reiser Jun 5 '12 at 19:58
    
personally I don't see any reason you should have to. The age gate just has the user confirm they are at least x-years-old, then they continue to the home page, right? It sounds like a personal preference really –  davidisawesome Jun 5 '12 at 20:27
    
right, but i read that sometimes the bots have trouble getting through to the home page because of the age gate –  Jordan Reiser Jun 5 '12 at 20:27
1  
Do you really want crawlers to potentially show age sensitive information to people out of that age range? –  Rowland Shaw Jun 28 '12 at 7:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.