Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Recently, a site of mine was rejected due to "Login Only":

Login Only: During our review of your website, we found that the majority of pages on your site are behind a login, or there is restricted access. Please note that we will not approve applications for login-protected pages, as we are not able to review their content for acceptance into the program.

Although the site does require login to send content, it doesn't require any to view any page. How do I tell the Googlebot or whatever is used to crawl pages to AdSense that all the content is publicly available but registration is needed to post?

Current robots.txt

User-agent: *
Disallow: /moderation/
User-agent: Googlebot-Image
Disallow: /
User-agent: Mediapartners-Google
Allow: /
share|improve this question
    
can you link to the website? –  YardenST Nov 10 '12 at 23:26
    
@YardenST I'm afraid not. Can you help me with the provided information? –  Renan Nov 10 '12 at 23:49
    
that's a situation! Can you change the way pages are accessed? Maybe you are using some cookies or something that is blocking google crawlers from seeing the pages as public? –  Mohamed Anis Dahmani Nov 11 '12 at 2:33
    
@Renan Do you have any mechanism in your website, that prevent bots from accessing the site? –  YardenST Nov 11 '12 at 7:55
    
@YardenST I do have some restrictions in robots.txt. I added it in the question. –  Renan Nov 11 '12 at 15:19

1 Answer 1

I had the same problem and I resolved it.

Google sends crawler to visit your site. It´s automatic robot control, not human control. Your rejection is also automatic from this crawler.

If you want see what crawler really gets when he visits your site, you must using some text web browser like Lynx. After using this browser, I saw what my problem was:

  • The crawler need to see only clean links. If you have some other element nested inside the A element (like a span) the crawler won´t see it as a link. For the crawler, I had only one clean link and it was a link to login page.
  • The crawler doesn't support content filled in with AJAX.

Take a look to your page with Lynx browser to find the reason why you get rejection.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.