All Questions
3 questions
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
2
votes
1
answer
119
views
Override robots.txt exclusion to allow Mediapartners-Google for AdSense
I use robots.txt to block all robots.txt compliant bots from a directory like this:
User-agent: *
Disallow: /sub/*
How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
- 221
2
votes
1
answer
397
views
What is the difference between Disallow and Allow?
I had 2 questions in my mind:
what is the difference between Allow and Disallow?
which one should I use to allow Google bot for AdSense?
User-agent: Mediapartners-Google
Disallow:
User-agent: ...
