I have a website that largely features dynamic content. Every piece of dynamic content has it's own URL, such as "www.example.com/page/\*/\*"

The problem I have is that many of these pages require a user to be logged in. I have read about AdSense crawler login, and tried to implement it. However, I can find no mention of wildcards or regular expressions for the URL. As far as I can tell, AdSense crawler login only supports static URLs.

Is there any way to setup an AdSense crawler login that supports a range of dynamic URLs?

1 Answer 1


From the AdSense help page it says that they support entire directories that need login:

You'll need to provide the following information:

  • Restricted directory or URL: The URL that is blocked from our crawler's access.
  • ...

So you could tell them that /page/ requires login. If there are other pages in that directory that don't require login, AdSense crawler would also fetched them logged in.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.