1

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two kinds of links and either allows both of them, or none of them.

Typical links are:

  1. to be disallowed: https://www.example.org/index.html#!/user/lessons/53f9b74f264b02126b0300f2/intro

  2. to be allowed: https://www.example.org/index.html#!/public/lessons/53f9b74f264b02126b0300f2/intro

I have tried :

User-agent: *
Allow: /public/
Disallow: /

and this:

User-agent: *
Allow: /public/
Disallow: /user/

but either both were blocked or neither were blocked. Any ideas on how to solve this?

7
  • 1
    It may be that anything from #! will be ignored.
    – closetnoc
    Jul 15, 2018 at 17:59
  • I agree that that this makes it more complex to deal with, but google does crawl the site and finds thousands of links, so I would expect them to have a solution to blocking them as well. Jul 15, 2018 at 18:24
  • Except that robots.txt is not designed to be that grandular. It may be that you will need to use noindex or nofollow.
    – closetnoc
    Jul 15, 2018 at 18:32
  • I do have a noindex directive on the site, but I don't see how it can be used for this situation. The public links have the same html file as the non public(user) only the path is different. Any ideas? Jul 15, 2018 at 18:47
  • It sounds like whatever you do will have to be done in your CMS or code. I assume you have a login for private users. If that is the case, are you requiring a login for /user? If a login is required, then your work should be done assuming you have set this up correctly. Google will try and follow any link it finds. For users who are not logged on, they should not see private links. This includes Google unless you want these pages indexed also. I do not recommend indexing pages that ordinary search users cannot see. That is bad UX (user experience) in my book. Cheers!
    – closetnoc
    Jul 15, 2018 at 19:00

2 Answers 2

1

Summarizing the comments from closetnoc and my understanding:

robots.txt does not work for sites using #! to either allow or disallow folders in the 'fragment' ie, after the #!.

It can be used to change access to folders or files not preceded by #!. For example: Disallow: /backend/crawler

If you do not want links to be crawled, then the endpoint of the link should be noindexfile, such as after a redirect to a login page.

0

Try this.

User-agent: *
Disallow: */user/*
1
  • "Try this" does not make for a good answer. You should explain how and why this solves their problem.
    – John Conde
    Aug 18, 2018 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.