MEAN stack app robots.txt not performing as expected

Question

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two kinds of links and either allows both of them, or none of them.

Typical links are:

1. to be disallowed: https://www.example.org/index.html#!/user/lessons/53f9b74f264b02126b0300f2/intro

2. to be allowed: https://www.example.org/index.html#!/public/lessons/53f9b74f264b02126b0300f2/intro

I have tried :

User-agent: *
Allow: /public/
Disallow: /

and this:

User-agent: *
Allow: /public/
Disallow: /user/

but either both were blocked or neither were blocked. Any ideas on how to solve this?

Answer 1

Summarizing the comments from closetnoc and my understanding:

robots.txt does not work for sites using #! to either allow or disallow folders in the 'fragment' ie, after the #!.

It can be used to change access to folders or files not preceded by #!. For example: Disallow: /backend/crawler

If you do not want links to be crawled, then the endpoint of the link should be noindexfile, such as after a redirect to a login page.

Answer 2

Try this.

User-agent: *
Disallow: */user/*