I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two kinds of links and either allows both of them, or none of them.
Typical links are:
to be disallowed: https://www.example.org/index.html#!/user/lessons/53f9b74f264b02126b0300f2/intro
to be allowed: https://www.example.org/index.html#!/public/lessons/53f9b74f264b02126b0300f2/intro
I have tried :
User-agent: *
Allow: /public/
Disallow: /
and this:
User-agent: *
Allow: /public/
Disallow: /user/
but either both were blocked or neither were blocked. Any ideas on how to solve this?