I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two kinds of links and either allows both of them, or none of them.
Typical links are:
I have tried :
User-agent: * Allow: /public/ Disallow: /
User-agent: * Allow: /public/ Disallow: /user/
but either both were blocked or neither were blocked. Any ideas on how to solve this?