I have a headless wordpress site just using wordpress API.
My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress.
I set up robots.txt as follows:
User-agent: * Disallow: /wordpress/ Disallow: /wordpress/wp-admin/ Disallow: /wordpress/wp-login.php
However, looks like google still indexed pages under
/wordpress, and I have a coverage issue warning in google search console
Indexed, though blocked by robots.txt.
Most of the pages it indexed actually don't even exist, they 404. So I could allow google to try to crawl them and consequently de-index. But wow did Google even find these pages? I don't have them linked anywhere on my site or in my submitted sitemap.xml. E.g. something like
https://example.com/wordpress/product/awesome-productis indexed with
No information is available for this page., while the real page is also indexed and exists at
Other pages do actually exist - for example API endpoints that are consumed by my site. Should I care if these are indexed or not?