Google indexing Wordpress in subdirectory disallowed in robots.txt

Question

I have a headless wordpress site just using wordpress API.

My site is in the root at example.com, and wordpress live subdirectory example.com/wordpress.

I set up robots.txt as follows:

User-agent: *
Disallow: /wordpress/
Disallow: /wordpress/wp-admin/
Disallow: /wordpress/wp-login.php

However, looks like google still indexed pages under /wordpress, and I have a coverage issue warning in google search console Indexed, though blocked by robots.txt.

1. Most of the pages it indexed actually don't even exist, they 404. So I could allow google to try to crawl them and consequently de-index. But wow did Google even find these pages? I don't have them linked anywhere on my site or in my submitted sitemap.xml. E.g. something like https://example.com/wordpress/product/awesome-product is indexed with No information is available for this page., while the real page is also indexed and exists at https://example.com/product/awesome-product

2. Other pages do actually exist - for example API endpoints that are consumed by my site. Should I care if these are indexed or not?

Answer 1

Using Disallow in robots.txt doesn't prevent search engines from indexing pages, only from crawling them - this is a common SEO myth.

To answer your questions:

1. Googlebot finds pages not only through links and the sitemap, but also by reading the page's source code for URLs in JavaScript code, etc. It's quite clever.

2. I wouldn't worry too much about API endpoints being indexed.

The pages that you don't want to be indexed should serve up a meta noindex HTML tag, or a X-Robots-Tag: noindex HTTP header. Make sure to remove those pages from your robots.txt so that Google can crawl them and see the directive that they shouldn't be indexed.

More details here:

https://developers.google.com/search/reference/robots_meta_tag