Google is ignoring meta tags and robots.txt

Question

I have the following meta tags on my website (dev server):

  <meta name="robots" content="noindex">
  <meta name="googlebot" content="noindex">

And I also configured a robots.txt (validated). However, on Google Search Console, I can still search for content and get A LOT of results on Google, as other people on my team.

This is what I get on Google Search Console:

Robots file (www root folder)

Disallow: /
Disallow: /frontend/
Disallow: /frontend
Allow: /login/
Allow: /login
Allow: /portal/
Allow: /portal
Allow: /store/
Allow: /store

I basically have one server with 3 folders (4 projects):

1. Login is a SSO
2. portal is a user portal
3. store is and eCommerce.
4. front-end is a pure static front-end without password and logic of the other 3 folders (dev sandbox). This is the one I'm trying to block Google keeps indexing.

After reading more about the subject and per the suggestion I removed robots.the text completely from the server file and maintained the meta tags.

After that, I forced removal and crawling:

• https://www.google.com/webmasters/tools/removals
• https://search.google.com

Result: it keeps indexing and I can still find on the search engine in multiple computers I own and different browsers:

Answer 1

First, robots.txt Disallow is not compatible with meta noindex tags. Because disallowing in robots.txt prevents crawling, it means that Googlebot will never see the noindex and may continue to index your website. When Google indexes sites that are blocked by robots.txt it shows them without knowing the words that are on the page. See How to resolve Google “Indexed, though blocked by robots.txt”

You should allow Googlebot to crawl your entire site so that it sees the noindex meta tags. Your robots.txt file should look like:

User-Agent: *
Disallow:

You should ensure that the <meta name="robots" content="noindex"> is on every page of your site. If you put it on just your home page, then only your home page will get de-indexed. There is no need to use the second tag just for Googlebot because Googlebot obeys either.

Once your tags are on your site, Googlebot will de-index your pages within 24 hours of next crawling them. It may take Google a couple months to re-crawl your entire site and remove the whole thing from its index.

To speed up the process you can continue to use the Google URL removal tool. The problem with the tool is that it only allows you to remove one page at a time. If you have lots of pages indexed, it could take a lot of work to use that tool for all your pages.