I have the following meta tags on my website (dev server):
<meta name="robots" content="noindex"> <meta name="googlebot" content="noindex">
And I also configured a
However, on Google Search Console, I can still search for content and get A LOT of results on Google, as other people on my team.
Robots file (www root folder)
Disallow: / Disallow: /frontend/ Disallow: /frontend Allow: /login/ Allow: /login Allow: /portal/ Allow: /portal Allow: /store/ Allow: /store
I basically have one server with 3 folders (4 projects):
- Login is a SSO
- portal is a user portal
- store is and eCommerce.
- front-end is a pure static front-end without password and logic of the other 3 folders (dev sandbox). This is the one I'm trying to block Google keeps indexing.
After reading more about the subject and per the suggestion I removed robots.the text completely from the server file and maintained the meta tags.
After that, I forced removal and crawling:
Result: it keeps indexing and I can still find on the search engine in multiple computers I own and different browsers: