We have a web application that handles many websites that have test and production environment.
This application has only one folder and it's hosted with IIS.
We already have noindex nofollow
, index follow
on respective domains but it's not working for static files (images, pdf documents etc.)
Is there a way to setup the robots.txt
disallowing every single test domain?
Example
Disallow: test.domain1.com
Disallow: test.domain2.com
etc.
Take note that the domains are 200+ and they are being added/removed very fastly so robots.txt
would be a good solution for us. We do not have access to the google search engine of each domain to request a removal from google index. Also the solution should work for bing and other search engines.
Can you see any way to achieve this?