We have a web application that handles many websites that have test and production environment.
This application has only one folder and it's hosted with IIS.
We already have
index follow on respective domains but it's not working for static files (images, pdf documents etc.)
Is there a way to setup the
robots.txt disallowing every single test domain?
Disallow: test.domain1.com Disallow: test.domain2.com
Take note that the domains are 200+ and they are being added/removed very fastly so
robots.txt would be a good solution for us. We do not have access to the google search engine of each domain to request a removal from google index. Also the solution should work for bing and other search engines.
Can you see any way to achieve this?