We have some microservices and as an example, Language Service's URL is the following;
http://prod-lang-service.example.com/
We have been getting errors (NotFoundException) on Rollbar that Google bots are calling this URL;
http://prod-lang-service.example.com/robots.txt
This is the Curl command;
curl --request GET 'http://prod-lang-service.example.com/robots.txt' \
--header 'cf-connecting-ip:66.249.66.34' \
--header 'accept-encoding:gzip' \
--header 'x-forwarded-port:443' \
--header 'x-forwarded-for:66.249.66.34, 172.70.174.230, 66.249.66.34' \
--header 'host:prod-lang-service.example.com' \
--header 'accept:text/plain,text/html,*/*' \
--header 'user-agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
--header 'x-amzn-trace-id:Root=1-6525fb3c-1626d9360cb93e3825dae376' \
--header 'connection:upgrade' \
--header 'x-forwarded-proto:https' \
--header 'cdn-loop:cloudflare' \
--header 'cf-visitor:{"scheme":"https"}' \
--header 'cf-ray:814359dad9149c48-IAD' \
--header 'cf-ipcountry:US'
Should I block this and other microservice URLs from robots.txt
from www.example.com/robots.txt
?
robots.txt
on your main domain to block URLs on a sub-domain. However, that isn't possible. Why wouldn't you just create arobots.txt
file on the sub-domain?