0

We have some microservices and as an example, Language Service's URL is the following;

http://prod-lang-service.example.com/

We have been getting errors (NotFoundException) on Rollbar that Google bots are calling this URL;

http://prod-lang-service.example.com/robots.txt

This is the Curl command;

curl --request GET 'http://prod-lang-service.example.com/robots.txt' \
--header 'cf-connecting-ip:66.249.66.34' \
--header 'accept-encoding:gzip' \
--header 'x-forwarded-port:443' \
--header 'x-forwarded-for:66.249.66.34, 172.70.174.230, 66.249.66.34' \
--header 'host:prod-lang-service.example.com' \
--header 'accept:text/plain,text/html,*/*' \
--header 'user-agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
--header 'x-amzn-trace-id:Root=1-6525fb3c-1626d9360cb93e3825dae376' \
--header 'connection:upgrade' \
--header 'x-forwarded-proto:https' \
--header 'cdn-loop:cloudflare' \
--header 'cf-visitor:{"scheme":"https"}' \
--header 'cf-ray:814359dad9149c48-IAD' \
--header 'cf-ipcountry:US'

Should I block this and other microservice URLs from robots.txt from www.example.com/robots.txt?

3
  • Is that curl command how you are emulating the exact headers that Googlebot is sending? Commented Oct 11, 2023 at 10:14
  • 1
    It sounds like you are proposing using robots.txt on your main domain to block URLs on a sub-domain. However, that isn't possible. Why wouldn't you just create a robots.txt file on the sub-domain? Commented Oct 11, 2023 at 10:15
  • @StephenOstermiller thanks we will proceed creating sub-domain level robots.txt and disallow there. Cheers! Commented Oct 12, 2023 at 5:18

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.