3

I would like to not have OpenAI, ChatGPT or the potentially associated Bingbot scrape my website. The latter seems to have a defined bot name. What about the first two? They seem to be using Common Crawl scraper among other sources, so that's another one to block in robots.txt, with what name though? Any other items to add to robots.txt to ban OpenAI? Is it known if they honor robots.txt or like (e.g.) rogue Linkedin scraping just go and do it all rules be damned? Earlier discussions on this topic seem inconclusive.

2 Answers 2

4

Yes, it is possible both through a robots.txt declaration and to be restricted at the IP address level.

If you want to restrict it via robots.txt, you need the following declarations:

User-agent: ChatGPT-User
Disallow: /

If you want to block it at the IP address level, you need to block the following IP address: 23.98.142.176/28

More information could be found here: ChatGPT-User

2
  • Thanks for the pointer! that doesn't seem to be their scraper bot though (quote from the link: "not used for crawling the web"). I was also curious if Common Crawl CCBot, Bingbot or any other bots are helping them as well. May 17, 2023 at 1:48
  • Unfortunately, I can't find any official information. As far as I know they haven't used the crawling bots of the search engines to gather information.
    – Maniac
    May 18, 2023 at 18:35
4

they have a special useragent for their scraper now, it’s GPTBot

https://platform.openai.com/docs/gptbot

i would also recomment blocking Googlebot, since they declared to use everything they can access for AI training

https://tech.co/news/google-could-use-your-data-train-ai

2
  • thanks! googlebot drives a ton of traffic so no, thank you. openai drives nothing. Aug 8, 2023 at 10:16
  • Hi Ivan, the protocol is to tick the correct answer, and to upvote it if you havent already Aug 9, 2023 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.