I would like to not have OpenAI, ChatGPT or the potentially associated Bingbot scrape my website. The latter seems to have a defined bot name. What about the first two? They seem to be using Common Crawl scraper among other sources, so that's another one to block in robots.txt, with what name though? Any other items to add to robots.txt to ban OpenAI? Is it known if they honor robots.txt or like (e.g.) rogue Linkedin scraping just go and do it all rules be damned? Earlier discussions on this topic seem inconclusive.
2 Answers
Yes, it is possible both through a robots.txt declaration and to be restricted at the IP address level.
If you want to restrict it via robots.txt, you need the following declarations:
User-agent: ChatGPT-User
Disallow: /
If you want to block it at the IP address level, you need to block the following IP address: 23.98.142.176/28
More information could be found here: ChatGPT-User
-
Thanks for the pointer! that doesn't seem to be their scraper bot though (quote from the link: "not used for crawling the web"). I was also curious if Common Crawl CCBot, Bingbot or any other bots are helping them as well. May 17 at 1:48
-
Unfortunately, I can't find any official information. As far as I know they haven't used the crawling bots of the search engines to gather information.– ManiacMay 18 at 18:35
they have a special useragent for their scraper now, it’s GPTBot
https://platform.openai.com/docs/gptbot
i would also recomment blocking Googlebot, since they declared to use everything they can access for AI training
-
thanks! googlebot drives a ton of traffic so no, thank you. openai drives nothing. Aug 8 at 10:16
-
Hi Ivan, the protocol is to tick the correct answer, and to upvote it if you havent already Aug 9 at 14:50