Skip to main content
Share Your Experience: Take the 2024 Developer Survey

New answers tagged

0 votes

How do I stop Amazon from crawling a website?

Setting a robots.txt policy does not "prevent" any crawler from indexing your website. It is more like politely asking not to do it. The robot may respect your request or ignore it. Since ...
Jan Iłowski's user avatar

Top 50 recent answers are included