As I'm new and do not have much insight about security what I have found while scraping websites using scrapy and such tools is that most of the websites like google while naming there image or other files have a long string of characters attached to them in their urls.Like a url of image from google will have a structure like - lh3.googlecontent.com/b/img/svwjhhjwhbwJNShvxwnkjbwjbxwhbxjHBhwbxjhwbx.png

Does it make it safe from getting crawled in bulk by using tools like wget etc because I have never found a problem crawling them or getting them downloaded.

  • Sorry but this question is off-topic because it is not about a website you operate. Also, security-related questions are best asked on the Security Stack. – Simon Hayter Jul 13 '18 at 20:19
  • Thanks.. I believe security stack had a similar one..so that helped. – Vishal Sharma Jul 13 '18 at 23:11