I was reviewing the logs of my website (WordPress), and I saw a line like this :
myWebsite:80 22.214.171.124 - - [17/Jan/2014:09:05:53 +0000] "GET myUrl == HTTP/1.1" 404 5716 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
So a bot called
AhrefsBot was visiting
The problem is that I removed the page
myUrl weeks ago.
So why I am seeing this bot still requesting it?
How did it find the URL
myUrl, especially when I'm sure that there are no pages that link to it? And how do I avoid these kind of 404 pages?