I was reviewing the logs of my website (WordPress), and I saw a line like this :
myWebsite:80 5.10.83.28 - - [17/Jan/2014:09:05:53 +0000] "GET myUrl == HTTP/1.1" 404 5716 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
So a bot called AhrefsBot
was visiting myUrl
.
The problem is that I removed the page myUrl
weeks ago.
So why I am seeing this bot still requesting it?
How did it find the URL myUrl
, especially when I'm sure that there are no pages that link to it? And how do I avoid these kind of 404 pages?