Why are bots repeatedly crawling pages of my site that I never created?

Question

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not exist on the site.

And there were hundreds if not thousands of variations of these non-existent URLs, meaning that I was getting about 5000 requests a day and only about 250 of them were actually real visitors on real URLs. I was also confused when I noticed that when visiting these URLs, instead of returning a 404 error as I would have expected, the HTML of the homepage showed up, but without any styles or javascript.

A possible solution for me could be for example to block robots from visiting /fpss/ in the robots.txt file, but I would like to understand why they are doing this in the first place.

Answer 1

Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site.

An old site

These URLs could be from a previous owner of your domain name, or from a previous version of the site that you have since taken down. Once bots discover URLs, they rarely forget them. Search engine bots such as Googlebot crawl URLs that haven't worked in decades.

Broken links or misconfiguration

Your site itself may be creating broken links. Sometimes a content management system (CMS) or a plugin for it will automatically create links in ways you don't expect.

Malware

Your site could be infected by malware that automatically creates the links. Even once malware is cleaned up, bots may continue to crawl the URLs it created.

Link spam

Some other site may be linking to your site in weird ways. It could be a mistake on their part, maybe they got the domain name wrong. It could be a malicious attempt to get your penalized by search engines.

Crawler hueristics

Googlebot is known to try to find links in JavaScript. It often interprets random bits as URLs even when they are not. For example having JavaScript such as s = '/fpss/track/73/' would cause Googlebot to crawl your example URL because the string literal has slashes in it.

---

As a next step you should make sure the URLs return the proper "404 Not Found" status rather than serve up the contents of your home page. You probably have some rewrite rule that is causing that behavior. You'll need to remove that rule. It is possible that malware created that rule. If so, you will need to remove that malware by installing a clean version of your site on your server.

Google Search Console or Bing Webmaster Tools may be able to tell you where their crawlers are finding the links to those URLs. Verify your site with those services and see if either of them reports a referrer for any of those URLs.

Blocking the URLs in robots.txt might be OK, but I wouldn't recommend it unless the crawling is causing your server to bog down. Search engines occasionally index URLs that are blocked by robots.txt. If they can crawl the URLs and see a proper error status instead, there will be no danger of having those URLs indexed. Once the URLs return an error status, they will put much less strain on your server anyway.

I would also suggest linking to your CSS and JS using root relative linking. When the hrefs start with a slash (/), then they will load properly even for files in a subdirectory.