1

I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access:www.example.com/index.php/{TOKEN}

That {TOKEN} is really random, no idea where it came from. I'm trying to respond that the pages don't exist by 301 redirecting to home page (not sure if this is a good idea).

This is causing my server to overload, because it is TONS OF REQUESTS! What should I do to stop this?

Access Log:

example.com 66.249.64.28 - - [21/Feb/2018:12:13:48 -0300] "GET /index.php/66t-2nkznwh_91f4690bjij1wbgziq- HTTP/1.1" 301 178 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Improve this question
4
  • I confirmed that that IP address does belong to Google, so it does appear to be the real Googlebot.
    – Stephen Ostermiller ♦
    Feb 21, 2018 at 16:07
  • How many requests per hour are we talking about here? It would probably more than 5,000 requests in an hour to overload a server by requesting redirecting URLs.
    – Stephen Ostermiller ♦
    Feb 21, 2018 at 16:09
  • Do these URLs show up in Google Search Console in the crawl error report? If so, that report might tell you why Google is crawling them. It might say where Googlebot found them.
    – Stephen Ostermiller ♦
    Feb 21, 2018 at 16:10
  • @StephenOstermiller It's a lot more of 5k request/hour. It's overloading my database server, because this URL is redirecting to Home Page and sending a lot API get requests. I don't know if it show up in google search console, I didn't have the google site verification property in my account for this domain, soon I will know about this. Thanks for the feedback
    – Arthur lauck do nascimento
    Feb 21, 2018 at 16:40

2 Answers 2

Reset to default
1
  • "What should I do". As an immediate action I'd set a rule in web-server config (e.g. .htaccess) to respond with 404 to that. 404 is in case you have no /index.php as a valid path on your server. Dong so will at least drop load from your interpreter (I assume it's PHP).
  • Next I'd put a rule in robots.txt to forbid such a path from indexing. It should completely stop Google from crawling those URIs and stop spending crawl budget on them.
  • After that I'd search for links to your site using one of those URIs. Who knows, maybe it will help you to find the reason where those links are coming to Google from. What if that's your own site?

That's it I think.

PS 301 is not a good idea I think. From my experience Bot will be coming back from time to time to confirm that redirection is still there. I guess that's not what you want. Moreover 404 is really fits better per definition.

a random URL that doesn't exist

Improve this answer
4
  • Google should already be getting a 404. The OP should confirm this for us. If this is the case, blocking using the robots.txt file will stop Google from seeing the 404 errors and the links will not get properly marked in the index. If the OP is seeing 404 errors for this, they should do nothing. Especially, they should never mark a proper 404 as fixed. This makes things worse.
    – closetnoc
    Feb 21, 2018 at 18:10
  • @Arthurlauckdonascimento Do not block these pages using robots.txt. See my comment above. Cheers!!
    – closetnoc
    Feb 21, 2018 at 18:12
  • Sorry I didn't post everything. See full comment below. I stopped redirecting non existing pages and started to throwing 404 errors. It would be a problem set a rule in nginx with regexp, because the token was random and not had the same characters count. Added to robots.txt: Crawl-delay: 25 Disallow: /index.php/ No idea where that link is coming from, for sure is not ours. It always has a random TOKEN in the end.
    – Arthur lauck do nascimento
    Feb 21, 2018 at 18:20
  • Google doesn't use the crawl delay directive in robots.txt, so that won't work. You might be able to use the fact that the tokens are long as part of a regex. Something like index\.php/[a-zA-Z0-9\-_]{25,}$
    – Stephen Ostermiller ♦
    Feb 21, 2018 at 18:49
0

I've found that Googlebot crawls URLs on my site that don't exist, doesn't have content and isn't linked from any pages. Studies have shown that it appears that Google is typing words into the search bars of websites and crawling the results of the search.

You can limit the crawl requests that Googlebot makes to your site in webmaster console.

If you feel that 301 redirecting this page back to the homepage isn't helping Google crawl your site, you can set the header status to 403 forbidden on that page. This will potentially stop Googlebot from going there. If it's in a specific directory, you can also disallow robots in robots.txt.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.