10

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map.

Recently we had begun to notice that the site was hitting the free map search limit (roughly 25,000 per 24 hours) without a notable increase in overall traffic. I turned on some additional logging to try and find what the problem was.

It turns out that Googlebot is pushing through thousands of searches on this map. This is a small sample:

2017-07-09 23:56:22,719 [7] INFO  ShopLanding - [Thread 41] Google Maps: searched G23 received OK from 66.249.66.221
2017-07-09 23:56:35,469 [7] INFO  ShopLanding - [Thread 10] Google Maps: searched CA6 received OK from 66.249.66.221
2017-07-09 23:57:24,563 [7] INFO  ShopLanding - [Thread 48] Google Maps: searched BN14 received OK from 66.249.66.223
2017-07-09 23:58:00,970 [7] INFO  ShopLanding - [Thread 42] Google Maps: searched CB4 received OK from 66.249.66.221
2017-07-09 23:58:13,064 [7] INFO  ShopLanding - [Thread 54] Google Maps: searched DY9 received OK from 66.249.66.221
2017-07-09 23:59:18,722 [7] INFO  ShopLanding - [Thread 59] Google Maps: searched TS3 received OK from 66.249.66.223
2017-07-09 23:59:53,223 [7] INFO  ShopLanding - [Thread 49] Google Maps: searched S45 received OK from 66.249.66.221

Is there a way that I can stop Google from pushing through so many requests? This is eating a significant proportion of the free allowance. Legitimate searches seem to be under about 200 per day.

EDIT

The site is built on C# ASP.NET. The store search is using POST, the URL doesn't change on submit. I can post a sample of IIS logs tomorrow morning to confirm this behaviour.

Improve this question
7
  • Do the items Googlebot is searching for have any meaning? "G23" and "CA6" don't mean anything to me. Googlebot doesn't usually submit random data to forms. It usually only crawls drop down options, pre-filled data, or query links. Are those values on the site somewhere as a list? Also, what do your URLs look like for these requests?
    – Stephen Ostermiller ♦
    Commented Jul 10, 2017 at 12:48
  • @StephenOstermiller they are partial UK postcodes, specifically the district identifier. These should be POST requests for AJAX, I will check the IIS logs though.
    – Burgi
    Commented Jul 10, 2017 at 13:10
  • Aside: What "free map search" is this?
    – MrWhite
    Commented Jul 10, 2017 at 14:57
  • 1
    @SamWeaver the IP resolves as Google bot and the corresponding IIS log entry has the Googlebot UserAgent. Admittedly both of those can be spoofed.
    – Burgi
    Commented Jul 10, 2017 at 18:31
  • 1
    Does your store locator use query string (GET request)? What is rough URL structure of your store locator page, does it change (query string) with user search? If you can explain the functional / tech side bit more in detail then it can be answered in a more effective way.
    – TopQnA
    Commented Jul 10, 2017 at 23:08

1 Answer 1

Reset to default
11

To stop googlebot from searching via googlemaps put a file named robots.txt in the root of your domain. e.g. https://www.wikipedia.org/robots.txt

Sample robots.txt:

User-agent: Googlebot
Disallow: /search-store/

Where /search-store/ is the page that sends the request to google maps.

If it happens to be something else than Googlebot, you can try disabling all crawling to this page with :

User-agent: *
Disallow: /search-store/

Note that it won't stop misbehaving scripts that ignore robots.txt.

Improve this answer
6
  • 10
    Note that you don't want Google and other search engines spidering these pages, anyway!
    – Ari Davidow
    Commented Jul 10, 2017 at 15:31
  • 1
    I am not sure if the blocking the entire search store is the right option, we might need to understand the query string etc. Meaning, let Google read but don't send too many requests. Blocking entire search store should be the last option.
    – TopQnA
    Commented Jul 10, 2017 at 23:04
  • Rate limiting search might be another option, let's say after 25 searches, you need to cool down for an hour between each search. It would stop bots and nefarious people, but not regular users.
    – satibel
    Commented Jul 11, 2017 at 5:00
  • If the page is just for searching a store's location, I doubt it would be useful to have it indexed in the first place. It would depend on the site structure. Also robots.txt is easier/faster (thus cheaper) to implement as a quick fix than taking some hours to rate limit. @TopQnA
    – satibel
    Commented Jul 11, 2017 at 5:08
  • Well, store location is very important for the user and most people search for a store with reference to the location in Google and if store locator can generate a map with useful info with unique URL for each store then it would be a much more useful for the user. And that is why blocking should be the last option.
    – TopQnA
    Commented Jul 11, 2017 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.