Return to Question

replaced http://webmasters.stackexchange.com/ with https://webmasters.stackexchange.com/

Source Link

edited Apr 13, 2017 at 12:33

I'm very disappointed that Googlebot is apparently ignoring my robots.txt.

I have the following sole entry for my /robots.txt:

    location = /robots.txt {
        return  200
"## $host ##
\n
#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile\nDisallow: /
\n
";
    }

It shows up as follows:

## constantine.su ##


#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile
Disallow: /

Yet now, years after having had to adopt the robots.txt as above years after having had to adopt the robots.txt as above, which always worked, I'm now repeatedly getting the following misleading User-Agent entries in my access_log:

66.249.66.163 - - [20/May/2016:07:19:50 -0700] "GET / HTTP/1.1" 200 3314 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

What's going on? Where do I report these misbehaving bots and Google's clear abuse of the crawling privilege?

I'm very disappointed that Googlebot is apparently ignoring my robots.txt.

I have the following sole entry for my /robots.txt:

    location = /robots.txt {
        return  200
"## $host ##
\n
#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile\nDisallow: /
\n
";
    }

It shows up as follows:

## constantine.su ##


#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile
Disallow: /

Yet now, years after having had to adopt the robots.txt as above, which always worked, I'm now repeatedly getting the following misleading User-Agent entries in my access_log:

What's going on? Where do I report these misbehaving bots and Google's clear abuse of the crawling privilege?

I'm very disappointed that Googlebot is apparently ignoring my robots.txt.

I have the following sole entry for my /robots.txt:

    location = /robots.txt {
        return  200
"## $host ##
\n
#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile\nDisallow: /
\n
";
    }

It shows up as follows:

## constantine.su ##


#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile
Disallow: /

Yet now, years after having had to adopt the robots.txt as above, which always worked, I'm now repeatedly getting the following misleading User-Agent entries in my access_log:

What's going on? Where do I report these misbehaving bots and Google's clear abuse of the crawling privilege?

Source Link

asked May 21, 2016 at 18:12

cnst

Googlebot-Mobile ignoring robots.txt, pretending to be Googlebot

I'm very disappointed that Googlebot is apparently ignoring my robots.txt.

I have the following sole entry for my /robots.txt:

    location = /robots.txt {
        return  200
"## $host ##
\n
#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile\nDisallow: /
\n
";
    }

It shows up as follows:

## constantine.su ##


#Dear Google, we do not appreciate fake User-Agent strings 
#that span 3 lines, and quadruplicate requests per page.
User-agent: Googlebot-Mobile
Disallow: /

Yet now, years after having had to adopt the robots.txt as above, which always worked, I'm now repeatedly getting the following misleading User-Agent entries in my access_log:

What's going on? Where do I report these misbehaving bots and Google's clear abuse of the crawling privilege?