Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers:

X-Robots-Tag: noindex, nofollow

But once in a while I see this in the page logs:

HOST: 208-115-111-71-reverse.wowrack.com
USER_AGENT: Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)

What is that wowrack.com? And how do I stop it crawling my pages?

share|improve this question
    
You probably want to lock the page down a bit more than just robots.txt or robots meta tag if it's used for financial transactions. Both of those are not replacements for authentication. –  John Mueller Nov 27 '13 at 18:20
    
@JohnMueller: No, obviously the page itself uses the internal login process to connect to PayPal, etc. and if it's loaded by a crawler it fails. I just don't want to run this authentication for nothing. –  MikeF Nov 28 '13 at 7:31
add comment

2 Answers

up vote 5 down vote accepted

The reason is that your directive in the X-Robots-Tag is for indexation, not crawling.

[EDIT] Explicit reference to this point is made here: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

This document details how Google handles the page-level indexing settings allow you to control how Google makes content available through search results. You can specify these by including a meta tag on (X)HTML pages or in an HTTP header.

(emphasis mine)

[/END EDIT]

To prevent crawling of this page, you should consider blocking this specific page using an exclusion in your robots.txt file. More information here: http://www.robotstxt.org/robotstxt.html

If this doesn't work (as not all crawlers respect this file), then you can look at blocking by IP or domain.

share|improve this answer
1  
In my opinion, if the crawler doesn't respect the X-Robots-Tag, the robots.txt directive won't work either. –  Zistoloen Nov 26 '13 at 15:37
    
Not all crawlers are that sophisticated, but your point is valid that a brute-force block for some crawlers is the only alternative when they ignore directives. –  Mike Hudson Nov 27 '13 at 2:29
add comment

Several web crawlers don't respect the X-Robots-Tag in HTTP headers. I think this is the case for wowrack.com.

To block crawling of your page for this web crawler, you can use .htaccess (if you use Apache as a web server). Add these lines to block IP address in your .htaccess file:

order allow,deny
deny from 208.115.111.
allow from all

To be more aggressive, you can also block the hostname but it's not the better solution:

order allow,deny
deny from wowrack.com
allow from all
share|improve this answer
    
Thanks. Just curious, what is that wowrack.com? –  MikeF Nov 25 '13 at 23:39
1  
It's a cloud hosting company. It looks like someone is running a spider script from one of their accounts. You might consider blocking an IP range instead (deny from 208.115.111.). –  dan Nov 26 '13 at 0:52
    
I wouldn't use deny from <hostname> directives. They slow the server down (reference). Instead, I would suggest denying based on user agent as described here –  Stephen Ostermiller Nov 26 '13 at 1:39
1  
@StephenOstermiller I'm only seeing this in that reference: Denying by IP is much less server intensive than denying by host. Did you see the warning in the second link: Access control by User-Agent is an unreliable technique, since the User-Agent header can be set to anything at all, at the whim of the end user. There are quite a few security modules that would be more efficient, and blocking the unwanted/abusive IP's using a firewall is even more so, but adding an IP range to the config file is probably the easiest for the OP to do. –  dan Nov 26 '13 at 3:21
1  
I agree with dan on this case, I didn't realize that Wowrack was a web hosting company. I don't think blocking user-agent can be useful on this case, especially if it's a running script. I have edited my answer. –  Zistoloen Nov 26 '13 at 8:43
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.