Forgive me if this is in the wrong area - I thought considering the topic, it belonged here (as opposed to in WP). Correct me or move it if I am wrong.

I've been looking around one of the sites I control (one we've been having all kinds of little problems with) and checked my 404 logs. Usually I expect to see the generated captcha images and other random entries, but today I saw several of the site's pages on there with "/(" appended at the end, all with "PHPCrawl" as the User Agent.

As far as I can figure, this is an open source script available for developers to use at their discretion. I've not used anything of the sort. I don't believe anyone else in control of the site has.

Is it possible one of the plugins could have caused these entries? Is it automated from some search engine? Is it something I should be worried about hurting the site or its SEO?

  • It is unlikely that a WP plugin would need to crawl the site (it'd have db access anyway so there is really no need). Still, it isn't impossible. To rule that out lets see a list of your installed plugins.
    – krowe
    Sep 23 '13 at 17:59
  • @krowe I think it's unlikely too, especially since I use most of these elsewhere, but as I said, it's already been a problem site and I just want to cover all my bases... - Akismet (disabled) - Conditional Widgets - Contact Form 7 - Contact Form DB - Easy WP SMTP - Google Analytics - Google XML Sitemaps - Really Simple CAPTCHA - SEO Ultimate - Slideshow - WP Customer Reviews Sep 23 '13 at 18:20
  • SEO Ultimate looks suspiciously like something which may be crawling the site. Google Analytics obviously crawls your site too but it doesn't use PHPCrawl.
    – krowe
    Sep 23 '13 at 18:28
  • @krowe AFAIK, SEO Ultimate is pretty legit & pretty widely used. This is one I use across several sites and hasn't posed any problems. It mostly helps with title/meta rewriting and certain header/footer inserts & other tags without having to edit code. Sep 23 '13 at 19:19
  • I wasn't suggesting otherwise. Even legit apps can be misconfigured or have bugs for whatever reason. Of the extensions you've listed it is the only one I can see even possibly doing this. It is just as likely (if not more so) that the bot running this was not yours at all. You can check that by looking at your access logs for the requests made by this bot. If they are not originating from one of your IPs then it probably has nothing to do with your setup.
    – krowe
    Sep 23 '13 at 20:16

Is it possible one of the plugins could have caused these entries?

Highly unlikely unless one of your plugins is attempting to index the site. But the plugins have access to the wp-posts table so there isn't any need to spider via the front end.

Is it automated from some search engine?

Potentially, but not one of the major ones as those Spiders are easily and readily identifiable in your logs. I can't think of any legitimate search engine that would use PHPCrawl. You should start tracing the IP addresses that are causing the 404 errors.

Is it something I should be worried about hurting the site or its SEO?

There's not really enough information to answer this definitively. Someone is crawling your site via a script but there's no way to know why or to what purpose they will use the data. Should you be worried? Probably not...this kind of stuff happens all the time. If the IP address proves to be from an area that you are not concerned about, add a deny line to your .htaccess file and block them. Will it affect SEO? Again, most likely not. It is remotely possible that someone is spidering pages to create a spammy page of links but careful checking of your analytics will reveal any weird backlinks and you can then handle those easily enough.


The situation bears watching but is unlikely to be a problem.

  • Thanks, could you point me in the right direction for tracing these things? I'm just the developer, so I don't have access to other tracking things (Analytics) - only pass the coded theme along and get blamed for any problems that follow. :\ I can use plugins & coding from within the site itself. I don't know much about HTTP responses & such. (I feel like that sounded really dumb) Sep 23 '13 at 19:21
  • Often you can look up the user-agent or IP address in online user-agent databases like this one, and then click on the resulting IP's for WHOIS information. In this case, it's a generic user-agent for this script, which could be used by anyone. No cause for concern.
    – dan
    Sep 24 '13 at 3:11
  • Another good site to research potentially sketchy IP addresses would be stopforumspam.com
    – JCL1178
    Sep 24 '13 at 3:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.