Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I recently ran Xenu link checker on my site. I link to several Wikipedia pages and ever wiki page generates a 403 code. Here is an example of the message:

http://en.wikipedia.org/wiki/Alaska
    error code: 403 (forbidden request), linked from page(s):
http://www.mysite.com/page.html

I know a 403 error code is:

A 403 Forbidden is an HTTP status code returned by a web server when a user 
requests a web page or media that the server does not allow them to. In other 
words, the server can be reached, but the server declined to allow access to 
the page. 

My question is why does it generate that error? I can click the page without issue. Does it have something to do with the checker running scans?

share|improve this question
add comment

1 Answer 1

up vote 3 down vote accepted

It's probably because Xenu is trying to pull up the page to verify the link is valid but Wikipedia is blocking it because it is an unauthorized bot. They probably are blocking a long list of crawlers and bots that are not related to search engines (i.e. link checkers, spammers, scrapers, etc).

share|improve this answer
    
Okay, that makes since. Thanks for the information. –  Lynda Oct 17 '11 at 20:00
    
User-agent: Xenu Disallow: / - EN.Wikipedia.org robots.txt –  danlefree Oct 18 '11 at 1:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.