Tagged Questions
2
votes
1answer
75 views
Is there a spider / link checker that can start deep inside a login-protected site
We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
2
votes
3answers
309 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
3
votes
1answer
481 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
