The site-crawler tag has no wiki summary.
0
votes
1answer
17 views
Prevent Advert Images Appearing in Search Spider Crawls
While testing out how web pages on my site appear to search engine spiders using http://www.browseo.net/ I noticed that advert images appear in the view. However, when I test a site like ...
3
votes
0answers
344 views
What is a good open source web crawler? [closed]
I'm looking for a good open source web crawler and i found these:
DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
2
votes
2answers
86 views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
0
votes
3answers
73 views
Free standalone crawlers
Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors?
Something that shows me all 404's, errors etc
5
votes
1answer
144 views
What is User-Agent “AF_ID=<xxxxxx>”
We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like "AF_ID=". I've looked around for common ...
0
votes
1answer
211 views
Link error checking tool for linux [duplicate]
I use Xenu Kubj Sleuth to find out 404s and server errors on websites, but I'm not enjoying it, is there a better free tool out there that can preferably run in bash in linux that does the same thing ...
2
votes
2answers
207 views
Tool to generate list of redirections (sitemap for .htaccess)
Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site?
I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
1
vote
1answer
288 views
Web crawler tool that generate a list of all the anchors used on a site
I'm looking for a web crawler tool that'll go through all the anchor text on a whole web site and generate a list for me to check. I want to make sure all the anchors are really descriptive outside ...
0
votes
2answers
262 views
If an visitors IP address contains “google” or a similar keyword, does this mean they were a crawler?
I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, ...
2
votes
3answers
738 views
What user agent should I set?
There is Ask bot, which sets this header:
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Considering this, I have the following questions:
If I'm writing a web crawler named Goofy, what user agent ...
0
votes
2answers
145 views
Where can I host this?
I'm looking for a VPS or dedicated that can have a web crawler running all day long. It would gather data and then add it to some websites.
These websites currently have around 1k unique visits/day, ...
2
votes
3answers
315 views
Blocking all search engines except the big ones
I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and ...
18
votes
6answers
998 views
Good tool to crawl my site and help me find dead link and unlinked files
I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
