Tagged Questions

0
votes
1answer
40 views

Google adsense crawler unable to access your page [closed]

I have a problem which I have been trying to solve by Googling it but it seems I can’t get no answer. I have been using plain PHP and HTML on my site and everything was ok concerning my Google ...
3
votes
0answers
348 views

What is a good open source web crawler? [closed]

I'm looking for a good open source web crawler and i found these: DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
0
votes
0answers
17 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
2
votes
3answers
309 views

User-agent identification and SEO crawler database

I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values. I know there are 'trap' links that you can use to ...
2
votes
2answers
72 views

Is there a weight for html link?

I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google? 1. <a ...
0
votes
3answers
73 views

Free standalone crawlers

Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors? Something that shows me all 404's, errors etc
3
votes
1answer
481 views

Why is BingBot causing so many 404 errors by removing letters from url's?

Since the ForceRecrawl problem went away Bing has come back with some new trends. I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
2
votes
0answers
916 views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?

So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
1
vote
1answer
61 views

Does a Registrar forwarding your Domain to your hosting site url, violate Google AdSense TOS?

NameCheap offers to forward traffic to a url of your choosing, which is really handy if you are using a free web hosting service, and/or ever want to move to try another free host someday. However, I ...