A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
2answers
29 views

Private folders which are not indexable

I have a requirement where I need to distribute about 400 copies of a pdf document (about 6 MB) and .pub files, that are personalized for each user. Every user must have access to only his/her set of ...
0
votes
2answers
440 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
39 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
1
vote
0answers
16 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
1
vote
0answers
56 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
0
votes
0answers
28 views

Stopping JavaScript savvy bots from getting into GA in the first place

I'm aware that bots can be filtered out at the profile level in GA, but I hadn't considered the effect of letting all that bogus data into the web property, and therefore ultimately counting against ...
0
votes
0answers
79 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
0
votes
0answers
351 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
0answers
122 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.