A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
3answers
16 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
2
votes
1answer
78 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
1
vote
1answer
141 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
1answer
66 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
2
votes
0answers
29 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
0answers
102 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
28 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
1
vote
0answers
60 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
0
votes
0answers
16 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
0
votes
0answers
18 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
0
votes
0answers
439 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...