A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

-1
votes
3answers
18 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
2
votes
1answer
81 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
1
vote
1answer
42 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
1
vote
1answer
163 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
1answer
34 views

Trick to index infinite scroll content on Google?

I have a Question & Answer website, sort of like stackoverflow. When you open it, it displays 20 latest questions. When scroll to bottom of page, and it fetches 20 more, and so on, infinitely. ...
0
votes
1answer
76 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
1
vote
0answers
111 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
31 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
1
vote
0answers
60 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
0
votes
0answers
10 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
0
votes
0answers
18 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
0
votes
0answers
19 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
0
votes
0answers
451 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...