Crawling is a process associated with the actions of Bots. Search engine bots or scrapers will generally 'Crawl' a website to obtain data.
0
votes
2answers
150 views
GWT: reporting crawling errors for non existing links
Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed.
They all mention ...
3
votes
1answer
108 views
How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache
The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
1
vote
1answer
36 views
Webmaster tools reports fake 404 errors
I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
1
vote
1answer
53 views
Ajax crawling : snapshot page dissimilar with user page
For ajax-crawling purpose I make snapshots pages. My problem is user view pages and snapshot pages are dissimilar (tags h1, h2...).
Example:
user view page: http://www.meomap.net/#!/place/1
...
1
vote
1answer
50 views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
0
votes
1answer
149 views
Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?
Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites.
Do ...
2
votes
0answers
916 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
1
vote
0answers
28 views
Does Google bot always treat dot as word separator?
One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
