A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
3answers
47 views

Google crawl some pages, but not exist on website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as "vy81TYYP/lraynb-20151123-dkd04-2290.html" page which has never existed on a my developed website. This ...
-1
votes
3answers
24 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
3
votes
2answers
30 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but ...
2
votes
1answer
133 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
16 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's ...
0
votes
1answer
26 views

How google crawls webpages without actual page files

For example, I have an index.php file on my server, google will crawl this and retrieve the information. I also have a file named product.php, this dynamically gets content from a database to ...
2
votes
0answers
220 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
52 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
143 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
1
vote
0answers
59 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - ...
1
vote
0answers
36 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is ...
1
vote
0answers
55 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
1
vote
0answers
66 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
0
votes
0answers
11 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
0
votes
0answers
42 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or ...
0
votes
0answers
25 views

Fetch as Google AngularJS Duplicate Entries

I have pages built in AngularJS that when I fetch as Google in WMT or when I switch my user-agent to Googlebot using a browser extension, duplicate entries are displayed. For example, if there is an ...
0
votes
0answers
220 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
0answers
115 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
0
votes
0answers
29 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
0
votes
0answers
21 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
0
votes
0answers
576 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...