A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
2answers
71 views

BingBot hitting multiple subdomains all at the same time, causing panic

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this: 01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A 01:23:11 ...
0
votes
1answer
32 views

AOL search engine bot name?

Can somebody tell me the name of the AOL search engine bot? I need to allow only this bot in robots.txt.
0
votes
2answers
22 views

What to do about Crawler errors after capital revamp of my website?

So, recently I was re-building my website, putting on another CMS, optimizing and so on... And now I wisited Google Webmaster tools - and it tells I have a lot of Web crawler 404 not found Errors... ...
0
votes
1answer
39 views

How long is the penalty for Duplicate ecommerce content after it has been ressurected

I am fixing all of the duplicate content on my ecommerce site with all orignal descriptions etc. How long does it take google to start ranking it again? I used to have a good ranking that converted ...
2
votes
2answers
82 views

How to make page content crawlable , which is requested in script from ajax on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? I went to Google Webmasters tool to ensure that whether Google crawler could see the content ...
0
votes
2answers
407 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
0
votes
1answer
41 views

DNS mistake. Google crawled the wrong server. What to do?

Problem Exists Between Keyboard And Chair I have a couple of similar looking domains on Coudflare where I manage their DNS. An old forum and my new still MVP site. As many of you may have ...
0
votes
2answers
110 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
1answer
78 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
1answer
99 views

SEO preparation for upcoming new site

I booked a domain and I created an image saying that the site is soon to be launched. What SEO preparations should I make to avoid harming my SEO ranking? Because my page is blank, the image itself ...
1
vote
1answer
77 views

Can AdSense crawler view pages that require cookies?

Details I require users to agree to terms and conditions before they can view several pages on my site. Once they have agreed a cookie is set and they can proceed to the webpage. If a user somehow ...
1
vote
5answers
6k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
0
votes
1answer
25 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
2
votes
2answers
91 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
0
votes
1answer
74 views

Will making my AJAX app serve HTML snapshots improve SEO?

Currently, a small ajax web app of mine uses an anchor hijacking method to display content. The user clicks an anchor, I cancel the default browser event and funnel the link into an AJAX call, after ...
1
vote
1answer
633 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
1
vote
3answers
2k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
1
vote
2answers
98 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
0
votes
2answers
772 views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
2
votes
2answers
305 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
1
vote
2answers
358 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
3answers
8k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
2
votes
2answers
567 views

What is Google's tolerance about cloaking with unwanted crawlers (fraudsters, scammers, etc)?

We have a classified ad website. Some fraudsters, scammers and kids crawl the content. Some can be: aggressive, running several EC2 with multi-threaded curl requests resulting in 100'000s requests ...
1
vote
1answer
65 views

Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code. How can I go about diagnosing this issue? The website is eCommerce business using the Magento ...
1
vote
2answers
43 views

Google Webmaster Tools linked from lists dead links

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link ...
11
votes
2answers
819 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
1
vote
1answer
266 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
29 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
1
vote
3answers
60 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
0
votes
2answers
178 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
0answers
3 views

Will I get penalised by google for overflow hidden? [duplicate]

I have 2 divs next to each other, div A and div B. Div A displays a whole wordpress post and div B displays about 15 excerpts of posts with the same tag. To get this looking nice I set the hight of ...
13
votes
4answers
3k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
3
votes
2answers
253 views

Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?

here is my htaccess file...... RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR] RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$ RewriteRule ...
1
vote
1answer
98 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
1answer
50 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
3
votes
2answers
151 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
1
vote
1answer
107 views

How to block a spam bot .htaccess proper formatting?

I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following; 88.80.205.219 - - ...
0
votes
1answer
49 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
2
votes
2answers
40 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
6
votes
1answer
317 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
2
votes
4answers
447 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
0
votes
1answer
31 views

How to force search engines not to crawl my site, is there any special mark on them? [duplicate]

I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex. i dont really trust robot.txt. Because they can simply ignore it and continue crawling. I would ...
0
votes
1answer
82 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
3
votes
2answers
358 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
0
votes
1answer
65 views

Baidu spider does not obey DNS TTL

I moved my website from one VPS to another on 16th April. The A record of the domain name has a TTL of 86400. I made the IP address change at 19:00 GMT 16th April. I do not want the site to face ...
5
votes
3answers
131 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
0
votes
1answer
58 views

Why does bot traffic have a high (e.g. 100%) bounce rate in GA?

Since bots may request many pages at a time, why do they so often have a 100% bounce rate? It seem unlikely that they would only request one page, then leave for over 30 minutes, then request another ...
0
votes
2answers
63 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?