A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
2answers
54 views

How to make page content crawlable , which is requested in script from ajax on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? I went to Google Webmasters tool to ensure that whether Google crawler could see the content ...
2
votes
2answers
219 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
1
vote
2answers
308 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
3answers
7k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
0
votes
1answer
30 views

How long is the penalty for Duplicate ecommerce content after it has been ressurected

I am fixing all of the duplicate content on my ecommerce site with all orignal descriptions etc. How long does it take google to start ranking it again? I used to have a good ranking that converted ...
2
votes
2answers
549 views

What is Google's tolerance about cloaking with unwanted crawlers (fraudsters, scammers, etc)?

We have a classified ad website. Some fraudsters, scammers and kids crawl the content. Some can be: aggressive, running several EC2 with multi-threaded curl requests resulting in 100'000s requests ...
0
votes
2answers
374 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
0
votes
2answers
77 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
1answer
71 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
1answer
67 views

Can AdSense crawler view pages that require cookies?

Details I require users to agree to terms and conditions before they can view several pages on my site. Once they have agreed a cookie is set and they can proceed to the webpage. If a user somehow ...
1
vote
1answer
50 views

Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code. How can I go about diagnosing this issue? The website is eCommerce business using the Magento ...
1
vote
2answers
33 views

Google Webmaster Tools linked from lists dead links

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link ...
11
votes
2answers
799 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
0answers
38 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
1
vote
1answer
145 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
24 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
1
vote
3answers
46 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
0
votes
2answers
166 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
0answers
3 views

Will I get penalised by google for overflow hidden? [duplicate]

I have 2 divs next to each other, div A and div B. Div A displays a whole wordpress post and div B displays about 15 excerpts of posts with the same tag. To get this looking nice I set the hight of ...
12
votes
4answers
3k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
2
votes
2answers
64 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
3
votes
2answers
246 views

Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?

here is my htaccess file...... RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR] RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$ RewriteRule ...
1
vote
1answer
83 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
1answer
45 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
3
votes
2answers
136 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
1
vote
1answer
68 views

How to block a spam bot .htaccess proper formatting?

I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following; 88.80.205.219 - - ...
0
votes
1answer
47 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
1
vote
1answer
615 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
2
votes
2answers
40 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
6
votes
1answer
311 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
2
votes
4answers
397 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
0
votes
1answer
30 views

How to force search engines not to crawl my site, is there any special mark on them? [duplicate]

I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex. i dont really trust robot.txt. Because they can simply ignore it and continue crawling. I would ...
0
votes
1answer
61 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
3
votes
2answers
288 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
0
votes
1answer
44 views

Baidu spider does not obey DNS TTL

I moved my website from one VPS to another on 16th April. The A record of the domain name has a TTL of 86400. I made the IP address change at 19:00 GMT 16th April. I do not want the site to face ...
5
votes
3answers
101 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
0
votes
1answer
50 views

Why does bot traffic have a high (e.g. 100%) bounce rate in GA?

Since bots may request many pages at a time, why do they so often have a 100% bounce rate? It seem unlikely that they would only request one page, then leave for over 30 minutes, then request another ...
0
votes
2answers
59 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?
1
vote
1answer
65 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
1
vote
2answers
86 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
1
vote
4answers
5k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
1
vote
1answer
63 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
1
vote
1answer
37 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
6
votes
1answer
162 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
7
votes
1answer
89 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
3
votes
5answers
3k views

Why deny access to website for msnbot/bingbot?

I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
1answer
41 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
0
votes
2answers
96 views

How do you determine when to block an IP address or range of addresses?

When analyzing traffic to some of my sites, I almost always notice a large amount of traffic from obscure sources like Japan, Russia, and other countries that really have no business visiting my ...