Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
362 views

How to make image only website friendly to googlebot? [duplicate]

so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created. ...
3
votes
2answers
112 views

Is offering Google Bot a lite version of site skin considered cloaking?

If you have multiple templates a user can choose from on a site (like a forum), and you offer Google Bot a "lite" skin, is this considered cloaking? Or is this considered okay? The actual content ...
2
votes
3answers
705 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
3
votes
2answers
752 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
2
votes
2answers
200 views

Get google to index JavaScript pages as separate pages

I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
2
votes
1answer
185 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
2
votes
1answer
63 views

multilingual domains both crawled and come up to the wrong audience

I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.: domain.com ja.domain.com es.domain.com I'm ...
1
vote
2answers
462 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
2answers
113 views

I don't think my site is being detected/crawled properly/having backlinks, and it's ruining SEO [closed]

This is really frustrating. I'm setting up a computer repairs business called Darley Tech. I have listed my site on many online directories to encourage backlinks. I have relevant and useful ...
2
votes
2answers
219 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
1
vote
2answers
384 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
6
votes
2answers
233 views

How frequently does Googlebot fetch sitemaps? Does it depend on PageRank?

How frequently does Google fetch sitemaps? I am now working with a high traffic website that normally has 30 new posts per minute. It currently it provides a sitemap which only includes the 100 ...
0
votes
1answer
73 views

How do I get Google to crawl my content when it's only displayed when you fill in a form?

I have a webpage. It has a form and the "results" section is blank. When the user searches for items, and a list that pops up, he/she chooses one option from list and then the corresponding results ...
1
vote
1answer
146 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
2answers
348 views

How can I tell GoogleBot that a subdirectory is now a subdomain?

I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
2
votes
1answer
348 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
1
vote
3answers
3k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
0
votes
0answers
18 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
2
votes
4answers
741 views

Google bots are severely affecting site performance

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on ...
1
vote
0answers
179 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
5
votes
2answers
243 views

Robots.txt and pattern matching

Adding this to my robots.txt User-agent: * Disallow: /*action=*$ How does robots not recognizing wild cards handle this?
2
votes
1answer
2k views

Google on capital letters in url

I created one of my websites with the following url structure: http://domain.com/Animal/Pig http://domain.com/Pigs If I change the url map to lower case letters instead, will google reindex those ...
1
vote
2answers
200 views

When will my old page stop appearing on Google? [duplicate]

Possible Duplicate: Old pages still appearing in Google? I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
3
votes
1answer
480 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
2
votes
2answers
89 views

Forbid content in google results summary

I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages. Each page have an adress, some content and ...
2
votes
1answer
1k views

How to tell Google to stop crawling my website via IP?

After a database server migration, I've noticed that GoogleBot started getting errors when trying to crawl my site. The reason seems to be that it is hitting my site via IP address (and due to my ...
6
votes
1answer
3k views

link rel=“alternate”, multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
0
votes
2answers
158 views

Hide elements for Google

I have some elements which I don't want Google to see. Usually repetitive content, e.g. header and sidebar. As Googlebot doesn't handle AJAX requests, I load these resources with AJAX and insert them ...
0
votes
4answers
251 views

Hide your website links from search engne

My Website Link is posted in someone else website, i asked for removal but can't get reply, google crawl his website and my website link is posted on that how to control that is that is possible? i ...
1
vote
1answer
172 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
0
votes
2answers
610 views

When using Mod_rewrite does google bot see the original url or the rewritten url?

If i use the bellow code in my .htacess file would google bot (or any other search spiders) see the example.tumblr.com url or would they see the same url as the user ie. example.co.uk/blog ? ...
0
votes
2answers
134 views

Anyone heard of 'tank tracking'?

Someone I know told me about some seriously cutting edge blackhat SEO that he called 'tank tracking'. He said that it is some sort of code (he believes written in Python) that 'sits' around the ...
4
votes
1answer
231 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
-3
votes
2answers
149 views

Does Google bot crawl unused tags?

If a website a called foo.com has the following css, #LinkBuilder{ background:url("www.LoremEpsum.com"); } and the css id LinkBuilder is actually not getting used anywhere in the html of foo.com, ...
2
votes
1answer
405 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
1
vote
2answers
95 views

Is using Progressive enhancement, overriding links with .js considered cloaking and can it be see by search engines?

I'm using .load to load content from a page /about.html, so that search engines can see what is going on. I've made the links that trigger the .load <a> tags to the page /about.html but for ...
2
votes
1answer
585 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
3
votes
1answer
188 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
2
votes
2answers
1k views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
0
votes
1answer
443 views

How does Google feel about overflow: auto?

I am planning to add some keyword rich text to my homepage at the very top. I have written almost a mini essay and putting all this text at the very top left of my page wouldn't look too nice. I was ...
4
votes
2answers
935 views

Does duplicate content inside your own website matter for SEO? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm not referring to the duplicate content where you copy content from another site and ...
2
votes
2answers
280 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
1
vote
1answer
134 views

Could this be considered duplicate content? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm building a dating site at the moment. On the homepage I have a paginated list of all ...
2
votes
2answers
415 views

Is hiding content with JavaScript or “text-indent: -9999px” bad for SEO?

So apparently hiding content using "display: none" is bad for SEO and seen by googlebot as being deceptive. This according to a lot of the posts I read online and questions even on this site. But ...
2
votes
1answer
284 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
4
votes
1answer
1k views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
5
votes
3answers
534 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
2
votes
2answers
1k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
1
vote
2answers
245 views

sitemap and crawler

I am still newbie in the entire world of SEO. Let's take a look at my example. I have a website very similar to blog style site. There few patterns for urls, all of the are very simple and allowed to ...