Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
1answer
26 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
0
votes
1answer
40 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
0
votes
1answer
18 views

404 code/header for search engines, on removed user content?

I just got an email, from a former user on my website He was complaining that Google still shows the contact page he created on my site, even though he deleted it a month ago This is the first time ...
0
votes
1answer
27 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
0answers
20 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
1
vote
1answer
49 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
1
vote
1answer
21 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
4
votes
5answers
139 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
1
vote
1answer
28 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
2
votes
2answers
52 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
0
votes
0answers
18 views

Confused about logfiles with googlebot

Im still in the process of making my site crawlable through google even though its all AJAX content. So I got it where if the URL contains 'escaped_fragment_ it retrieves the right information form ...
2
votes
1answer
33 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
0
votes
0answers
15 views

Searching for Googlebot in server logs [closed]

I'm trying to use Heorku Logentries to track Googlebot's activity on my server. (Although, I think this could broadly apply to tracking Googlebot activity on any server.) I can see that I have results ...
0
votes
0answers
7 views

Links removed from google search results [duplicate]

For my website www.trickwarehouse.com i have created one post on keyloggers. When i search in google that what are keyloggers? then my website's post position was 16 but after 1 hour, when i checked ...
0
votes
1answer
50 views

google webmaster tools, site maps and indexing, number of pages indexed reduces with no new sitemap

It seems to take two weeks ish to get updates on to the google index, which seems fair enough as I imagine googles index of the whole internet could be rather large. However, the some changes to my ...
2
votes
1answer
68 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
1
vote
2answers
47 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
17 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
0answers
38 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
810 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
40 views

Does Fetch as Googlebot still support their ajax-crawling proposal?

I spent half a day implementing the server side html generation for modal pages based on their proposal (link), but it seems like the Fetch as Googlebot functionality in Webmaster tools completely ...
2
votes
2answers
83 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
1answer
44 views

What if my HTML snapshot for Googlebot doesn't contain the same content as the dynamically created page?

According to Google here: The HTML snapshot must contain the same content as the dynamically created page. What if my returned HTML Snapshot doesn't contain the same content as the actual ...
8
votes
2answers
70 views

Informing Googlebot for deprecated pages

I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a ...
1
vote
1answer
26 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
0answers
45 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
6
votes
4answers
149 views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
3
votes
3answers
44 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
1
vote
1answer
48 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
2answers
66 views

SEO penalty for 301 redirecting googlebot only?

I have a website that is splitting into two. For the sake of easily describing the what and why, let's say I run a classifieds site and I want to move all the adult content to its own domain. (very ...
3
votes
1answer
43 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
0
votes
1answer
82 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
0
votes
1answer
84 views

How do I find when Googlebot will visit(crawl) my pages again?

I implemented the unavailable_after tag in my pages like the below and set to unavailable after 3 days but in Google Webmaster Tools I got 404 for that page. <META NAME="GOOGLEBOT" ...
1
vote
2answers
63 views

Regarding Google Index -> Remove URLs

I have a question regarding Google index -> Remove URLs from Google Webmaster Tools. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't ...
6
votes
2answers
82 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
2
votes
1answer
37 views

Google not indexing my Mediawiki's jpg files description pages

My website es.wikineos.com has been running for a few years with the MediaWiki Content Management System like the one they use at Wikipedia. Google seems to be indexing my site correctly at a quick ...
0
votes
1answer
84 views

How to get Google to index dynamic pages generated from user IP address

I am creating a local community directory site which identifies the location of the user by IP address and dynamically renders the content related to the specific community identified by the IP ...
2
votes
1answer
136 views

How to tell Google my content will expire after a certain period? [duplicate]

Right now I'm facing issues with 404 (Not Found) errors, because my content will deleted automatically after two months. I cannot change this behavior, because it is a real estate web site and once it ...
5
votes
1answer
90 views

How to deal with non-relevant keywords in Google Webmaster Tools

Google webmaster tools is showing keywords such as "cookies" in my keyword list. This is probably because we have links to our long legal disclaimer on cookies. We're a B2B service, so clearly I ...
4
votes
2answers
578 views

Hijacked meta description

On my site: faithhopeandfiction someone hacked into my website and now it's displaying wrong Meta description on Google search results. I've verified this using "view source" that meta description and ...
2
votes
2answers
64 views

Google search results not showing my events rich snippets but does in test tool

When I search on "tennessee basketball tournaments", I see my result, but my events dont show up. I believe I did everything right since this is an ajax page, but I tell Google that ...
1
vote
1answer
26 views

Older post not indexed, new post indexed right away?

I have a post I created yesterday that was indexed rather quickly by Google. However, I have two other post that I created within the past 4 days that are not indexed and I cannot find a trace of ...
0
votes
0answers
42 views

Googlebot can't access my site

For some reason, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt everything checks out, so I'm not sure ...
1
vote
1answer
20 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
0
votes
2answers
60 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
1
vote
2answers
68 views

Does the Instant Preview in Google webmaster tools takes Robot.txt in account?

Is that the way to go If I want to visually see what the googlebot see? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as google bot, It fetches ok, ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
0
votes
1answer
39 views

Can Googlebot read CamelCase file names?

This has been a question I have been wondering for quite some time now. Can Googlebot, and other search engine crawlers, read file names (such as images) that are in CamelCase? For example, if I ...
2
votes
2answers
102 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
1
vote
1answer
64 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...