Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

6
votes
2answers
76 views

Base64 encoded images and availability of their metadata for googlebot

If i embed an image into a page as an img src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for google imagebot?
0
votes
2answers
53 views

Drop in traffic but increase in pages crawled per day

I saw a drop in traffic of my website in early September 2014. But, if I check the "Crawl Stats" in Google Webmaster Tools, I see an increase in "Pages Crawled per day" and "Kilobytes downloaded ...
1
vote
1answer
26 views

Is it possible to include news into Google News if there is a non-news content under the same section?

We want to include our news into Google News, and now performing research how to properly do this. At the moment we have the same base URL for news and non-news articles (analysis, interviews, etc.), ...
0
votes
1answer
20 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
4
votes
3answers
175 views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; ...
0
votes
2answers
33 views

Does Googlebot (and/or search engines) index a forwarded page? [duplicate]

Let's say I have example.com domain, and I force the user to use the HTTPS over HTTP. The question is as browsers just accept and load the forwarded/new page (when the request for http://example.com ...
1
vote
3answers
61 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
0
votes
2answers
51 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
0
votes
1answer
19 views

404 code/header for search engines, on removed user content?

I just got an email, from a former user on my website He was complaining that Google still shows the contact page he created on my site, even though he deleted it a month ago This is the first time ...
0
votes
1answer
27 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
2answers
45 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
1
vote
1answer
57 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
1
vote
1answer
21 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
4
votes
5answers
142 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
1
vote
1answer
32 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
2
votes
2answers
66 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
1
vote
0answers
20 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
2
votes
1answer
54 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
0
votes
0answers
15 views

Searching for Googlebot in server logs [closed]

I'm trying to use Heorku Logentries to track Googlebot's activity on my server. (Although, I think this could broadly apply to tracking Googlebot activity on any server.) I can see that I have results ...
0
votes
0answers
8 views

Links removed from google search results [duplicate]

For my website www.trickwarehouse.com i have created one post on keyloggers. When i search in google that what are keyloggers? then my website's post position was 16 but after 1 hour, when i checked ...
0
votes
1answer
62 views

google webmaster tools, site maps and indexing, number of pages indexed reduces with no new sitemap

It seems to take two weeks ish to get updates on to the google index, which seems fair enough as I imagine googles index of the whole internet could be rather large. However, the some changes to my ...
2
votes
1answer
74 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
1
vote
2answers
49 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
24 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
819 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
72 views

Does Fetch as Googlebot still support their ajax-crawling proposal?

I spent half a day implementing the server side html generation for modal pages based on their proposal (link), but it seems like the Fetch as Googlebot functionality in Webmaster tools completely ...
2
votes
2answers
91 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
1answer
52 views

What if my HTML snapshot for Googlebot doesn't contain the same content as the dynamically created page?

According to Google here: The HTML snapshot must contain the same content as the dynamically created page. What if my returned HTML Snapshot doesn't contain the same content as the actual ...
8
votes
2answers
71 views

Informing Googlebot for deprecated pages

I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a ...
1
vote
1answer
28 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
0answers
49 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
6
votes
4answers
217 views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
3
votes
3answers
45 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...
1
vote
1answer
50 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
2answers
79 views

SEO penalty for 301 redirecting googlebot only?

I have a website that is splitting into two. For the sake of easily describing the what and why, let's say I run a classifieds site and I want to move all the adult content to its own domain. (very ...
3
votes
1answer
47 views

Odd/faulty page links being served to Googlebot

Yahoo recently bought Tumblr, where I write fairly long form scientific pieces. I noticed recently that a new script had been added to the end of pages, and it seems to be interacting badly with the ...
0
votes
1answer
107 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
0
votes
1answer
94 views

How do I find when Googlebot will visit(crawl) my pages again?

I implemented the unavailable_after tag in my pages like the below and set to unavailable after 3 days but in Google Webmaster Tools I got 404 for that page. <META NAME="GOOGLEBOT" ...
1
vote
2answers
65 views

Regarding Google Index -> Remove URLs

I have a question regarding Google index -> Remove URLs from Google Webmaster Tools. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't ...
6
votes
3answers
98 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
2
votes
1answer
43 views

Google not indexing my Mediawiki's jpg files description pages

My website es.wikineos.com has been running for a few years with the MediaWiki Content Management System like the one they use at Wikipedia. Google seems to be indexing my site correctly at a quick ...
0
votes
1answer
98 views

How to get Google to index dynamic pages generated from user IP address

I am creating a local community directory site which identifies the location of the user by IP address and dynamically renders the content related to the specific community identified by the IP ...
2
votes
1answer
145 views

How to tell Google my content will expire after a certain period? [duplicate]

Right now I'm facing issues with 404 (Not Found) errors, because my content will deleted automatically after two months. I cannot change this behavior, because it is a real estate web site and once it ...
5
votes
1answer
102 views

How to deal with non-relevant keywords in Google Webmaster Tools

Google webmaster tools is showing keywords such as "cookies" in my keyword list. This is probably because we have links to our long legal disclaimer on cookies. We're a B2B service, so clearly I ...
4
votes
2answers
588 views

Hijacked meta description

On my site: faithhopeandfiction someone hacked into my website and now it's displaying wrong Meta description on Google search results. I've verified this using "view source" that meta description and ...
2
votes
2answers
74 views

Google search results not showing my events rich snippets but does in test tool

When I search on "tennessee basketball tournaments", I see my result, but my events dont show up. I believe I did everything right since this is an ajax page, but I tell Google that ...
1
vote
1answer
26 views

Older post not indexed, new post indexed right away?

I have a post I created yesterday that was indexed rather quickly by Google. However, I have two other post that I created within the past 4 days that are not indexed and I cannot find a trace of ...
0
votes
0answers
43 views

Googlebot can't access my site

For some reason, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt everything checks out, so I'm not sure ...
1
vote
1answer
20 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...