Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
49 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
1answer
37 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
1answer
63 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
202 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
1
vote
2answers
103 views

Regarding Google Index -> Remove URLs

I have a question regarding Google index -> Remove URLs from Google Webmaster Tools. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't ...
1
vote
1answer
20 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
1
vote
3answers
77 views

Create a way for Googlebot to find the deep pages in my site when users would find these pages via site search

I have a website like Foursquare, where users search for business in a specific location. The users can click on markers inside the map to view details about the companies/stores, and these details ...
1
vote
1answer
239 views

Can Google bots interpret hyphen separated words as a single keyword?

I have a french website which frequently includes the word rendez-vous. But, Google Webmaster Tools content keywords tells me the #1 keyword is rendez which is not what I want. Even worse, the word ...
1
vote
1answer
43 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
1answer
487 views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
1
vote
1answer
442 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
3answers
975 views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
1answer
167 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
1
vote
1answer
94 views

Why old (301) links stay on Google when breaking site down to multiple domains

Some background: We did have single site and single domain (let's call it mainsite.com) with product information, however things have changed since and product database has grown fast. So we decided ...
1
vote
3answers
327 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
1
vote
1answer
380 views

How to make image only website friendly to googlebot? [duplicate]

so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created. ...
1
vote
2answers
478 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
106 views

Bots and scripts [duplicate]

Possible Duplicate: Will AJAX-fetched material be invisible to search engines? Can bots understand what's going on in javascript or vbscript? Most of the social networking modules and ad ...
1
vote
1answer
534 views

Googlebot visit but no cache update - why?

I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
1
vote
0answers
46 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
1answer
111 views

Is creating Ajax Google Snapshot with less content than original page considered cloaking?

I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions: I have a full AJAX web site with really a lot of pages which differ from one to the other with a ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
1
vote
0answers
67 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
1
vote
0answers
77 views

I still get “Googlebot can't access your site” messages after removing a site from Google Webmaster Tools

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
1
vote
1answer
18 views

Can Googlebot crawl and index data shown on a map (powered by Google)?

I have a web service that includes mainly data on map. Can Google fetch these data (asynchronously loaded )? The solution I'm opting right now is to set the data URLs in a regular sitemap, but there ...
1
vote
0answers
70 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...
1
vote
0answers
12 views

How do google bots work for sites that are database driven? [duplicate]

I didn't see this question already asked but I also don't know if this may be called something in particular so my apologies. While learning SQL and PHP I was curious to know how do bots search a ...
1
vote
0answers
34 views

What resolution does Google use while crawling a site with responsive/adaptive design? [duplicate]

I use conditional loading and media queries, JS, and CSS techniques for building responsive/adaptive designed websites. The HTML output for these sites depends on the number of horizontal and ...
1
vote
0answers
13 views

How do services that make subsites (ie. new subdomains) get those sites indexed by Google and other search enginex? [duplicate]

Services like tumblr, github pages, quora blogs, etc. make new subdomains when a user signs up or makes a site. So the subdomains are like: blogname.tumblr.com , pagename.github.io, ...
1
vote
0answers
71 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
1
vote
1answer
114 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
1
vote
0answers
56 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
2answers
358 views

How can I tell GoogleBot that a subdirectory is now a subdomain?

I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
1
vote
0answers
181 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
1
vote
2answers
165 views

Confused by Fetch as Googlebot behaviour

I have a site that I 'm trying to index and I am a bit confused by what I should expect to see in the result of the Fetch as Googlebot tool. The site uses hashbangs, so according to Google it should ...
1
vote
2answers
187 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
0
votes
3answers
140 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
0
votes
1answer
194 views

Googlebot following invalid URLs. How do I troubleshoot the problem?

I'm getting errors generated by Googlebot while following invalid URLs. Checking with Google Webmaster tools confirmed that the problem was the bot: I've checked the source code and the HTML ...
0
votes
2answers
221 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
0
votes
2answers
59 views

Lynx shows nothing but “loading” when opening a site

The site I help to administer appears very poorly in Google search results. While trying to tackle the issue from several angles I've tried to open the site in Lynx browser (the intention was to see ...
0
votes
2answers
141 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
0
votes
1answer
81 views

Facelets SEO advice

Before I deploy the new version of my site, I would like to know a little more about facelets and SEO. My previous version of site was written in JSF and I noticed that when I submitted it to the ...
0
votes
1answer
22 views

How to tell how long google thinks it has been since your website was updated?

I never really thought 'freshness' was a problem for my site until I recently used the silktide 'nibbler' website grading report / scanner. It listed my 'freshness' as a big old 0 and said my site ...
0
votes
1answer
15 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
0
votes
2answers
58 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
0
votes
1answer
34 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
2answers
103 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
0
votes
1answer
144 views

How do I find when Googlebot will visit(crawl) my pages again?

I implemented the unavailable_after tag in my pages like the below and set to unavailable after 3 days but in Google Webmaster Tools I got 404 for that page. <META NAME="GOOGLEBOT" ...
0
votes
1answer
105 views

Is it safe to manipulate with DOM (using JavaScript) to move up in the search results?

I have some text written for SEO purposes and it is located right after the opening body tag, so it is at the top of the page. That doesn't look great so this text is moved to the bottom of the page ...