Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
1answer
54 views

Is creating Ajax Google Snapshot with less content than original page considered cloaking?

I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions: I have a full AJAX web site with really a lot of pages which differ from one to the other with a ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
1
vote
0answers
60 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
1
vote
2answers
89 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
1
vote
0answers
12 views

How do google bots work for sites that are database driven? [duplicate]

I didn't see this question already asked but I also don't know if this may be called something in particular so my apologies. While learning SQL and PHP I was curious to know how do bots search a ...
1
vote
0answers
34 views

What resolution does Google use while crawling a site with responsive/adaptive design? [duplicate]

I use conditional loading and media queries, JS, and CSS techniques for building responsive/adaptive designed websites. The HTML output for these sites depends on the number of horizontal and ...
1
vote
0answers
70 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
1
vote
1answer
107 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
1
vote
0answers
56 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
2answers
340 views

How can I tell GoogleBot that a subdirectory is now a subdomain?

I had about a million pages of a catalog indexed under a subdirectory, and now that's moved to a subdomain. GoogleBot is crawling each one of them and getting a 301 redirect to the new location. Even ...
1
vote
0answers
177 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
1
vote
2answers
161 views

Confused by Fetch as Googlebot behaviour

I have a site that I 'm trying to index and I am a bit confused by what I should expect to see in the result of the Fetch as Googlebot tool. The site uses hashbangs, so according to Google it should ...
1
vote
2answers
174 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
0
votes
3answers
122 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
0
votes
1answer
188 views

Googlebot following invalid URLs. How do I troubleshoot the problem?

I'm getting errors generated by Googlebot while following invalid URLs. Checking with Google Webmaster tools confirmed that the problem was the bot: I've checked the source code and the HTML ...
0
votes
2answers
138 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
0
votes
2answers
55 views

Lynx shows nothing but “loading” when opening a site

The site I help to administer appears very poorly in Google search results. While trying to tackle the issue from several angles I've tried to open the site in Lynx browser (the intention was to see ...
0
votes
2answers
139 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
0
votes
1answer
80 views

Facelets SEO advice

Before I deploy the new version of my site, I would like to know a little more about facelets and SEO. My previous version of site was written in JSF and I noticed that when I submitted it to the ...
0
votes
1answer
15 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
0
votes
2answers
43 views

Does Googlebot (and/or search engines) index a forwarded page? [duplicate]

Let's say I have example.com domain, and I force the user to use the HTTPS over HTTP. The question is as browsers just accept and load the forwarded/new page (when the request for http://example.com ...
0
votes
2answers
57 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
0
votes
1answer
30 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
2answers
72 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
0
votes
1answer
116 views

How do I find when Googlebot will visit(crawl) my pages again?

I implemented the unavailable_after tag in my pages like the below and set to unavailable after 3 days but in Google Webmaster Tools I got 404 for that page. <META NAME="GOOGLEBOT" ...
0
votes
1answer
98 views

Is it safe to manipulate with DOM (using JavaScript) to move up in the search results?

I have some text written for SEO purposes and it is located right after the opening body tag, so it is at the top of the page. That doesn't look great so this text is moved to the bottom of the page ...
0
votes
1answer
114 views

Need Google to recrawl the pages

I've changed the meta description tag in many of my site pages. Should I select Fetch as Google->Fetch->Add to Index->URL and all linked pages in GWT? Will Google recrawl the whole site faster than ...
0
votes
1answer
78 views

Problem with Google crawler and cached results

I have a client that has 3 domain names. For the sake of this argument I will call them www.website1.com, www.website2.com and www.website3.com. He used to have a website package with VistaPrint and ...
0
votes
1answer
74 views

Why did the Googlebot add “sp=1” to my URL's?

Googlebot crawled my site for the first time today, and added something strange. It appended &sp=1 to my URL's. It seems like it's trying to gain authorized access? Normally I have ...
0
votes
2answers
155 views

Hide elements for Google

I have some elements which I don't want Google to see. Usually repetitive content, e.g. header and sidebar. As Googlebot doesn't handle AJAX requests, I load these resources with AJAX and insert them ...
0
votes
2answers
133 views

Anyone heard of 'tank tracking'?

Someone I know told me about some seriously cutting edge blackhat SEO that he called 'tank tracking'. He said that it is some sort of code (he believes written in Python) that 'sits' around the ...
0
votes
1answer
236 views

Google AdSense bot's algorithm and behavior [closed]

I am interesting in Google AdSense bot's algorithm and behavior with web site. I did not work with AdSense and I do not have account. So I need your help to understand: 1) Google's bot from time to ...
0
votes
1answer
45 views

Can Googlebot detect duplicate video content?

I downloaded a lot of videos from YouTube and uploaded them to my hosting. Can Googlebot-video can detect that my video is duplicate?
0
votes
2answers
129 views

Drop in traffic but increase in pages crawled per day

I saw a drop in traffic of my website in early September 2014. But, if I check the "Crawl Stats" in Google Webmaster Tools, I see an increase in "Pages Crawled per day" and "Kilobytes downloaded ...
0
votes
1answer
49 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
0
votes
1answer
57 views

Can Googlebot read CamelCase file names?

This has been a question I have been wondering for quite some time now. Can Googlebot, and other search engine crawlers, read file names (such as images) that are in CamelCase? For example, if I ...
0
votes
1answer
37 views

How to see my site's problems with Google?

I recently made a website: Boun Schedule Planner, which aims for the search query "Boun Schedule Planner" It was like 5th or 6th for a few days. But now it's not even in the first 3 pages. There are ...
0
votes
1answer
94 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
0
votes
1answer
296 views

Will using page headings (h1, h2, h3 etc) as internal links encourage Google sitelinks? [duplicate]

I am trying to change my site's page headings to better emphasize the structure of the page's content hierarchy. I am hoping this will encourage Google Sitelinks and list snippets. Current headings ...
0
votes
1answer
35 views

Why does my htm pages have a number attached at the end of the link?

When I wanted to fetch a certain page with htm extension I noticed an attachment after the htm extenstion http://www.example.com/IP_814.htm#.Uk1_MtKsh8E Google could not fetch this page it gave the ...
0
votes
2answers
222 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
2answers
2k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
0
votes
2answers
93 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...
0
votes
1answer
88 views

How can I create a sitemap for 2000 products powered by a single php page?

Usually I generate sitemaps using www.xml-sitemaps.com and the limit is 500 pages. My website has barely 30, but I have a lot of products in my database, so the page product.php becomes ...
0
votes
1answer
60 views

How does Google crawler find pages?

Will Google crawler find a page that is not linked to? I.e. pages accessible by manually modifying the url, like http://www.site.com/node/1/[secret keyword]. Note that it is only secret in that there ...
0
votes
1answer
72 views

What time does Googlebot crawl sites

Daily evening I sync the localhost copy of website to the production one. Yesterday I could not do it and I did so today at 10:00 am. Common sense says that Googlebot should be crawling the site when ...
0
votes
1answer
32 views

How do search engines treat out-tracked links

Some websites have out tracking / outgoing click tracking so that when you click on a link such as mysite.com on example.com you momentarily get taken to example.com/outgoing/mysite.com which then ...
0
votes
2answers
116 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
4answers
251 views

Hide your website links from search engne

My Website Link is posted in someone else website, i asked for removal but can't get reply, google crawl his website and my website link is posted on that how to control that is that is possible? i ...