Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

3
votes
1answer
77 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
1
vote
3answers
269 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
0answers
34 views

What resolution does Google use while crawling a site with responsive/adaptive design? [duplicate]

I use conditional loading and media queries, JS, and CSS techniques for building responsive/adaptive designed websites. The HTML output for these sites depends on the number of horizontal and ...
0
votes
0answers
12 views

How do services that make subsites (ie. new subdomains) get those sites indexed by Google and other search enginex? [duplicate]

Services like tumblr, github pages, quora blogs, etc. make new subdomains when a user signs up or makes a site. So the subdomains are like: blogname.tumblr.com , pagename.github.io, ...
3
votes
1answer
317 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
1
vote
1answer
150 views

Subdomain and Googlebot can't access your site

I have a subdomain and Google Webmaster Tools, told me this: Googlebot can't access your site This is not the first time that I receive this message from Google's Webmaster Tools. I need to ...
5
votes
1answer
210 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. http://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
2
votes
1answer
361 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
0
votes
1answer
94 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
2
votes
2answers
811 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
1
vote
5answers
1k views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
2
votes
1answer
40 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
1
vote
1answer
183 views

Creating sitemap for Googlebot - how to mark dynamic content / dynamic subpages?

I have a website that is an Internet forum. This forum has many categories, and a single category page that contains a lot of subpages with listed threads. This Internet forum is brand new, and about ...
1
vote
2answers
91 views

Somehow Google considers a properly 301'd URL as 200 and is still indexing the new content in old page?

We redirected all the old URL's to new ones properly using .htaccess. The problem is Google, somehow is still finding content in the old page (which it shouldn't) and stores it in the cache rather ...
0
votes
1answer
296 views

Will using page headings (h1, h2, h3 etc) as internal links encourage Google sitelinks? [duplicate]

I am trying to change my site's page headings to better emphasize the structure of the page's content hierarchy. I am hoping this will encourage Google Sitelinks and list snippets. Current headings ...
1
vote
1answer
40 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
4answers
917 views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
0
votes
1answer
221 views

In addition to Google's First Click Free, should you whitelist search engine bots past a paywall?

Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's FCF requirements, your first 5 hits to a subscriber-only pages with .google. as the referrer, you ...
4
votes
2answers
190 views

When Googlebot sees a link, will it click it or navigate to it?

My site uses pushState and JSON data to display content. So, for example, this might appear on my page: <a href="/some/page">some page</a> The JavaScript then prevents the default ...
2
votes
1answer
305 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
5
votes
1answer
620 views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
3
votes
2answers
576 views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
1
vote
1answer
397 views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
3
votes
2answers
157 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
2
votes
1answer
7k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
378 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
1
vote
1answer
420 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
0
votes
1answer
35 views

Why does my htm pages have a number attached at the end of the link?

When I wanted to fetch a certain page with htm extension I noticed an attachment after the htm extenstion http://www.example.com/IP_814.htm#.Uk1_MtKsh8E Google could not fetch this page it gave the ...
2
votes
2answers
80 views

Will multilingual content on same page works to avoid content duplication

We have a website, which is having old/new vehicle deals for purchase. We are getting these data from third party website (legally) and all the pages are in English language. So, Here our issue is ...
7
votes
2answers
3k views

Google-bot sees “Sorry, we have no imagery here” on pages with Google Maps

I have a site with Google Maps on most of the pages. When inspecting content keywords in Google Webmaster tools, content keywords identified by Google-bot for the site include "imagery", "sorry" and ...
2
votes
4answers
1k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
1
vote
3answers
819 views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
3answers
145 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...
1
vote
0answers
70 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
0
votes
1answer
98 views

Is it safe to manipulate with DOM (using JavaScript) to move up in the search results?

I have some text written for SEO purposes and it is located right after the opening body tag, so it is at the top of the page. That doesn't look great so this text is moved to the bottom of the page ...
2
votes
1answer
165 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
0
votes
1answer
114 views

Need Google to recrawl the pages

I've changed the meta description tag in many of my site pages. Should I select Fetch as Google->Fetch->Add to Index->URL and all linked pages in GWT? Will Google recrawl the whole site faster than ...
1
vote
2answers
224 views

Should I expect Search Engines (particularly Google) to crawl epub and mobi files

Some of our content we are publishing as epub and mobi eBook format files. Will search engines crawl and index these in the same way as PDFs currently are. If the answer is yes, could you point me to ...
0
votes
2answers
223 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
1
vote
1answer
158 views

Will Google index a PHP header redirect?

I have a new domain which has an index.php file. Inside this file, I get user's IP, save it and redirect to another page on another domain. So there is no output, only header info. Will Google index ...
1
vote
1answer
159 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
4
votes
3answers
545 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
3
votes
2answers
164 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
2
votes
2answers
502 views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
5
votes
3answers
2k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
0
votes
2answers
2k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
2
votes
3answers
1k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
1
vote
2answers
417 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
107 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
93 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...