Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
1answer
42 views

Why my webpages body content is not taken into account to display in google SERP?

It's confusing for me that I have content on my webpages, however, when I search with the site: operator it is not taken into account. There is something like "Thank you for contacting us...." ...
2
votes
1answer
194 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
2
votes
2answers
86 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
2
votes
1answer
61 views

Google not indexing my Mediawiki's jpg files description pages

My website es.wikineos.com has been running for a few years with the MediaWiki Content Management System like the one they use at Wikipedia. Google seems to be indexing my site correctly at a quick ...
2
votes
2answers
97 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
2
votes
1answer
41 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
2
votes
1answer
8k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
180 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
2
votes
1answer
626 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
2
votes
1answer
318 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
2
votes
2answers
89 views

Forbid content in google results summary

I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages. Each page have an adress, some content and ...
2
votes
1answer
1k views

How to tell Google to stop crawling my website via IP?

After a database server migration, I've noticed that GoogleBot started getting errors when trying to crawl my site. The reason seems to be that it is hitting my site via IP address (and due to my ...
2
votes
1answer
409 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
70 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
66 views

URLs for dialogs from Bootstrap modals popups are getting indexed in search engines, but without nav they aren't good landing pages

On my side, I use bootstrap modals (aka dialogs / overlays) with remote content to display the detail view of some things (for example for detail view of user reviews/comments). The code looks like ...
2
votes
2answers
51 views

For performance reasons: have different pages for search engines and users

I have a SEO question for single page applications. We're working on a SPA (not yet public) that displays tables of numeric data to users. The data volume is large, number aggregations are costly ...
2
votes
1answer
43 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
2
votes
1answer
157 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
2
votes
1answer
71 views

What if my HTML snapshot for Googlebot doesn't contain the same content as the dynamically created page?

According to Google here: The HTML snapshot must contain the same content as the dynamically created page. What if my returned HTML Snapshot doesn't contain the same content as the actual ...
2
votes
2answers
181 views

Google search results not showing my events rich snippets but does in test tool

When I search on "tennessee basketball tournaments", I see my result, but my events dont show up. I believe I did everything right since this is an ajax page, but I tell Google that ...
2
votes
3answers
285 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
2
votes
1answer
114 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
2
votes
2answers
127 views

Region specific language in Google Search results?

I have a dynamic English language site. If a user views the site in the US (with a US IP address) some of the site's language is altered so that it make more sense to a US audience. Anywhere else, ...
2
votes
1answer
192 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
2
votes
1answer
404 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
2
votes
1answer
425 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
2
votes
1answer
97 views

How to have Google crawl pages that are only up for a week

We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
2
votes
2answers
686 views

How can I block visitors located in the U.S. while still allowing the Googlebot?

Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page. While we do have these ...
2
votes
1answer
632 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
2
votes
1answer
468 views

Will Google crawl session based website

I have a website, it is split into 3 categories but using PHP its an all-in one kind of style. When a user chooses a category on the home page a session is set, this is then used to set the style and ...
2
votes
1answer
129 views

Can bots crawl through “checkbox” form elements which act as a filter for products?

For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of ...
2
votes
1answer
524 views

Removing Google Webmaster URL Parameters

Some time before I have created a post for removing full site from Google Index, after following many advices I have removed large number of pages from Google Index but now facing another problem as ...
2
votes
1answer
75 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
1answer
185 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
2
votes
1answer
63 views

multilingual domains both crawled and come up to the wrong audience

I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.: domain.com ja.domain.com es.domain.com I'm ...
2
votes
2answers
225 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
1answer
348 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
2
votes
1answer
592 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
2
votes
1answer
107 views

Crawl Errors (404) Showing Up in WMT

I see from WMT that my site has 4 (404) crawl errors, each Linked from 2 separate pages. As this is a small directory listings site it would be difficult to delete the missing URLs from the db each ...
2
votes
1answer
438 views

Prevent Googlebot from following JavaScript redirect

I run a website called isitsnowingoutside.com as kind of an experimental project. As I am still learning- it is far from perfect. The basic premise is the site uses your location to determine if it is ...
2
votes
2answers
284 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
1answer
322 views

Google search : pubdate issue

At some point this year, Google Search started to show the publication date. When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...
1
vote
2answers
116 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
332 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
5answers
1k views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
1
vote
3answers
205 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
1
vote
4answers
1k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
201 views

When will my old page stop appearing on Google? [duplicate]

Possible Duplicate: Old pages still appearing in Google? I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
1
vote
2answers
246 views

sitemap and crawler

I am still newbie in the entire world of SEO. Let's take a look at my example. I have a website very similar to blog style site. There few patterns for urls, all of the are very simple and allowed to ...
1
vote
3answers
160 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...