Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
2answers
4k views

robots.txt for a CodeIgniter based website?

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application public sql style system design images ...
2
votes
3answers
98 views

suspicious crawler activity

I'm noticing that I get accesses 66.249.66.198 - - [01/Jul/2011:17:13:46 +0200] "GET /img/clip.incubus.torrent.phtml HTTP/1.1" 404 143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
2
votes
3answers
159 views

Backlinks: How does a company's name as the anchor text help with SEO?

Often I see footer links like: SEO by COMPANY NAME How does this help with SEO other than helping a company rank for "Company Name"? When nearby words like "SEO by" get picked up by a search engine ...
2
votes
3answers
379 views

Google Webmaster Tools reporting crawl error 404 (Not found) for a page which has never existed

Google Webmaster Tools is reporting a crawl error "404 (Not Found)" for an .html page which has never existed on a web site I developed. The page being reported is "tyuiabmog.html", which doesn't ...
2
votes
2answers
152 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
2
votes
2answers
317 views

Googlebot requesting invalid URL

I have a web app which emails me exceptions automatically. This morning there was an error relating to a url: /Catalog/LiveCatalog?id=ylwpfqzts id is invalid (should be a guid) and caused an error ...
2
votes
1answer
45 views

Why my webpages body content is not taken into account to display in google SERP?

It's confusing for me that I have content on my webpages, however, when I search with the site: operator it is not taken into account. There is something like "Thank you for contacting us...." ...
2
votes
1answer
301 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
2
votes
2answers
98 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
2
votes
1answer
73 views

Google not indexing my Mediawiki's jpg files description pages

My website es.wikineos.com has been running for a few years with the MediaWiki Content Management System like the one they use at Wikipedia. Google seems to be indexing my site correctly at a quick ...
2
votes
2answers
104 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
2
votes
1answer
43 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
2
votes
1answer
8k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
211 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
2
votes
1answer
685 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
2
votes
1answer
378 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
2
votes
2answers
92 views

Forbid content in google results summary

I put a professional directory project online since few weeks, Google has started to index some pages but i just noticed a flaw in the design of the pages. Each page have an adress, some content and ...
2
votes
1answer
1k views

How to tell Google to stop crawling my website via IP?

After a database server migration, I've noticed that GoogleBot started getting errors when trying to crawl my site. The reason seems to be that it is hitting my site via IP address (and due to my ...
2
votes
1answer
416 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
28 views

rel=“prev” and rel=“next” inserting using javascript

does adding the rel="next" tag after the page is loaded with a small javascript still useful for SEO and google? i have a problem in my code, basically the total number of pages in the catalog is ...
2
votes
1answer
112 views

Using Fetch as Google on AngularJS

I am working on an eCommerce site and I have been asked to look at the SEO implications of our product listing page. I have done all the necessary steps to get the webmaster tools working fine on one ...
2
votes
1answer
169 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
140 views

URLs for dialogs from Bootstrap modals popups are getting indexed in search engines, but without nav they aren't good landing pages

On my side, I use bootstrap modals (aka dialogs / overlays) with remote content to display the detail view of some things (for example for detail view of user reviews/comments). The code looks like ...
2
votes
2answers
52 views

For performance reasons: have different pages for search engines and users

I have a SEO question for single page applications. We're working on a SPA (not yet public) that displays tables of numeric data to users. The data volume is large, number aggregations are costly ...
2
votes
1answer
49 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
2
votes
1answer
192 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
2
votes
1answer
80 views

What if my HTML snapshot for Googlebot doesn't contain the same content as the dynamically created page?

According to Google here: The HTML snapshot must contain the same content as the dynamically created page. What if my returned HTML Snapshot doesn't contain the same content as the actual ...
2
votes
2answers
276 views

Google search results not showing my events rich snippets but does in test tool

When I search on "tennessee basketball tournaments", I see my result, but my events dont show up. I believe I did everything right since this is an ajax page, but I tell Google that ...
2
votes
3answers
320 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
2
votes
1answer
159 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
2
votes
2answers
136 views

Region specific language in Google Search results?

I have a dynamic English language site. If a user views the site in the US (with a US IP address) some of the site's language is altered so that it make more sense to a US audience. Anywhere else, ...
2
votes
1answer
226 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
2
votes
1answer
477 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
2
votes
1answer
468 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
2
votes
1answer
99 views

How to have Google crawl pages that are only up for a week

We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
2
votes
2answers
788 views

How can I block visitors located in the U.S. while still allowing the Googlebot?

Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page. While we do have these ...
2
votes
1answer
728 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
2
votes
1answer
531 views

Will Google crawl session based website

I have a website, it is split into 3 categories but using PHP its an all-in one kind of style. When a user chooses a category on the home page a session is set, this is then used to set the style and ...
2
votes
1answer
134 views

Can bots crawl through “checkbox” form elements which act as a filter for products?

For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of ...
2
votes
1answer
662 views

Removing Google Webmaster URL Parameters

Some time before I have created a post for removing full site from Google Index, after following many advices I have removed large number of pages from Google Index but now facing another problem as ...
2
votes
1answer
76 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
1answer
188 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
2
votes
1answer
64 views

multilingual domains both crawled and come up to the wrong audience

I have a domain with multilingual support through subdomains. Each language version is identical besides the titles which are translated. e.g.: domain.com ja.domain.com es.domain.com I'm ...
2
votes
2answers
232 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
1answer
348 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
2
votes
1answer
600 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
2
votes
1answer
110 views

Crawl Errors (404) Showing Up in WMT

I see from WMT that my site has 4 (404) crawl errors, each Linked from 2 separate pages. As this is a small directory listings site it would be difficult to delete the missing URLs from the db each ...
2
votes
1answer
507 views

Prevent Googlebot from following JavaScript redirect

I run a website called isitsnowingoutside.com as kind of an experimental project. As I am still learning- it is far from perfect. The basic premise is the site uses your location to determine if it is ...
2
votes
2answers
294 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
1answer
332 views

Google search : pubdate issue

At some point this year, Google Search started to show the publication date. When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...