Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

2
votes
4answers
1k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
1
vote
3answers
1k views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
3answers
205 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...
1
vote
0answers
73 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
0
votes
1answer
123 views

Is it safe to manipulate with DOM (using JavaScript) to move up in the search results?

I have some text written for SEO purposes and it is located right after the opening body tag, so it is at the top of the page. That doesn't look great so this text is moved to the bottom of the page ...
2
votes
1answer
249 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
0
votes
1answer
140 views

Need Google to recrawl the pages

I've changed the meta description tag in many of my site pages. Should I select Fetch as Google->Fetch->Add to Index->URL and all linked pages in GWT? Will Google recrawl the whole site faster than ...
1
vote
2answers
324 views

Should I expect Search Engines (particularly Google) to crawl epub and mobi files

Some of our content we are publishing as epub and mobi eBook format files. Will search engines crawl and index these in the same way as PDFs currently are. If the answer is yes, could you point me to ...
0
votes
2answers
340 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
1
vote
1answer
225 views

Will Google index a PHP header redirect?

I have a new domain which has an index.php file. Inside this file, I get user's IP, save it and redirect to another page on another domain. So there is no output, only header info. Will Google index ...
1
vote
1answer
188 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
4
votes
3answers
909 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
3
votes
2answers
227 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
2
votes
2answers
859 views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
5
votes
3answers
3k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
0
votes
2answers
3k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
2
votes
3answers
2k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
1
vote
2answers
819 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
126 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
96 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...
1
vote
1answer
142 views

Google attempting crawl of URLs with parameter randomly placed in URL

We have a weird SEO issue; In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors. There are a set of several hundred dynamic pages which we need to maintain (though not ...
1
vote
1answer
210 views

Google visits on Statcounter. How?

In my website’s Statcounter report, I found page views from Google. I thought it may be a Google bot’s visit but I found that particular visit comes through a search query (example.com+keyword) and it ...
1
vote
3answers
175 views

New or same URL on reactivation of user generated content?

We run a classified ad website. Our users can temporarily deactivate their classified ads. From a SEO point of view, when they reactivate it, should the ad use the same URL or use a new one? Each ...
2
votes
1answer
100 views

How to have Google crawl pages that are only up for a week

We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
3
votes
1answer
407 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
2
votes
2answers
902 views

How can I block visitors located in the U.S. while still allowing the Googlebot?

Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page. While we do have these ...
0
votes
1answer
110 views

How can I create a sitemap for 2000 products powered by a single php page?

Usually I generate sitemaps using www.xml-sitemaps.com and the limit is 500 pages. My website has barely 30, but I have a lot of products in my database, so the page product.php becomes ...
1
vote
2answers
115 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
9
votes
3answers
2k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
0
votes
1answer
142 views

Mobile site redirection and Google crawlers

I have a HTML website where users can select a product (going through categories/subcategories) and then when they choose the product they can customize it in a Flash application. I would like to ...
3
votes
1answer
470 views

Google crawlers screen size

Imagine a I have two types of images (A and AAA(heavier)) for the same website: depending on the size of the screen it will show one image or other. Now I'm wondering what images (A or AAA) loads ...
2
votes
2answers
257 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
1
vote
1answer
107 views

Why old (301) links stay on Google when breaking site down to multiple domains

Some background: We did have single site and single domain (let's call it mainsite.com) with product information, however things have changed since and product database has grown fast. So we decided ...
0
votes
1answer
510 views

How to find first cached date and time of a website?

I found some of my webpage contents has been copied in other website. I need to check the cache date and time of both pages so that i can compare and guess which page is original and which one is ...
2
votes
1answer
880 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
1
vote
2answers
214 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
1
vote
3answers
217 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
1
vote
1answer
498 views

Stop Google crawling links completely

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
0
votes
1answer
87 views

Problem with Google crawler and cached results

I have a client that has 3 domain names. For the sake of this argument I will call them www.website1.com, www.website2.com and www.website3.com. He used to have a website package with VistaPrint and ...
0
votes
1answer
78 views

How does Google crawler find pages?

Will Google crawler find a page that is not linked to? I.e. pages accessible by manually modifying the url, like http://www.site.com/node/1/[secret keyword]. Note that it is only secret in that there ...
0
votes
1answer
92 views

What time does Googlebot crawl sites

Daily evening I sync the localhost copy of website to the production one. Yesterday I could not do it and I did so today at 10:00 am. Common sense says that Googlebot should be crawling the site when ...
2
votes
1answer
608 views

Will Google crawl session based website

I have a website, it is split into 3 categories but using PHP its an all-in one kind of style. When a user chooses a category on the home page a session is set, this is then used to set the style and ...
0
votes
1answer
82 views

Why did the Googlebot add “sp=1” to my URL's?

Googlebot crawled my site for the first time today, and added something strange. It appended &sp=1 to my URL's. It seems like it's trying to gain authorized access? Normally I have ...
2
votes
1answer
148 views

Can bots crawl through “checkbox” form elements which act as a filter for products?

For example, my site has a list of products, with a filter box which allows you to narrow by price, features, etc. My question is, can google crawl this form, see the url results (which is a bunch of ...
0
votes
1answer
36 views

How do search engines treat out-tracked links

Some websites have out tracking / outgoing click tracking so that when you click on a link such as mysite.com on example.com you momentarily get taken to example.com/outgoing/mysite.com which then ...
4
votes
2answers
154 views

Google SEO - Migrating website from a sub-directory of another website to its own domain name

At the present moment I have a website hosted on example.com/myWebsite, where example.com hosts in its root directory a different website. I have the domain example.net, which redirects to ...
1
vote
1answer
614 views

Googlebot getting “301 Moved Permanently” according to “Fetch as Googlebot” in Webmaster Tools

I launched a website four days ago, and now when I try to crawl the website like Google bot in "Google webmaster tools" I end up with a response like this: HTTP/1.1 301 Moved Permanently Date: Thu, ...
2
votes
1answer
777 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
7
votes
3answers
771 views

Do Search Engine Spider's submit POST forms?

If I have content behind a POST form (a select box, in my case) will Googlebot, or other Search Engine Spiders be able to crawl the content behind that form? For example, if I were to have 4 ...
4
votes
1answer
389 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...