Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

4
votes
2answers
503 views

Hijacked meta description

On my site: faithhopeandfiction someone hacked into my website and now it's displaying wrong Meta description on Google search results. I've verified this using "view source" that meta description and ...
3
votes
1answer
107 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
2
votes
1answer
17 views

Google search results not showing my events rich snippets but does in test tool

When I search on "tennessee basketball tournaments", I see my result, but my events dont show up. I believe I did everything right since this is an ajax page, but I tell Google that ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
1
vote
1answer
20 views

Older post not indexed, new post indexed right away?

I have a post I created yesterday that was indexed rather quickly by Google. However, I have two other post that I created within the past 4 days that are not indexed and I cannot find a trace of ...
0
votes
2answers
324 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
0
votes
0answers
34 views

Googlebot can't access my site

For some reason, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt everything checks out, so I'm not sure ...
1
vote
1answer
19 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
1
vote
2answers
28 views

Does the Instant Preview in Google webmaster tools takes Robot.txt in account?

Is that the way to go If I want to visually see what the googlebot see? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as google bot, It fetches ok, ...
0
votes
2answers
41 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
5
votes
5answers
975 views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
0
votes
1answer
21 views

Can Googlebot read CamelCase file names?

This has been a question I have been wondering for quite some time now. Can Googlebot, and other search engine crawlers, read file names (such as images) that are in CamelCase? For example, if I ...
2
votes
1answer
66 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
2
votes
2answers
71 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
1
vote
1answer
46 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
2
votes
2answers
83 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
1
vote
1answer
22 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
0
votes
0answers
26 views

How can I stop Googlebot can't access your site messages from a site that's moved to another developer

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
6
votes
1answer
94 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
0
votes
1answer
75 views

Several <a href=“#”>link</a> on one page - Effect on SEO and User Experience?

I've been researching the question if several <a href=“#”>link</a> links on one page have a bad impact on SEO and User Experience, for quite a while now but didn't really come to a ...
1
vote
3answers
63 views

Create a way for Googlebot to find the deep pages in my site when users would find these pages via site search

I have a website like Foursquare, where users search for business in a specific location. The users can click on markers inside the map to view details about the companies/stores, and these details ...
0
votes
0answers
6 views

Can google crawl data on googlemap?

I have a web service that includes mainly data on map. Can google fetch these data (asynchronously loaded )? the solution I'm opting right now is to set the data urls in a regular sitemap but there ...
0
votes
1answer
32 views

How to see my site's problems with Google?

I recently made a website: Boun Schedule Planner, which aims for the search query "Boun Schedule Planner" It was like 5th or 6th for a few days. But now it's not even in the first 3 pages. There are ...
2
votes
2answers
49 views

Region specific language in Google Search results?

I have a dynamic English language site. If a user views the site in the US (with a US IP address) some of the site's language is altered so that it make more sense to a US audience. Anywhere else, ...
0
votes
3answers
64 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
81 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
3
votes
2answers
98 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
0
votes
0answers
43 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...
1
vote
3answers
2k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
0
votes
2answers
47 views

Question about external JavaScript and SEO [duplicate]

I have a question about external JavaScript and SEO. Assume I have a site called example.com with and index page of: <html> <body> <div class='main-container'> <script ...
2
votes
1answer
194 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
1
vote
1answer
4k views

How to fix Google 404 not found Crawl Errors?

I was checking on Google webmater tool for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
1
vote
2answers
270 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
6
votes
3answers
917 views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
1
vote
1answer
77 views

Prevent Googlebot from following JavaScript redirect

I run a website called isitsnowingoutside.com as kind of an experimental project. As I am still learning- it is far from perfect. The basic premise is the site uses your location to determine if it is ...
3
votes
1answer
37 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img ...
1
vote
1answer
86 views

Can Google bots interpret hyphen separated words as a single keyword?

I have a french website which frequently includes the word rendez-vous. But, Google Webmaster Tools content keywords tells me the #1 keyword is rendez which is not what I want. Even worse, the word ...
3
votes
1answer
72 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
0
votes
0answers
29 views

Sudden decrease in number of pages indexed in Google [duplicate]

One week ago, I updated the sitemap and pinged Google and other search engines to alert them to the change. However, the site now has lost like around 50 indexed pages (yes the site is my personal ...
3
votes
2answers
100 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
1
vote
0answers
11 views

How do google bots work for sites that are database driven? [duplicate]

I didn't see this question already asked but I also don't know if this may be called something in particular so my apologies. While learning SQL and PHP I was curious to know how do bots search a ...
2
votes
1answer
193 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
2
votes
3answers
94 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
0
votes
0answers
8 views

Disabling Single Event and 301 Redirect [duplicate]

Is it bad practice to have a 301 redirect for an event that has been disabled or deleted from a system to the list view? or should a 404 page? I dont want Google to ban me or anything.
0
votes
2answers
43 views

Lynx shows nothing but “loading” when opening a site

The site I help to administer appears very poorly in Google search results. While trying to tackle the issue from several angles I've tried to open the site in Lynx browser (the intention was to see ...
3
votes
1answer
48 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
1
vote
3answers
744 views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
3
votes
1answer
55 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
1
vote
0answers
162 views

Site returning 404 header to Google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by Google at all (which is the main issue but I ...
1
vote
0answers
65 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?