Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
18 views

Older post not indexed, new post indexed right away?

I have a post I created yesterday that was indexed rather quickly by Google. However, I have two other post that I created within the past 4 days that are not indexed and I cannot find a trace of ...
0
votes
0answers
32 views

Googlebot can't access my site

For some reason, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt everything checks out, so I'm not sure ...
1
vote
1answer
19 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
0
votes
2answers
37 views

Should I add the WWW version of my site to Google Webmaster Tools or Non-WWW or both?

I've found here similar topics but not exactly similar and without satisfying references - I'm going to launch a new domain, I understood it's better to have one pattern - whether www or non-www, so ...
1
vote
2answers
27 views

Does the Instant Preview in Google webmaster tools takes Robot.txt in account?

Is that the way to go If I want to visually see what the googlebot see? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as google bot, It fetches ok, ...
5
votes
5answers
963 views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
0
votes
1answer
21 views

Can Googlebot read CamelCase file names?

This has been a question I have been wondering for quite some time now. Can Googlebot, and other search engine crawlers, read file names (such as images) that are in CamelCase? For example, if I ...
2
votes
2answers
70 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
1
vote
1answer
42 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
1
vote
1answer
20 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
0
votes
0answers
26 views

How can I stop Googlebot can't access your site messages from a site that's moved to another developer

I had a client who moved their site a while back to another developer. When they moved it they took down the site and put up a coming soon page ( which is currently what's there). Googlebot warned me ...
6
votes
1answer
92 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
0
votes
1answer
69 views

Several <a href=“#”>link</a> on one page - Effect on SEO and User Experience?

I've been researching the question if several <a href=“#”>link</a> links on one page have a bad impact on SEO and User Experience, for quite a while now but didn't really come to a ...
1
vote
3answers
62 views

Create a way for Googlebot to find the deep pages in my site when users would find these pages via site search

I have a website like Foursquare, where users search for business in a specific location. The users can click on markers inside the map to view details about the companies/stores, and these details ...
0
votes
0answers
6 views

Can google crawl data on googlemap?

I have a web service that includes mainly data on map. Can google fetch these data (asynchronously loaded )? the solution I'm opting right now is to set the data urls in a regular sitemap but there ...
0
votes
1answer
31 views

How to see my site's problems with Google?

I recently made a website: Boun Schedule Planner, which aims for the search query "Boun Schedule Planner" It was like 5th or 6th for a few days. But now it's not even in the first 3 pages. There are ...
2
votes
2answers
47 views

Region specific language in Google Search results?

I have a dynamic English language site. If a user views the site in the US (with a US IP address) some of the site's language is altered so that it make more sense to a US audience. Anywhere else, ...
0
votes
3answers
62 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
0
votes
0answers
40 views

Why does Google keep hitting only one MP3 file?

I have a bunch of Mp3 files on my site, and all if them are in the sitemap to allow search engines that log mp3 files to know about them. For some reason, every day or two, Google keeps hitting the ...
2
votes
1answer
64 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
0
votes
2answers
42 views

Question about external JavaScript and SEO [duplicate]

I have a question about external JavaScript and SEO. Assume I have a site called example.com with and index page of: <html> <body> <div class='main-container'> <script ...
1
vote
1answer
75 views

Prevent Googlebot from following JavaScript redirect

I run a website called isitsnowingoutside.com as kind of an experimental project. As I am still learning- it is far from perfect. The basic premise is the site uses your location to determine if it is ...
3
votes
1answer
35 views

Will Google follow link if anchor tag content is blocked?

This is a followup on a question I was asking here: Cropping images & SEO Basically, if I have something like this: <a href="<Full_Res_Image>"><img ...
1
vote
1answer
81 views

Can Google bots interpret hyphen separated words as a single keyword?

I have a french website which frequently includes the word rendez-vous. But, Google Webmaster Tools content keywords tells me the #1 keyword is rendez which is not what I want. Even worse, the word ...
3
votes
1answer
71 views

Relative URLs confuse Google crawler

I'm having some odd problems that are apparently being generated by Google crawler. Bing isn't these errors, so it's kind of confusing. The problem seems to be because of relative links. Google ...
0
votes
0answers
29 views

Sudden decrease in number of pages indexed in Google [duplicate]

One week ago, I updated the sitemap and pinged Google and other search engines to alert them to the change. However, the site now has lost like around 50 indexed pages (yes the site is my personal ...
3
votes
2answers
99 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
1
vote
0answers
11 views

How do google bots work for sites that are database driven? [duplicate]

I didn't see this question already asked but I also don't know if this may be called something in particular so my apologies. While learning SQL and PHP I was curious to know how do bots search a ...
2
votes
3answers
92 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
0
votes
0answers
8 views

Disabling Single Event and 301 Redirect [duplicate]

Is it bad practice to have a 301 redirect for an event that has been disabled or deleted from a system to the list view? or should a 404 page? I dont want Google to ban me or anything.
0
votes
2answers
41 views

Lynx shows nothing but “loading” when opening a site

The site I help to administer appears very poorly in Google search results. While trying to tackle the issue from several angles I've tried to open the site in Lynx browser (the intention was to see ...
3
votes
1answer
48 views

Is there a way to stop a keyword from appearing in Google Webmaster Tools?

I added a web site a while ago to Google's index and it contains lots of links with the text "Permalink" that, well, link to permalinks of different content/pages. Now "Permalink" is the #1 keyword ...
3
votes
1answer
55 views

Using a canonical link with non-duplicate pages

I want to use canonical tag for index.php and all versions of index.php below: index.php index.php?something index.php?somethingsomething I also have URLs like: someproduct.html ...
1
vote
3answers
134 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
0answers
34 views

What resolution does Google use while crawling a site with responsive/adaptive design? [duplicate]

I use conditional loading and media queries, JS, and CSS techniques for building responsive/adaptive designed websites. The HTML output for these sites depends on the number of horizontal and ...
0
votes
0answers
12 views

How do services that make subsites (ie. new subdomains) get those sites indexed by Google and other search enginex? [duplicate]

Services like tumblr, github pages, quora blogs, etc. make new subdomains when a user signs up or makes a site. So the subdomains are like: blogname.tumblr.com , pagename.github.io, ...
3
votes
1answer
156 views

Google is not indexing my site although it is crawling it

I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message: URL and ...
1
vote
1answer
67 views

Subdomain and Googlebot can't access your site

I have a subdomain and Google Webmaster Tools, told me this: Googlebot can't access your site This is not the first time that I receive this message from Google's Webmaster Tools. I need to ...
5
votes
1answer
117 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. http://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
2
votes
1answer
187 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
0
votes
1answer
49 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
2
votes
2answers
202 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
1
vote
5answers
415 views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
2
votes
1answer
36 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
1
vote
1answer
122 views

Creating sitemap for Googlebot - how to mark dynamic content / dynamic subpages?

I have a website that is an Internet forum. This forum has many categories, and a single category page that contains a lot of subpages with listed threads. This Internet forum is brand new, and about ...
1
vote
2answers
67 views

Somehow Google considers a properly 301'd URL as 200 and is still indexing the new content in old page?

We redirected all the old URL's to new ones properly using .htaccess. The problem is Google, somehow is still finding content in the old page (which it shouldn't) and stores it in the cache rather ...
0
votes
1answer
120 views

Will using page headings (h1, h2, h3 etc) as internal links encourage Google sitelinks?

I am trying to change my site's page headings to better emphasize the structure of the page's content hierarchy. I am hoping this will encourage Google Sitelinks and list snippets. Current headings ...
1
vote
1answer
38 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
4answers
149 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
0
votes
1answer
83 views

In addition to Google's First Click Free, should you whitelist search engine bots past a paywall?

Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's FCF requirements, your first 5 hits to a subscriber-only pages with .google. as the referrer, you ...