Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
2answers
115 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
1answer
614 views

Googlebot getting “301 Moved Permanently” according to “Fetch as Googlebot” in Webmaster Tools

I launched a website four days ago, and now when I try to crawl the website like Google bot in "Google webmaster tools" I end up with a response like this: HTTP/1.1 301 Moved Permanently Date: Thu, ...
1
vote
2answers
464 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
1
vote
3answers
5k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
1answer
142 views

Could this be considered duplicate content? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm building a dating site at the moment. On the homepage I have a paginated list of all ...
1
vote
1answer
340 views

Is Googlebot case sensitive? Should I use a 301 redirect to choose a preferred URL case?

The Googlebot reports the following: Pages with duplicate meta descriptions /Default.aspx?p=2 /default.aspx?p=2 Should I use a 301 redirect from Default to default or vice versa? Is Googlebot ...
1
vote
1answer
4k views

How to fix Google 404 not found Crawl Errors?

I was checking on Google webmater tool for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
1
vote
1answer
847 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
1
vote
1answer
106 views

Googlebot fetching lots of URLs with a “workaround” parameter

A certain page on our website keeps generating entries like the following on our IIS6 logs: 2011-08-16 00:53:34 W3SVC943830190 10.20.101.146 GET /abc workaround=407707368055555 80 - 66.249.85.65 - ...
1
vote
1answer
44 views

Fetch as Google fails in Search Console (Webmaster Tools)

Most of the time, the Fetch as Google succeeds only partially. There's always some resources that cannot be retrieved. It says the URLs are temporarily unreachable. I thought it was something to do ...
1
vote
1answer
93 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
1
vote
1answer
36 views

How does an adsense bot determine the “Above the fold” space on mobile and desktop sites?

I ran an experiment last night by replacing a small ad for a mobile site thats located near the top of the page with a larger box ad, but I moved that ad towards the bottom of the page. Ten hours ...
1
vote
2answers
263 views

Conflicting test results when using The Mobile-Friendly Test and PageSpeed Insights

Our website uses an open source product to detect user agent and based on that, it servers mobile theme or desktop theme to users. I know the exact difference between mobile-friendly test and ...
1
vote
1answer
59 views

How to tell googlebot crawl more page from my site? [duplicate]

I have a small local news site, every one increase hundreds of articles. I have set a crontab, list new articles' url into a sitemap by hour. for example, I have 24 sitemaps which is generated by ...
1
vote
3answers
67 views

Will providing an internal graph allow the Google bot to index my site faster, and update it more often?

A programmer, but a very inexperienced webmaster here. I am building a social site that gets content generated literally every day. On top, the content of my already indexed pages changes quite often ...
1
vote
2answers
167 views

How to let Google's crawler know about new Atom/RSS updates?

What is the right way of letting Google's crawler know about new Atom/RSS updates? The documentation about sitemap is clear about how to ping the crawler, but when it comes to Atom/RSS, there is ...
1
vote
1answer
604 views

I get General HTTP error: 404 not found when submiting sitemap?

I just bought 2 domain from domaindiscount24.com. I get an error: "General HTTP error: 404 not found" when submitting sitemap. Then, I use "fetch as googlebot" and my website URL, it showed "not ...
1
vote
1answer
246 views

Google Adsbot suddenly crawling odd pages

I have an error checker on my password reset page which tells me everytime someone comes to reset their password without a valid token. I have noticed within the last couple of days that the ...
1
vote
2answers
264 views

Google crawl rate dropped after activating CloudFront

Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user ...
1
vote
1answer
579 views

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

I've been noticing this new Google range hitting our feeds over the last couple weeks. It's bordering on abusing our resources with multiple hits in short timeframe. The 173.194.90.xxx IP's fall under ...
1
vote
1answer
29 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
1
vote
1answer
212 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
1
vote
1answer
47 views

Is it possible to include news into Google News if there is a non-news content under the same section?

We want to include our news into Google News, and now performing research how to properly do this. At the moment we have the same base URL for news and non-news articles (analysis, interviews, etc.), ...
1
vote
3answers
138 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
1
vote
1answer
95 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
1answer
47 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
1answer
85 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
304 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
1
vote
2answers
130 views

Regarding Google Index -> Remove URLs

I have a question regarding Google index -> Remove URLs from Google Webmaster Tools. Basically I have replaced my site completely but upon searching Google, it returned some old pages that didn't ...
1
vote
1answer
20 views

How to recover after malicious code removal?

I just removed some malicious code from one of my client's website's theme. It was a result of a pirated theme the client was using and had ignored some hidden code with hyperlinks to betting sites. ...
1
vote
3answers
80 views

Create a way for Googlebot to find the deep pages in my site when users would find these pages via site search

I have a website like Foursquare, where users search for business in a specific location. The users can click on markers inside the map to view details about the companies/stores, and these details ...
1
vote
2answers
65 views

Can Googlebot crawl and index data shown on a map (powered by Google)?

I have a web service that includes mainly data on map. Can Google fetch these data (asynchronously loaded )? The solution I'm opting right now is to set the data URLs in a regular sitemap, but there ...
1
vote
1answer
390 views

Can Google bots interpret hyphen separated words as a single keyword?

I have a french website which frequently includes the word rendez-vous. But, Google Webmaster Tools content keywords tells me the #1 keyword is rendez which is not what I want. Even worse, the word ...
1
vote
1answer
45 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
1answer
752 views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
1
vote
1answer
572 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
1
vote
3answers
1k views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
1answer
188 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
1
vote
1answer
107 views

Why old (301) links stay on Google when breaking site down to multiple domains

Some background: We did have single site and single domain (let's call it mainsite.com) with product information, however things have changed since and product database has grown fast. So we decided ...
1
vote
3answers
359 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
1
vote
1answer
455 views

How to make image only website friendly to googlebot? [duplicate]

so I making this portfolio site for some artist. She draw all the menus and texts manually(on paper), then scanned, some "post processing" in photoshop and that's how the whole layout was created. ...
1
vote
2answers
587 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
106 views

Bots and scripts [duplicate]

Possible Duplicate: Will AJAX-fetched material be invisible to search engines? Can bots understand what's going on in javascript or vbscript? Most of the social networking modules and ad ...
1
vote
1answer
548 views

Googlebot visit but no cache update - why?

I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
1
vote
1answer
51 views

When will google report correct results with its fetch tool?

I visited Google Webmaster Tools today after resubmitting sitemaps for my website, and while google is happy to index more HTML pages of my site, it however has not indexed a single image yet. So I ...
1
vote
0answers
67 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
1
vote
2answers
46 views

Duplicate Sitemap URLs

I'm in the process of replacing my old sitemaps with a new form which should better represent our site. However, as bots can take days or even weeks to scan all of our urls after submitted I do not ...
1
vote
0answers
178 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site http://goo.gl/xH2Ffy suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
1
vote
2answers
119 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...