Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
0answers
50 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
0
votes
0answers
31 views

google displays wrong language on multi language website because it takes cookies into consideration

I have a multiple language website that uses both subdomains and cookies to display the right language. Here is the logic on the website: de subdomain will display the German version fr subdomain ...
0
votes
0answers
12 views

Remove all pages from site [duplicate]

I have a staging server that accidentally got indexed by google. I want to remove ALL pages from google for that site: i.e. stagingserver.mycompany.com. I added a robots.txt to the root of the site ...
1
vote
1answer
60 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
0
votes
1answer
20 views

Changing pagination, and having more sections per page

I'm upgrading an old website, from having 25 results per page (40 pages), to having 50-100 per page (5-10 pages) This will change the pagination, and maybe affect search engines in a bad way.. So I'm ...
3
votes
1answer
73 views

Why is the date displayed in the Google search result is 3 days older than the actual publication date?

Yesterday I published an article on my website, if I go on Google it says that it has been published 3 days ago. Why? I've WordPress, on a Centos 6,5 VM. System time is wrong of 20 min of my server, ...
3
votes
1answer
88 views

Does every page HAVE to be mobile friendly even if its almost unimportant?

I received the following regarding a URL of a page recommending a feature that only applies to desktop users: Faulty Redirect URLs redirect to an irrelevant landing page instead of the ...
1
vote
1answer
30 views

Fetch as Google Bot not working for `www.` version of site but is for `non-www.` version - Webmaster Tools

Ive recently acquired a new domain that i want to point at our existing domain, ive setup hosting for the new domain and then done a 301 redirect in the .htaccess file of the new domain to my existing ...
0
votes
1answer
11 views

Testing server indexed in Google [duplicate]

Assume our website domain to be www.example.com. And we have created a separate testing environment within our main domain: Say : www.project.example.com. So we have two versions within our main ...
0
votes
2answers
41 views

Soft 404 for smartphones

We have redirected all non-existing pages to home page. It was fine for some days. Few days back i saw increase in the number of soft 404 errors for desktop. And the errors were the page urls that ...
0
votes
2answers
37 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
1
vote
1answer
25 views

How does an adsense bot determine the “Above the fold” space on mobile and desktop sites?

I ran an experiment last night by replacing a small ad for a mobile site thats located near the top of the page with a larger box ad, but I moved that ad towards the bottom of the page. Ten hours ...
4
votes
3answers
114 views

For webpages using jQuery, will the Googlebot index them on “page load” or after the “page ready” state?

Will the Googlebot index a webpage on "page load" or after the "page ready" state. The reason I'm asking is because I'm trying to change my h1 and h2 tags using jQuery on page Load.
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
4
votes
3answers
161 views

How do I prevent Google from indexing text that is generated by JavaScript?

Google is executing the JavaScript on my password creator website and indexing the random passwords that get generated by the JavaScript. The random text that is generated could look spammy to ...
4
votes
1answer
62 views

Does Google now factor the prominence of how a link is rendered into their PageRank algorithm?

Google's John Mueller now says that Googlebot renders the pages it crawls to determine what content is on them. They do this because so many sites are now heavily JavaScript centric and this allows ...
1
vote
2answers
169 views

Is this really GoogleBot, or is this a spammer?

I found this IP in my access log posting repeatedly thousands of times. Is this really GoogleBot, or is it a spammer? I hesitate to block it. 185.62.188.98 - - [13/May/2015:18:54:33 -0400] "POST ...
1
vote
2answers
38 views

Duplicate Sitemap URLs

I'm in the process of replacing my old sitemaps with a new form which should better represent our site. However, as bots can take days or even weeks to scan all of our urls after submitted I do not ...
3
votes
3answers
74 views

Facets: Googlebot encountered extremely large numbers of links on your site

We are encountering the following issue: Googlebot encountered extremely large numbers of links on your site. The site is an ecommerce site and has around 12M pages indexed in Google. From the ...
3
votes
1answer
129 views

Webmaster Tools shows URL errors linked from http://com.example.www/:http

I have a bog standard WordPress 4.2.1 site. I recently added it to Webmaster Tools and it is showing me URL errors (404s) that are linked from a weird URL: http://com.example.www/:http (I have ...
2
votes
1answer
35 views

rel=“prev” and rel=“next” inserting using javascript

does adding the rel="next" tag after the page is loaded with a small javascript still useful for SEO and google? i have a problem in my code, basically the total number of pages in the catalog is ...
0
votes
0answers
9 views

Block search engine from indexing Currency Words as content keywords [duplicate]

I have one website with currency converter on it and Google WMT is considering the currency words as Content keywords. Is there anyway to block search engine from indexing this Currency Words like ...
0
votes
1answer
41 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
1
vote
0answers
84 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site http://goo.gl/xH2Ffy suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
1
vote
1answer
221 views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
2
votes
1answer
181 views

Using Fetch as Google on AngularJS

I am working on an eCommerce site and I have been asked to look at the SEO implications of our product listing page. I have done all the necessary steps to get the webmaster tools working fine on one ...
1
vote
2answers
91 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
3
votes
3answers
318 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...
1
vote
0answers
47 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
8
votes
4answers
486 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
1
vote
2answers
230 views

Conflicting test results when using The Mobile-Friendly Test and PageSpeed Insights

Our website uses an open source product to detect user agent and based on that, it servers mobile theme or desktop theme to users. I know the exact difference between mobile-friendly test and ...
1
vote
1answer
23 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
0
votes
0answers
21 views

Google Crawler constantly reports ~2-3% errors

For the last year and a half, I have a constant 2-3% error rate while crawling my web site. Types of errors are "Server error: No response". I've switched ISP, but errors persist. Debugging of ...
0
votes
0answers
17 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...
0
votes
0answers
13 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
0
votes
1answer
100 views

Does Googlebot see dynamically loaded content? [closed]

Our modelling agency website uses the Photolux Wordpress theme. Our front page uses the same masonry template that the demo site: Demo. The only difference is that we've modified the masonry elements ...
2
votes
1answer
268 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
67 views

Using table tag to structure a web page decrease referencing quality? [duplicate]

I am student in web programing. I use massively <table> ... </table> to structure my web page. My professor once told me it could decrease the referencing quality for my websites, so I ...
1
vote
2answers
53 views

Would it hurt my rankings to have the same URL display similar content depending on the visitor's country?

I manage a website that has different Web pages that use the same URL structure, and I don't use = & ? But depending on your location we will show you different links. Let say you are in the UK ...
1
vote
1answer
53 views

How to tell googlebot crawl more page from my site? [duplicate]

I have a small local news site, every one increase hundreds of articles. I have set a crontab, list new articles' url into a sitemap by hour. for example, I have 24 sitemaps which is generated by ...
1
vote
1answer
63 views

What URL with French words does Google like?

We are looking for an answer to "what to do with French URL's?" We are going to generate a lot of French URL's this include characters like: L'test tést thîs is a test So we have a few ...
2
votes
2answers
186 views

URLs for dialogs from Bootstrap modals popups are getting indexed in search engines, but without nav they aren't good landing pages

On my side, I use bootstrap modals (aka dialogs / overlays) with remote content to display the detail view of some things (for example for detail view of user reviews/comments). The code looks like ...
0
votes
0answers
7 views

How to influence the info column in Google's search result? [duplicate]

Google often shows an info column next to the search results, if it recognizes your search query to be directly related to a "thing" they know. Example: You search for "beatles" and get a box with ...
1
vote
3answers
65 views

Will providing an internal graph allow the Google bot to index my site faster, and update it more often?

A programmer, but a very inexperienced webmaster here. I am building a social site that gets content generated literally every day. On top, the content of my already indexed pages changes quite often ...
1
vote
2answers
128 views

How to let Google's crawler know about new Atom/RSS updates?

What is the right way of letting Google's crawler know about new Atom/RSS updates? The documentation about sitemap is clear about how to ping the crawler, but when it comes to Atom/RSS, there is ...
1
vote
0answers
105 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
4
votes
2answers
878 views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
0
votes
1answer
42 views

How to tell how long google thinks it has been since your website was updated?

I never really thought 'freshness' was a problem for my site until I recently used the silktide 'nibbler' website grading report / scanner. It listed my 'freshness' as a big old 0 and said my site ...
1
vote
1answer
444 views

I get General HTTP error: 404 not found when submiting sitemap?

I just bought 2 domain from domaindiscount24.com. I get an error: "General HTTP error: 404 not found" when submitting sitemap. Then, I use "fetch as googlebot" and my website URL, it showed "not ...