Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
0answers
146 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
1
vote
2answers
42 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
3answers
43 views

Strategy for mobile redirection and using alternative and canonical links

I had a separate mobile site uploaded so as to make Google happy. My big problem is that I had a desktop page that was once showing in the top 3 results on Google, and now it and its mobile ...
1
vote
1answer
126 views

Fetch as Google fails in Search Console (Webmaster Tools)

Most of the time, the Fetch as Google succeeds only partially. There's always some resources that cannot be retrieved. It says the URLs are temporarily unreachable. I thought it was something to do ...
2
votes
1answer
81 views

Is google penalizing SEO of websites with non device-width viewports?

These articles make it look like they are: https://support.google.com/webmasters/answer/6101188?hl=en https://developers.google.com/web/fundamentals/layouts/rwd-fundamentals/set-the-viewport This ...
1
vote
3answers
241 views

Googlebot couldn't get all resources for this page

When I fetch and render it is showing partial. It looks fine. The same way Google sees it is the same way visitors see it. The reason for this is that I've added BuySellaAd codes in a widget onto my ...
2
votes
1answer
73 views

When will Google report correct results with its fetch tool?

I visited Google Webmaster Tools(Google Search Console) today after resubmitting sitemaps for my website, and while Google is happy to index more HTML pages of my site, it however has not indexed a ...
4
votes
0answers
553 views

How Googlebot discovers new website? [duplicate]

I have purchased a domain and uploaded with HTML Page and When I search in Google it shows my website. Now my question is How Google will discover my new website? I have not used ...
0
votes
0answers
67 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
0
votes
1answer
94 views

Why does Google Cache break my iframes?

In HTML5, iframe comes as a replacement for frameset. However, when I load my iframe-based page in SeaMonkey through Google Cache, the page appears empty, other than the standard Google Cache header ...
3
votes
1answer
71 views

Replace all images when search engine crawler visits site

I have written code that will detect when a user is a search engine and i dont include css files when this happens to save bandwidth. I was also thinking i could replace ALL images on the page with a ...
3
votes
1answer
119 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
1
vote
0answers
53 views

How post or page of facebook, twitter, forum or classified sites (HIGH PR) show instantally in google or other search engines

My question is about SEO. How post or page of facebook, twitter, forum or classified sites(HIGH PR) show instantally in google or other search engines. As i know google take few days for crawl. Then ...
1
vote
0answers
95 views

Googlebot follows rel=“nofollow” links

I would like to hide all pages of our website from being read or indexed by Googlebot, except several thousand of pages which were selected for indexing. The site was launched in April 2015. The ...
0
votes
0answers
96 views

google displays wrong language on multi language website because it takes cookies into consideration

I have a multiple language website that uses both subdomains and cookies to display the right language. Here is the logic on the website: de subdomain will display the German version fr subdomain ...
0
votes
0answers
14 views

Remove all pages from site [duplicate]

I have a staging server that accidentally got indexed by google. I want to remove ALL pages from google for that site: i.e. stagingserver.mycompany.com. I added a robots.txt to the root of the site ...
1
vote
1answer
203 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
2
votes
4answers
63 views

Changing pagination, and having more sections per page

I'm upgrading an old website, from having 25 results per page (40 pages), to having 50-100 per page (5-10 pages) This will change the pagination, and maybe affect search engines in a bad way.. So I'm ...
4
votes
1answer
215 views

Why is the date displayed in the Google search result is 3 days older than the actual publication date?

Yesterday I published an article on my website, if I go on Google it says that it has been published 3 days ago. Why? I've WordPress, on a Centos 6,5 VM. System time is wrong of 20 min of my server, ...
3
votes
1answer
103 views

Does every page HAVE to be mobile friendly even if its almost unimportant?

I received the following regarding a URL of a page recommending a feature that only applies to desktop users: Faulty Redirect URLs redirect to an irrelevant landing page instead of the ...
1
vote
1answer
104 views

Fetch as Google Bot not working for `www.` version of site but is for `non-www.` version - Webmaster Tools

Ive recently acquired a new domain that i want to point at our existing domain, ive setup hosting for the new domain and then done a 301 redirect in the .htaccess file of the new domain to my existing ...
0
votes
1answer
20 views

Testing server indexed in Google [duplicate]

Assume our website domain to be www.example.com. And we have created a separate testing environment within our main domain: Say : www.project.example.com. So we have two versions within our main ...
0
votes
2answers
66 views

Soft 404 for smartphones

We have redirected all non-existing pages to home page. It was fine for some days. Few days back i saw increase in the number of soft 404 errors for desktop. And the errors were the page urls that ...
1
vote
3answers
178 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
1
vote
1answer
58 views

How does an adsense bot determine the “Above the fold” space on mobile and desktop sites?

I ran an experiment last night by replacing a small ad for a mobile site thats located near the top of the page with a larger box ad, but I moved that ad towards the bottom of the page. Ten hours ...
5
votes
3answers
245 views

For webpages using jQuery, will the Googlebot index them on “page load” or after the “page ready” state?

Will the Googlebot index a webpage on "page load" or after the "page ready" state. The reason I'm asking is because I'm trying to change my h1 and h2 tags using jQuery on page Load.
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
4
votes
3answers
312 views

How do I prevent Google from indexing text that is generated by JavaScript?

Google is executing the JavaScript on my password creator website and indexing the random passwords that get generated by the JavaScript. The random text that is generated could look spammy to ...
4
votes
1answer
76 views

Does Google now factor the prominence of how a link is rendered into their PageRank algorithm?

Google's John Mueller now says that Googlebot renders the pages it crawls to determine what content is on them. They do this because so many sites are now heavily JavaScript centric and this allows ...
1
vote
3answers
738 views

Is this really GoogleBot, or is this a spammer?

I found this IP in my access log posting repeatedly thousands of times. Is this really GoogleBot, or is it a spammer? I hesitate to block it. 185.62.188.98 - - [13/May/2015:18:54:33 -0400] "POST ...
1
vote
2answers
98 views

Duplicate Sitemap URLs

I'm in the process of replacing my old sitemaps with a new form which should better represent our site. However, as bots can take days or even weeks to scan all of our urls after submitted I do not ...
3
votes
3answers
110 views

Facets: Googlebot encountered extremely large numbers of links on your site

We are encountering the following issue: Googlebot encountered extremely large numbers of links on your site. The site is an ecommerce site and has around 12M pages indexed in Google. From the ...
3
votes
1answer
236 views

Webmaster Tools shows URL errors linked from http://com.example.www/:http

I have a bog standard WordPress 4.2.1 site. I recently added it to Webmaster Tools and it is showing me URL errors (404s) that are linked from a weird URL: http://com.example.www/:http (I have ...
2
votes
1answer
85 views

rel=“prev” and rel=“next” inserting using javascript

does adding the rel="next" tag after the page is loaded with a small javascript still useful for SEO and google? i have a problem in my code, basically the total number of pages in the catalog is ...
0
votes
0answers
10 views

Block search engine from indexing Currency Words as content keywords [duplicate]

I have one website with currency converter on it and Google WMT is considering the currency words as Content keywords. Is there anyway to block search engine from indexing this Currency Words like ...
0
votes
1answer
148 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
1
vote
0answers
293 views

Google Webmaster Tools can't access robots.txt and the home page is “temporarily unreachable”

My site http://goo.gl/xH2Ffy suddenly disappeared from Google search and Webmaster Tools says it can't access robots.txt. Fetch as Google says "temporarily unreachable" for robots.txt and even for the ...
1
vote
1answer
904 views

Webmaster Tools “Fetch as Google” doesn't fetch my js/css/images on a CDN

I've discovered that when I do Google Webmaster Tools --> Fetch as Google on a page on my site (let's call it http://example.com), it shows the static assets (js, css, and images) as Temporarily ...
2
votes
1answer
643 views

Using Fetch as Google on AngularJS

I am working on an eCommerce site and I have been asked to look at the SEO implications of our product listing page. I have done all the necessary steps to get the webmaster tools working fine on one ...
1
vote
2answers
171 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
3
votes
3answers
598 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...
1
vote
0answers
57 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
9
votes
4answers
1k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
1
vote
2answers
328 views

Conflicting test results when using The Mobile-Friendly Test and PageSpeed Insights

Our website uses an open source product to detect user agent and based on that, it servers mobile theme or desktop theme to users. I know the exact difference between mobile-friendly test and ...
1
vote
1answer
26 views

Changing headers for small screens - SEO implications

I'm changing some of my pages to better display in different devices (tables, smart phones etc). I want to figure out a good way of handling headings. So far my options are hide headings for small ...
0
votes
0answers
26 views

Google Crawler constantly reports ~2-3% errors

For the last year and a half, I have a constant 2-3% error rate while crawling my web site. Types of errors are "Server error: No response". I've switched ISP, but errors persist. Debugging of ...
0
votes
0answers
23 views

Question About Traffic From Google And GEO Location OF Website

I want to know lets say a person who lives in South Wimbledon, UK, runs a website and then outsource the website work (like writing articles and posting and other site management work etc..) to a ...
0
votes
0answers
14 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
0
votes
1answer
215 views

Does Googlebot see dynamically loaded content? [closed]

Our modelling agency website uses the Photolux Wordpress theme. Our front page uses the same masonry template that the demo site: Demo. The only difference is that we've modified the masonry elements ...