Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
0answers
12 views

How does a search engine work? [on hold]

How does a search engine work? How does search engine bots crawl our website server/host. Is there any legal agreement amount servers providers and search engines, for which they crawl servers for ...
-1
votes
1answer
39 views

Can Googlebot detect duplicate video content?

I downloaded a lot of videos from YouTube and uploaded them to my hosting. Can Googlebot-video can detect that my video is duplicate?
2
votes
1answer
42 views

Does Googlebot determine the value of a link by its position on the page?

It's well-known that the SEO value of a website basically depends on how much it has been referenced, linked, and shared. From Googlebot's point of view, is a link at the top of a page more valuable ...
1
vote
1answer
144 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
2answers
40 views

Can Googlebot read any hosting stats?

When Google Analytics isn't active for a website, can Googlebot (or any other Google entity) get any traffic stats about that website in any other way?
0
votes
2answers
429 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
3
votes
1answer
66 views

Is having content on the 'above the fold' area of webpage good for SEO?

I have content (article) which is visible only when scrolled down. I have dedicated the 'above the fold' area of all my article webpages for visuals - graphics and H1 (header 1). Is there any ...
0
votes
1answer
14 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
1
vote
4answers
870 views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
83 views

Does the Instant Preview in Google webmaster tools takes Robot.txt in account?

Is that the way to go If I want to visually see what the googlebot see? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as google bot, It fetches ok, ...
1
vote
1answer
46 views

Is creating Ajax Google Snapshot with less content than original page considered cloaking?

I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions: I have a full AJAX web site with really a lot of pages which differ from one to the other with a ...
2
votes
1answer
39 views

Why my webpages body content is not taken into account to display in google SERP?

It's confusing for me that I have content on my webpages, however, when I search with the site: operator it is not taken into account. There is something like "Thank you for contacting us...." ...
1
vote
1answer
198 views

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

I've been noticing this new Google range hitting our feeds over the last couple weeks. It's bordering on abusing our resources with multiple hits in short timeframe. The 173.194.90.xxx IP's fall under ...
1
vote
1answer
28 views

Google not indexing translated content using url params

I have a website that uses a URL param (http://example.com?lang=es) to translate each page content. I know that this option is not recommended, but technically is the only one I can provide atm. I ...
1
vote
1answer
25 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
0
votes
0answers
13 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
2
votes
2answers
44 views

For performance reasons: have different pages for search engines and users

I have a SEO question for single page applications. We're working on a SPA (not yet public) that displays tables of numeric data to users. The data volume is large, number aggregations are costly ...
0
votes
0answers
18 views

Fetch as google: Unreachable Error Message [duplicate]

A few days ago i changed my website CMS from vbulletin to MyBB 1.8.0 and all URLs structure changed, now I've encountering a very strange problem with "Fetch as Google"! Google can fetch my website ...
1
vote
1answer
46 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
0
votes
0answers
21 views

Is cloaking your site for disqus comments good for SEO? [duplicate]

I use "disqus" as the commenting platform on my website. There is quite a bit of SEO relevance for the comment content but since its generated through JAVASCRIPT. I can't get the best of it. Can I use ...
4
votes
3answers
922 views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
0
votes
2answers
64 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
6
votes
2answers
121 views

Base64 encoded images and availability of their metadata for googlebot

If i embed an image into a page as an img src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for google imagebot?
0
votes
2answers
112 views

Drop in traffic but increase in pages crawled per day

I saw a drop in traffic of my website in early September 2014. But, if I check the "Crawl Stats" in Google Webmaster Tools, I see an increase in "Pages Crawled per day" and "Kilobytes downloaded ...
2
votes
3answers
219 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
1
vote
1answer
29 views

Is it possible to include news into Google News if there is a non-news content under the same section?

We want to include our news into Google News, and now performing research how to properly do this. At the moment we have the same base URL for news and non-news articles (analysis, interviews, etc.), ...
7
votes
3answers
212 views

Does Googlebot support TLS v1.2? If not, what will my TLS v1.2-only site look like to it?

I currently redirect all non-HTTPS connections to HTTPS, and support TLS v1, TLS v1.1, and TLS v1.2. I reconfigured my testing server to accept only a very limited and secure cipher suite and TLSv1.2 ...
0
votes
1answer
39 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
1
vote
1answer
308 views

Is Googlebot case sensitive? Should I use a 301 redirect to choose a preferred URL case?

The Googlebot reports the following: Pages with duplicate meta descriptions /Default.aspx?p=2 /default.aspx?p=2 Should I use a 301 redirect from Default to default or vice versa? Is Googlebot ...
2
votes
0answers
33 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
4
votes
3answers
264 views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; ...
1
vote
3answers
73 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
0
votes
2answers
41 views

Does Googlebot (and/or search engines) index a forwarded page? [duplicate]

Let's say I have example.com domain, and I force the user to use the HTTPS over HTTP. The question is as browsers just accept and load the forwarded/new page (when the request for http://example.com ...
0
votes
2answers
57 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
3
votes
2answers
96 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
0
votes
1answer
20 views

404 code/header for search engines, on removed user content?

I just got an email, from a former user on my website He was complaining that Google still shows the contact page he created on my site, even though he deleted it a month ago This is the first time ...
2
votes
2answers
106 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
4
votes
5answers
149 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
0
votes
1answer
30 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
2
votes
1answer
98 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
1
vote
1answer
28 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
1
vote
1answer
55 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
2
votes
2answers
71 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
1
vote
2answers
402 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
0
votes
1answer
281 views

Will using page headings (h1, h2, h3 etc) as internal links encourage Google sitelinks? [duplicate]

I am trying to change my site's page headings to better emphasize the structure of the page's content hierarchy. I am hoping this will encourage Google Sitelinks and list snippets. Current headings ...
0
votes
0answers
18 views

Searching for Googlebot in server logs [closed]

I'm trying to use Heorku Logentries to track Googlebot's activity on my server. (Although, I think this could broadly apply to tracking Googlebot activity on any server.) I can see that I have results ...
2
votes
1answer
92 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
0
votes
0answers
8 views

Links removed from google search results [duplicate]

For my website www.trickwarehouse.com i have created one post on keyloggers. When i search in google that what are keyloggers? then my website's post position was 16 but after 1 hour, when i checked ...
0
votes
1answer
91 views

google webmaster tools, site maps and indexing, number of pages indexed reduces with no new sitemap

It seems to take two weeks ish to get updates on to the google index, which seems fair enough as I imagine googles index of the whole internet could be rather large. However, the some changes to my ...