Tagged Questions

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

0
votes
1answer
31 views

Can Googlebot detect duplicate video content?

I downloaded a lot of videos from YouTube and uploaded them to my hosting. Can Googlebot-video can detect that my video is duplicate?
2
votes
1answer
39 views

Does Googlebot determine the value of a link by its position on the page?

It's well-known that the SEO value of a website basically depends on how much it has been referenced, linked, and shared. From Googlebot's point of view, is a link at the top of a page more valuable ...
1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
0
votes
1answer
14 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
3
votes
1answer
66 views

Is having content on the 'above the fold' area of webpage good for SEO?

I have content (article) which is visible only when scrolled down. I have dedicated the 'above the fold' area of all my article webpages for visuals - graphics and H1 (header 1). Is there any ...
1
vote
2answers
39 views

Can Googlebot read any hosting stats?

When Google Analytics isn't active for a website, can Googlebot (or any other Google entity) get any traffic stats about that website in any other way?
2
votes
1answer
39 views

Why my webpages body content is not taken into account to display in google SERP?

It's confusing for me that I have content on my webpages, however, when I search with the site: operator it is not taken into account. There is something like "Thank you for contacting us...." ...
1
vote
1answer
157 views

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

I've been noticing this new Google range hitting our feeds over the last couple weeks. It's bordering on abusing our resources with multiple hits in short timeframe. The 173.194.90.xxx IP's fall under ...
1
vote
1answer
28 views

Google not indexing translated content using url params

I have a website that uses a URL param (http://example.com?lang=es) to translate each page content. I know that this option is not recommended, but technically is the only one I can provide atm. I ...
1
vote
1answer
25 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
0
votes
0answers
13 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
2
votes
2answers
44 views

For performance reasons: have different pages for search engines and users

I have a SEO question for single page applications. We're working on a SPA (not yet public) that displays tables of numeric data to users. The data volume is large, number aggregations are costly ...
0
votes
0answers
18 views

Fetch as google: Unreachable Error Message [duplicate]

A few days ago i changed my website CMS from vbulletin to MyBB 1.8.0 and all URLs structure changed, now I've encountering a very strange problem with "Fetch as Google"! Google can fetch my website ...
1
vote
1answer
45 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
0
votes
0answers
21 views

Is cloaking your site for disqus comments good for SEO? [duplicate]

I use "disqus" as the commenting platform on my website. There is quite a bit of SEO relevance for the comment content but since its generated through JAVASCRIPT. I can't get the best of it. Can I use ...
1
vote
1answer
43 views

Is creating Ajax Google Snapshot with less content than original page considered cloaking?

I've read a few posts about Google AJAX snapshot creation and I'm still asking myself a few questions: I have a full AJAX web site with really a lot of pages which differ from one to the other with a ...
6
votes
2answers
116 views

Base64 encoded images and availability of their metadata for googlebot

If i embed an image into a page as an img src with base64 data URI, are image's metadata (EXIF, IPTC, XMP) still available for google imagebot?
0
votes
2answers
104 views

Drop in traffic but increase in pages crawled per day

I saw a drop in traffic of my website in early September 2014. But, if I check the "Crawl Stats" in Google Webmaster Tools, I see an increase in "Pages Crawled per day" and "Kilobytes downloaded ...
1
vote
1answer
29 views

Is it possible to include news into Google News if there is a non-news content under the same section?

We want to include our news into Google News, and now performing research how to properly do this. At the moment we have the same base URL for news and non-news articles (analysis, interviews, etc.), ...
0
votes
1answer
37 views

Google wont apply adsense policies for session based websites and Ajax loaded sites?

I have seen so many websites (Mostly User generated content - Getting contents from Third party API's) still displaying Adult like contents (with images) but still they are running well with Google ...
4
votes
3answers
248 views

Bingbot requests from Google IP address

We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; ...
0
votes
2answers
41 views

Does Googlebot (and/or search engines) index a forwarded page? [duplicate]

Let's say I have example.com domain, and I force the user to use the HTTPS over HTTP. The question is as browsers just accept and load the forwarded/new page (when the request for http://example.com ...
1
vote
3answers
71 views

When Google gives up recrawling 301 that led to 404?

I've transferred a domain and made a mistake in the redirects (the URL structure is identical). Even though they went to the new domain, the error caused a 404 when crawled by Google bot. 10 days ...
0
votes
2answers
57 views

SEO: does Google index search result pages?

I have a site with a search form. When someone searches, the url for the results page has PHP variables in it and looks like: ...
0
votes
1answer
20 views

404 code/header for search engines, on removed user content?

I just got an email, from a former user on my website He was complaining that Google still shows the contact page he created on my site, even though he deleted it a month ago This is the first time ...
0
votes
1answer
30 views

Putting a website in a directory above the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution?

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
0
votes
2answers
64 views

Google Webmaster Tools crawl errors due to Google decoding links that were urlencoded by Codeigniter

In our pages we have internal links that are urlencoded because the URL may sometime contain special characters such as () or ! So in our source code we have a link: ...
2
votes
1answer
94 views

SEO: If another website uses a iframe from our website, would the links in the iframe appear to be coming from our website or from theirs?

we are creating a "app" on our website that allows affiliates to simply use a generated iframe on their website, which would produce a list of all our products (we sell event vouchers). The links ...
1
vote
1answer
26 views

How to avoid google from indexing like example.com/post/post/post/1-foobar.html?

I have a weird anomaly with one project. Where Google is indexing sites like: example.com/post/post/post/1-foobar.html While it should do it like: example.com/post/1-foobar.html I just ...
4
votes
5answers
149 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
1
vote
1answer
53 views

Does a jQuery “rel=nofollow” even work?

It's a fairly common practice for webmasters to rel=nofollow external links, and I frequently see it done with jQuery. The script typically looks something like this (off the cuff here): ...
2
votes
2answers
71 views

How to convince Google to list HTTPS version of website? [duplicate]

Having finally configured HTTPS for my company's website, I'm wondering now if there are any steps I can or should take to convince Google's search results to list the site as an HTTPS link rather ...
2
votes
0answers
32 views

Google isn't indexing AJAX URLs even after fetching them with _escaped_fragment

I'm still in the process of making my site crawlable through Google even though it's all AJAX content. I got it where if the URL contains escaped_fragment_ it retrieves the right information form the ...
2
votes
1answer
90 views

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more ...
0
votes
0answers
17 views

Searching for Googlebot in server logs [closed]

I'm trying to use Heorku Logentries to track Googlebot's activity on my server. (Although, I think this could broadly apply to tracking Googlebot activity on any server.) I can see that I have results ...
0
votes
0answers
8 views

Links removed from google search results [duplicate]

For my website www.trickwarehouse.com i have created one post on keyloggers. When i search in google that what are keyloggers? then my website's post position was 16 but after 1 hour, when i checked ...
0
votes
1answer
85 views

google webmaster tools, site maps and indexing, number of pages indexed reduces with no new sitemap

It seems to take two weeks ish to get updates on to the google index, which seems fair enough as I imagine googles index of the whole internet could be rather large. However, the some changes to my ...
2
votes
1answer
81 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
1
vote
2answers
56 views

Should you let Googlebot crawl and index your entire site before updating it again?

I have heard from various people that "Google has scanned x% of the site". Some have told me at times not to upload anything, nor do any changes on the site "since Google hasn't finished scanning the ...
1
vote
1answer
30 views

Does Googlebot share geo-location information to better crawl geo-aware websites?

I'm developing a website that allows users to share their location via JavaScript. Based on that location, either the content of the current page will change, or it will redirect a user to a page with ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
824 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
128 views

Does Fetch as Googlebot still support their ajax-crawling proposal?

I spent half a day implementing the server side html generation for modal pages based on their proposal (link), but it seems like the Fetch as Googlebot functionality in Webmaster tools completely ...
2
votes
2answers
105 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
2
votes
1answer
62 views

What if my HTML snapshot for Googlebot doesn't contain the same content as the dynamically created page?

According to Google here: The HTML snapshot must contain the same content as the dynamically created page. What if my returned HTML Snapshot doesn't contain the same content as the actual ...
8
votes
2answers
72 views

Informing Googlebot for deprecated pages

I publish timetables in my website. For example last year I published Number 2 bus Summer 2013 timetable. I has pretty good ranking on Google SERPs for number 2 bus timetable But this year I added a ...
1
vote
1answer
30 views

SEO - All navigation links or just relevant navigation links

I'm trying to decide what the best approach is for rendering the static HTML for the websites navigation links. I have two possible options 1) render all links for every page, and hide non-relevant ...
1
vote
0answers
55 views

How to handle URL encoded parameter separators when responding to AJAX crawled _escaped_fragment?

I've been reading about how to handle Google requested URL here There is an example from Google : Pretty URL: www.example.com?myquery#!key1=value1&key2=value2 Ugly URL: ...
7
votes
4answers
397 views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
3
votes
3answers
45 views

Indexed webpages stopped rising, while Submited keep rising normally

Last months I have noticed a strange behaviour in Google's Webmaster Tools. Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while ...