Tagged Questions

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

learn more… | top users | synonyms (1)

1
vote
1answer
143 views

Subdomain and Googlebot can't access your site

I have a subdomain and Google Webmaster Tools, told me this: Googlebot can't access your site This is not the first time that I receive this message from Google's Webmaster Tools. I need to ...
5
votes
1answer
195 views

How to interpret Google policy on crawling and indexing search results pages?

I just learned that we can be penalized for letting Google crawl and index our database search results. http://webmasters.stackexchange.com/a/55599/33777 Question: If they don't want to list Yellow ...
2
votes
1answer
343 views

Googlebot received Access Denied when crawling a directory

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory ...
0
votes
1answer
90 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
2
votes
2answers
749 views

Can Googlebot handle robots.txt with a 302 redirect?

In Google Webmaster Tools, I get an error: Google couldn't crawl your site because we were unable to access your site's robots.txt file The associated help says: If your robots.txt file ...
1
vote
5answers
1k views

How can I make Google crawl content behind an AJAX load more button

I have a feed that has 10 questions and answers loaded when you first open the page. And then I have a "Load More" button that gets you 10 new questions and answers everytime you click on it. How can ...
2
votes
1answer
39 views

I have a DoFollow backlink but I can't go directly to my site without a confirmation to leave it

I have a DoFollow backlink to a forum and I checked it. My concern is before I can reach my site using that link, there is a message: You're leaving the site, are you sure you wanted to continue? ...
1
vote
1answer
176 views

Creating sitemap for Googlebot - how to mark dynamic content / dynamic subpages?

I have a website that is an Internet forum. This forum has many categories, and a single category page that contains a lot of subpages with listed threads. This Internet forum is brand new, and about ...
1
vote
2answers
88 views

Somehow Google considers a properly 301'd URL as 200 and is still indexing the new content in old page?

We redirected all the old URL's to new ones properly using .htaccess. The problem is Google, somehow is still finding content in the old page (which it shouldn't) and stores it in the cache rather ...
0
votes
1answer
271 views

Will using page headings (h1, h2, h3 etc) as internal links encourage Google sitelinks? [duplicate]

I am trying to change my site's page headings to better emphasize the structure of the page's content hierarchy. I am hoping this will encourage Google Sitelinks and list snippets. Current headings ...
1
vote
1answer
40 views

Forum is removed but still being visited by Googlebot

I've removed Simple Machine Forum (SMF) from my blog and from Google's index . I've even removed the sub-directory /forum from my blog techwayz.com and blocked it through robots.txt file. ...
1
vote
4answers
858 views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
0
votes
1answer
209 views

In addition to Google's First Click Free, should you whitelist search engine bots past a paywall?

Our site has subscription-only pages - non-subscribed visitors see a snippet preview. As per Google's FCF requirements, your first 5 hits to a subscriber-only pages with .google. as the referrer, you ...
4
votes
2answers
186 views

When Googlebot sees a link, will it click it or navigate to it?

My site uses pushState and JSON data to display content. So, for example, this might appear on my page: <a href="/some/page">some page</a> The JavaScript then prevents the default ...
2
votes
1answer
282 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
5
votes
1answer
554 views

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page: <p>This is a paragraph that will be indexed by Google.</p> <!--googleoff: index--> <p>This is a ...
3
votes
2answers
542 views

Foolproof way to ensure Google news pulls the correct image for its thumbnails?

Google news results have an acompanying thumbnail next to articles that show up in the results. If Google's crawler can't find a thumbnail to pull from our site, it uses its next best guess from ...
1
vote
1answer
363 views

Do web crawlers/spiders index azure web sites?

For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is ...
3
votes
2answers
156 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
2
votes
1answer
6k views

Is “Identity not verified” a problem? If so, how to solve it?

When I select the context menu item "View Page Info" on my site (using Chrome), I see "Identity not verified" Should I be: (a) concerned (b) angry (c) nonchalant ? If one of the first two, what ...
2
votes
1answer
367 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
1
vote
1answer
409 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
0
votes
1answer
35 views

Why does my htm pages have a number attached at the end of the link?

When I wanted to fetch a certain page with htm extension I noticed an attachment after the htm extenstion http://www.example.com/IP_814.htm#.Uk1_MtKsh8E Google could not fetch this page it gave the ...
2
votes
2answers
80 views

Will multilingual content on same page works to avoid content duplication

We have a website, which is having old/new vehicle deals for purchase. We are getting these data from third party website (legally) and all the pages are in English language. So, Here our issue is ...
7
votes
2answers
3k views

Google-bot sees “Sorry, we have no imagery here” on pages with Google Maps

I have a site with Google Maps on most of the pages. When inspecting content keywords in Google Webmaster tools, content keywords identified by Google-bot for the site include "imagery", "sorry" and ...
2
votes
4answers
1k views

Does updating site too frequently hurt SEO rankings?

I recently updated my site, like 3-4 times in 2 months. At first everything was good, (i.e., Google was updating my pages' SERP content in 2-3 days). But now I did kind of a major change, I switched ...
1
vote
3answers
770 views

What are the problems of httponly and secure cookies?

We would like to set: all our cookies (session id, etc.) as "httponly" all security (session token, "remember me", etc.) cookies as "secure" Except compatiblity issues, what are the know problems ...
1
vote
3answers
141 views

Is it bad to hide content for SEO?

I'm designing a mobile CSS file for my website. It normally has a top navigation bar, like in here, Stack Exchange's sites. In my mobile version, I plan to hide the topbar, and put a button, which ...
1
vote
0answers
70 views

Suggest that Googlebot read “dezign” as “design”

I used the spelling "dezign" instead of "design" many times in my website -- even in my domain name. Is there any way to suggest that Googlebot read "dezign" word as "design" with correct spelling?
0
votes
1answer
98 views

Is it safe to manipulate with DOM (using JavaScript) to move up in the search results?

I have some text written for SEO purposes and it is located right after the opening body tag, so it is at the top of the page. That doesn't look great so this text is moved to the bottom of the page ...
2
votes
1answer
162 views

New URL after doing redirect using 301

If we do a redirect 301, one my expired job to another URL in my site, Google will record this redirection. But what if this original job URL appears again say after 1 year. Will Google remember this ...
0
votes
1answer
112 views

Need Google to recrawl the pages

I've changed the meta description tag in many of my site pages. Should I select Fetch as Google->Fetch->Add to Index->URL and all linked pages in GWT? Will Google recrawl the whole site faster than ...
1
vote
2answers
209 views

Should I expect Search Engines (particularly Google) to crawl epub and mobi files

Some of our content we are publishing as epub and mobi eBook format files. Will search engines crawl and index these in the same way as PDFs currently are. If the answer is yes, could you point me to ...
0
votes
2answers
205 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
1
vote
1answer
151 views

Will Google index a PHP header redirect?

I have a new domain which has an index.php file. Inside this file, I get user's IP, save it and redirect to another page on another domain. So there is no output, only header info. Will Google index ...
1
vote
1answer
153 views

Will Google crawl a news/RSS aggregator site?

I know how to code. I know how to set up a news/RSS aggregator site. But I don't know if Google will crawl it. Will Google crawl such a site? The site will have a listing of all the aggregated news ...
3
votes
3answers
501 views

How did Google find my unlinked newly created pages?

I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory anywhere, yet searching Google reveals that Google has ...
3
votes
2answers
160 views

Do error pages get indexed?

I have just started to put custom error pages on my site. But then I started wondering if google will index these error pages. I searched this up and I didn't get anything, even searched this site, so ...
2
votes
2answers
474 views

Is there any benefit to put rel='nofollow' on mailto links?

Since bots have a fixed budget to spend to crawl websites on a regular basis, does it makes sense to mark mailto links with rel='nofollow' to make sure them don't spend time on these (especially if ...
5
votes
3answers
2k views

Are penalties applied if you have too many 404 errors on your site?

Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
0
votes
2answers
2k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
1
vote
3answers
1k views

How much time does Googlebot take to crawl the page after submitting it to index

I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority. I've a page linking to newly added jobs ...
1
vote
2answers
400 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
106 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
93 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...
1
vote
1answer
113 views

Google attempting crawl of URLs with parameter randomly placed in URL

We have a weird SEO issue; In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors. There are a set of several hundred dynamic pages which we need to maintain (though not ...
1
vote
1answer
132 views

Google visits on Statcounter. How?

In my website’s Statcounter report, I found page views from Google. I thought it may be a Google bot’s visit but I found that particular visit comes through a search query (example.com+keyword) and it ...
1
vote
3answers
163 views

New or same URL on reactivation of user generated content?

We run a classified ad website. Our users can temporarily deactivate their classified ads. From a SEO point of view, when they reactivate it, should the ad use the same URL or use a new one? Each ...
2
votes
1answer
97 views

How to have Google crawl pages that are only up for a week

We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
3
votes
1answer
339 views

Can I use meta refresh instead of 301 redirect?

I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42. Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...