Tagged Questions

1
vote
3answers
44 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
1
vote
1answer
22 views

Directing crawlers to content in language per language sub-domain

I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per ...
0
votes
2answers
54 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
104 views

Get google to index JavaScript pages as separate pages

I have a product page with say 10 products, that loads up all the content when the page loads. Then JavaScript hides content till a product is selected. Then JavaScript hides and unhides information. ...
2
votes
1answer
94 views

How frequently Googlebot fetch sitemaps? Is it depending on PageRank?

How much frequently Google fetches sitemaps? I am now working with a high traffic website normally have 30 new posts per minute but currently it provides sitemaps which links to new 100 posts (3 ...
1
vote
2answers
638 views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
0answers
118 views

Site returning 404 header to google, not sure why

A Drupal site that works fine for regular users returns a 404 not found error when I try to use the W3C validator on it; it is also not being indexed by google at all (which is the main issue but I ...
2
votes
1answer
534 views

Google on capital letters in url

I created one of my websites with the following url structure: http://domain.com/Animal/Pig http://domain.com/Pigs If I change the url map to lower case letters instead, will google reindex those ...
1
vote
2answers
136 views

When will my old page stop appearing on Google? [duplicate]

Possible Duplicate: Old pages still appearing in Google? I recently bought a new address for my Blogger blog, from yannbane.blogspot.com to www.yannbane.com. However, www.yannbane.com ...
3
votes
1answer
387 views

Webmaster Tools show the “Total Indexed” number to be zero

I recently switched my domain from yannbane.blogspot.com to www.yannbane.com, and added my new domain to my Webmaster Tools account. After a few days, I went there and checked the data, and I was a ...
4
votes
1answer
853 views

link rel=“alternate”, multiple languages and canonical urls

Context: We've got a website which is available in multiple language versions. However, the content is the same. Each translated version of the site is available on a distinct subdomain. On some ...
0
votes
2answers
88 views

Hide elements for Google

I have some elements which I don't want Google to see. Usually repetitive content, e.g. header and sidebar. As Googlebot doesn't handle AJAX requests, I load these resources with AJAX and insert them ...
-3
votes
2answers
122 views

Does Google bot crawl unused tags?

If a website a called foo.com has the following css, #LinkBuilder{ background:url("www.LoremEpsum.com"); } and the css id LinkBuilder is actually not getting used anywhere in the html of foo.com, ...
3
votes
1answer
162 views

Google 404 errors on webmaster page

I have a lot of 404 errors on my webmaster's page. I don't know how google ended up on those urls. I checked my site many times to see if any links to those pages appear, but found nowhere. Is google ...
2
votes
2answers
361 views

What does Google do with indexed pages returning 403?

I guess that Google removes already indexed pages that are now returning the HTTP error 403. Unfortunately I can't find any definitive statement on that matter. Do you know the answer?
0
votes
1answer
217 views

How does Google feel about overflow: auto?

I am planning to add some keyword rich text to my homepage at the very top. I have written almost a mini essay and putting all this text at the very top left of my page wouldn't look too nice. I was ...
4
votes
2answers
664 views

Does duplicate content inside your own website matter for SEO? [duplicate]

Possible Duplicate: What is duplicate content and how can I avoid being penalized for it on my site? I'm not referring to the duplicate content where you copy content from another site and ...
1
vote
2answers
291 views

Is hiding content with JavaScript or “text-indent: -9999px” bad for SEO?

So apparently hiding content using "display: none" is bad for SEO and seen by googlebot as being deceptive. This according to a lot of the posts I read online and questions even on this site. But ...
3
votes
1answer
426 views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
1
vote
1answer
203 views

Google search : pubdate issue

At some point this year, Google Search started to show the publication date. When I updated my website, I noticed that the publication date was actually my birthday, simply because it's the first ...
1
vote
1answer
83 views

Bots and scripts [duplicate]

Possible Duplicate: Will AJAX-fetched material be invisible to search engines? Can bots understand what's going on in javascript or vbscript? Most of the social networking modules and ad ...
1
vote
1answer
527 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
1
vote
1answer
260 views

Should i use a 301 redirect? Is Googlebot case sensitive?

The Googlebot reports the following: Pages with duplicate meta descriptions /Default.aspx?p=2 /default.aspx?p=2 So my question is, should i use a 301 redirect from Default to default or vice ...
9
votes
2answers
239 views

How is Googlebot finding URLs that are only visible to authenticated users?

Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp. 94.254.xxx.xxx - - [02/Jul/2011:22:25:46 ...
1
vote
1answer
410 views

Googlebot visit but no cache update - why?

I have made a new plain vanilla HTML website. I have been making regular modifications to it on an almost daily basis. The site is hosted by hostmonster and as part of their service they offer ...
5
votes
1answer
528 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
3
votes
5answers
371 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
4answers
318 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
1answer
208 views

Google AdSense bot's algorithm and behavior [closed]

I am interesting in Google AdSense bot's algorithm and behavior with web site. I did not work with AdSense and I do not have account. So I need your help to understand: 1) Google's bot from time to ...
8
votes
3answers
2k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...