1
vote
1answer
57 views

Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code. How can I go about diagnosing this issue? The website is eCommerce business using the Magento ...
1
vote
2answers
37 views

Google Webmaster Tools linked from lists dead links

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link ...
2
votes
2answers
85 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
1
vote
1answer
49 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
69 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
0
votes
2answers
60 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?
1
vote
1answer
246 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
3
votes
3answers
111 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
1
vote
2answers
155 views

Fake Links Showing up in Google Webmaster Tools

Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
0
votes
2answers
172 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
2
votes
2answers
1k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> ...
0
votes
0answers
16 views

Redirecting all 404 pages in Wordpress is not good SEO, but what is? [duplicate]

Maybe a silly question, but if you have a 404 error, than Google does not want to index your site. Of course you are going to say: 'who said that, that's not true', I know... But Google does not like ...
1
vote
3answers
8k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
2
votes
1answer
228 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady ...
1
vote
1answer
73 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
2answers
496 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
2
votes
1answer
180 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
0
votes
2answers
389 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
3answers
2k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
5
votes
4answers
285 views

Total Indexed 0, 99 not selected in webmaster tools

I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
5
votes
1answer
232 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
3
votes
1answer
191 views

webmaster tools 500 crawl error for asp faceted navigation that does not exist

I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
8
votes
1answer
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
0
votes
1answer
79 views

How do I make the home page of the website to come up in the rankings than the internal pages? [duplicate]

Possible Duplicate: What are the best ways to increase your site's position in Google? Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search ...
1
vote
1answer
564 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
3
votes
1answer
369 views

How to interpret number of URL errors in Google webmaster tools

Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out ...
2
votes
2answers
1k views

404 errors showing in Google Webmaster Tools

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
2
votes
1answer
150 views

What's the extent of the “damage” to having a broken URL found by the spiders (showing in Google Webmaster Tools)?

An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site. I regularly watch for errors and fix database or coding issues when they arise. We ...
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...