1
vote
1answer
43 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
1
vote
1answer
133 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
5
votes
2answers
169 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
1
vote
4answers
149 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
1answer
186 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
4
votes
3answers
127 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
1
vote
3answers
270 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
1
vote
0answers
68 views

SEO strategy for multiple pages per entity [closed]

I'm trying to decide a good SEO strategy for a web-site with multiple pages per entity. For example, there are main pages for companies which summarize the main information and each sub-page and ...
0
votes
0answers
15 views

Redirecting all 404 pages in Wordpress is not good SEO, but what is? [duplicate]

Maybe a silly question, but if you have a 404 error, than Google does not want to index your site. Of course you are going to say: 'who said that, that's not true', I know... But Google does not like ...
2
votes
1answer
90 views

Google Adsense before/above article in Wordpress: SEO problem?

I have set a banner before my article / content in Wordpress on my single.php. Will this be a problem if we look at it in SEO terms? So that Google can't crawl my article or crawl it less good because ...
1
vote
2answers
89 views

Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web. For him, this means that the only way to not being indexed ...
0
votes
0answers
97 views

repeating content consequences on bootstrap visible-desktop, visible-tablet, visible-phone

I am using bootstrap visible-desktop, visible-tablet, visible-phone to show different orders or span divs. Please see this page example: http://www.getaveo.com/snoring-causes/what-causes-snoring.aspx ...
1
vote
3answers
64 views

Using Ajax #! for Google but site is not being crawled any more

We used to have all links in a normal way (thus with parameters?), we changed to AJAX loading cause we installed a music player. Here's an example: Old link: ...
0
votes
2answers
91 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
1
vote
5answers
995 views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
4
votes
3answers
195 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...
2
votes
1answer
97 views

google removed links from the index [duplicate]

I asked google to crawl a site through webmaster tool and include it to the index. How actually it was done: Firstly, i picked some urls manually. Approx. after half of a day those pages were ...
1
vote
1answer
63 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
2answers
1k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
2
votes
1answer
82 views

Are the following assumptions about indexing and ranking correct?

Are the following statements true or false: If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt) An indexed page always gets a ranking (high ...
2
votes
1answer
163 views

Googlebot keeps returning to crawl my page then the rankings fluctuate

I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google. I have made a ...
1
vote
0answers
22 views

Does google crawl css scripts to check of hidden content [duplicate]

Possible Duplicate: hidden texts and page rank Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
2
votes
2answers
80 views

Google crawls my website too slowly [duplicate]

Possible Duplicate: What are the best ways to increase a site's position in Google? I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
1
vote
1answer
105 views

Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.

When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
4
votes
2answers
513 views

Understanding Ajax crawling of search site

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
179 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
5
votes
5answers
206 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
0
votes
1answer
43 views

Google spidering: make content available but unsuable

I'm trying to figure out how to make it so that my member-based social network site can be spidered but is not possible for guests to post or interact with content in any way. What is the best way to ...
5
votes
3answers
709 views

Should I use nofollow on internal links that don't need to be crawled?

We have a product navigation engine that allow our customers to filter products based on attributes (size, color, etc.). We are implementing an upgrade that allows them to use "multi-select" ...
3
votes
1answer
843 views

Is it possible to block an entire site to Google then list exceptions?

I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
5
votes
1answer
594 views

Am I covering the major search engine spiders in my anti-crawl protection white list?

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
3
votes
5answers
453 views

How to make sure that Google indexes your site in less than a day?

I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
2
votes
1answer
258 views

Do all search engines obey nofollow rel attribute on hyperlinks

I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value. Is this the case. From my research Google, Yahoo and Bing seem to obey this attribute ...
1
vote
2answers
107 views

SEO - Crawl Errors

I've been optimizing my website for search engines, and for some reason I have a crawl error on a page that has not been found at 'www.peach-designs.com/a' which is not a page, is not a link, or is ...
1
vote
1answer
519 views

How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?

I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects. It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
3
votes
4answers
533 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
2
votes
1answer
237 views

Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?

We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us. However, in addition to the hierarchical page ...
11
votes
3answers
3k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
3
votes
2answers
4k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
0
votes
1answer
60 views

Possible reasons RSS generated content won't index

I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and ...
12
votes
4answers
442 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...