1
vote
2answers
32 views

What to do about crawler errors after removing non-essential parts of my website?

So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors. I am ...
0
votes
1answer
47 views

DNS mistake. Google crawled the wrong server. What to do?

Problem Exists Between Keyboard And Chair I have a couple of similar looking domains on Coudflare where I manage their DNS. An old forum and my new still MVP site. As many of you may have ...
0
votes
1answer
81 views

Will making my AJAX app serve HTML snapshots improve SEO?

Currently, a small ajax web app of mine uses an anchor hijacking method to display content. The user clicks an anchor, I cancel the default browser event and funnel the link into an AJAX call, after ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
820 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
102 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
0
votes
1answer
50 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
2
votes
2answers
40 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
1
vote
1answer
78 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
5
votes
3answers
151 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
1
vote
1answer
70 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
0
votes
2answers
121 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
1answer
32 views

Custom Sitemap or sitemap generator tool [closed]

I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
1
vote
1answer
320 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
2
votes
1answer
121 views

Upgrading website from XHTML 1.0 Transitional to HTML 5: how can it affect SERP positions?

We have an enough old but still supported website which is XHTML 1.0 Transitional now. We are going to change it to HTML 5 to add some new features in the future (microdata, ...
4
votes
3answers
185 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
1
vote
2answers
96 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
0
votes
1answer
314 views

Serve different content for web crawlers to fix SEO for SPA?

I have my site built as an SPA using Knockout. The main-page populates a list of items dynamically as the user scrolls and each item has a details-page which is also dynamically loaded. I have ...
2
votes
2answers
83 views

Does changing phone numbers in content just for bots qualify as cloaking?

I'm a small business owner (I run a little computer repair shop), and I'm attempting to track calls to my store from different advertising channels. I use three separate tracking phone numbers: one ...
1
vote
2answers
99 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
0
votes
1answer
130 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
1
vote
1answer
73 views

Can I make query strings produce separate pages?

I have a profile page with a URL like so: localhost/profile.php/?username=Bob I was wondering, if I had a separate <title> which changed according to the username, would they produce separate ...
4
votes
3answers
142 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
1
vote
1answer
266 views

Checking Request's user agent versus handling _escaped_fragment_ urls when using pushState only (no hashbangs)

I'm working on an AJAX-based application and thinking of how to make it crawlable by search engines. I've read the Google's AJAX crawling specification but I couldn't get my head around one thing: if ...
2
votes
1answer
245 views

How to use NOINDEX,FOLLOW with paginated results except index page

I am running a classified website. At the moment I am using lazy loading with jquery to load content while scrolling. I got to know that using jquery loading is not good for search-engine bots ...
0
votes
2answers
124 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
2
votes
2answers
1k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> ...
0
votes
1answer
21 views

Indexing a page in which the content is discovered in a map

We want to improve our site's SEO. The page is a GWT (javascript based) web page and its content is discovered through a map or a GPS. Now we have just a normal landing html page, and the map and ...
0
votes
1answer
101 views

News like pages get index, but not showing up in SERP

Well I have some crawlers to scrape news from different news agency websites and auto-publish it to my own website, the thing is my pages get index in Google, but even if I search the exact same ...
1
vote
0answers
70 views

SEO strategy for multiple pages per entity [closed]

I'm trying to decide a good SEO strategy for a web-site with multiple pages per entity. For example, there are main pages for companies which summarize the main information and each sub-page and ...
0
votes
0answers
16 views

Redirecting all 404 pages in Wordpress is not good SEO, but what is? [duplicate]

Maybe a silly question, but if you have a 404 error, than Google does not want to index your site. Of course you are going to say: 'who said that, that's not true', I know... But Google does not like ...
2
votes
2answers
225 views

How to do a JavaScript redirection to the same page without hurting my SEO?

I did something that does that when people land on my page for the first time, they get a JavaScript redirection to the same page, but with PHP $_GET URL queries, then this page redirect back to the ...
1
vote
1answer
64 views

What are the future implications of blocking a search engine now?

At a question asked previously at this site regarding Baidu's overactive crawling tendencies, one user comments: I'd only block a search engine spider if you never want to receive any search ...
8
votes
3answers
1k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
2
votes
1answer
93 views

Google Adsense before/above article in Wordpress: SEO problem?

I have set a banner before my article / content in Wordpress on my single.php. Will this be a problem if we look at it in SEO terms? So that Google can't crawl my article or crawl it less good because ...
2
votes
1answer
67 views

Avoiding crawl time for noindex pages

I have pages with noindex in robots meta tag (specifically inner site search results pages). Because my site has many pages and Google hasn't crawled them all and I'm trying to help his crawler focus ...
1
vote
1answer
73 views

What is the proper SEO way to add and remove links to our site and sitemap?

As content fluctuates on our site, we will obviously add the titles to the new pages to our sitemap, and link list for search engine indexing. Through time, certain links will become less relevant, ...
4
votes
1answer
69 views

Appropriate robots meta for “printer friendly” pages

We have many printer-friendly pages that have a link that adds ?pf=1 to the URL. On the ?pf=1 page would it be better to have noindex,nofollow or noindex,follow. I think that it would be better to ...
2
votes
1answer
196 views

How can Google crawl content accessed via filter check-boxes?

I built a website that is divided into many categories. Each page displays the main category, and within that there is a check-box filter (like Amazon and eBay have when searching for items). I want ...
1
vote
3answers
97 views

Using Ajax #! for Google but site is not being crawled any more

We used to have all links in a normal way (thus with parameters?), we changed to AJAX loading cause we installed a music player. Here's an example: Old link: ...
4
votes
2answers
263 views

Google Webmaster Tools reports fake 404 errors

I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
2
votes
1answer
599 views

What does “all” mean in meta robots tag?

Several websites have the following meta tag aimed at search engines like Google, Bing, etc.: <meta name="robots" content="all" /> What does it do?
2
votes
1answer
195 views

Ajax crawling : snapshot page dissimilar with user page

For ajax-crawling purpose I make snapshots pages. My problem is user view pages and snapshot pages are dissimilar (tags h1, h2...). Example: user view page: http://www.meomap.net/#!/place/1 ...
6
votes
2answers
299 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
0
votes
2answers
106 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
4
votes
3answers
204 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...
0
votes
2answers
782 views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
1
vote
1answer
99 views

SEO preparation for upcoming new site

I booked a domain and I created an image saying that the site is soon to be launched. What SEO preparations should I make to avoid harming my SEO ranking? Because my page is blank, the image itself ...
1
vote
1answer
80 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...