Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) as independent entities. I would like to prevent search engines from indexing these pages, but do not have access to /robots.txt (which would be much more ideal).

My question is, could I include the following at the top of these partial HTML files and get the desired results?

<meta name="robots" content="noindex, noarchive">

I guess there are two parts to this question.

  1. Will this cause any rendering issues in any browsers?
  2. Will search engines (at least Google & Bing) interpret this as intended?
share|improve this question

4 Answers 4

Since you don't have access to robots.txt then .htaccess is also out of the question.

You might consider a HTTP HEADER.

See: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

X-Robots-Tag: noindex

PHP code example:

<?php header("X-Robots-Tag: noindex", true); ?>
share|improve this answer

If those pages are made with a server side language, you can include a verification for a constant or a variable, if that exist, then the file can be loaded, if it doesn't exist, you can redirect to another file or just block it.

It's much better to send a redirection.

You can do that also with the .htaccess file detecting who is requesting the file, for instance, if a bot is requesting the file, and you only want to send it to browsers, then you can block or redirect the bot before it reaches the server side language.

share|improve this answer

You can put the tag on the top of an HTML document. It will not cause rendering issues, since meta elements are not displayed, and this type of meta does not affect the way the page is rendered. Search engines will process it as in any other HTML document.

HTML specifications do not require body or head tags (except XHTML, but nobody cares that, unless you serve your document with an XML content type). An HTML document always has body and head elements, but their tags are optional. If you have a document containing just

 Hello world

and if it is served as text/html, it will be processed as an HTML document containing a body with that text as content and a head element with no content, as if the document contained

<html>
<head>
</head>
<body>
Hello world
</body>
</html>

(This is invalid, since a title element is required, but this does not prevent browsers and search engines as processing the data as HTML.) If you add a meta tag at the start, it will automatically become the content of the head element.

share|improve this answer

<meta> is intended to be in the <head>. If you inject in into an existing web page, you will end up with invalid markup (after ajax)

But you are drowning in a cup of water :-) If it's intended to be loaded via ajax. Surely you will not have any link <a> to these files. If you don't link to the files, the bots will not index them.

So, you are trying to solve a non existent problem here.

share|improve this answer
    
Normally, you would be right. But they're being AJAXed in via the Fancybox plugin, which is configured to parse anchor tags to load via AJAX in the generated popup. –  Dan Oct 31 '12 at 5:20
    
One potential option, I suppose, would be to add a rel="nofollow" to each of these anchor tags. I just don't know what kind of support search engines outside of Google will give that approach. –  Dan Oct 31 '12 at 5:21
    
AFAIK at least Google and Bing honor rel="nofollow". I'm not sure about Yahoo.. Anyone stills using it? –  The Disintegrator Nov 1 '12 at 3:19
    
Or you could modify the plugin to use data-atributes (part of html5) –  The Disintegrator Nov 1 '12 at 3:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.