I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) as independent entities. I would like to prevent search engines from indexing these pages, but do not have access to /robots.txt (which would be much more ideal).

My question is, could I include the following at the top of these partial HTML files and get the desired results?

<meta name="robots" content="noindex, noarchive">

I guess there are two parts to this question.

  1. Will this cause any rendering issues in any browsers?
  2. Will search engines (at least Google & Bing) interpret this as intended?

5 Answers 5


You can put the tag on the top of an HTML document. It will not cause rendering issues, since meta elements are not displayed, and this type of meta does not affect the way the page is rendered. Search engines will process it as in any other HTML document.

HTML specifications do not require body or head tags (except XHTML, but nobody cares that, unless you serve your document with an XML content type). An HTML document always has body and head elements, but their tags are optional. If you have a document containing just

 Hello world

and if it is served as text/html, it will be processed as an HTML document containing a body with that text as content and a head element with no content, as if the document contained

Hello world

(This is invalid, since a title element is required, but this does not prevent browsers and search engines as processing the data as HTML.) If you add a meta tag at the start, it will automatically become the content of the head element.


<meta> is intended to be in the <head>. If you inject in into an existing web page, you will end up with invalid markup (after ajax)

But you are drowning in a cup of water :-) If it's intended to be loaded via ajax. Surely you will not have any link <a> to these files. If you don't link to the files, the bots will not index them.

So, you are trying to solve a non existent problem here.

  • Normally, you would be right. But they're being AJAXed in via the Fancybox plugin, which is configured to parse anchor tags to load via AJAX in the generated popup.
    – Dan
    Oct 31, 2012 at 5:20
  • One potential option, I suppose, would be to add a rel="nofollow" to each of these anchor tags. I just don't know what kind of support search engines outside of Google will give that approach.
    – Dan
    Oct 31, 2012 at 5:21
  • AFAIK at least Google and Bing honor rel="nofollow". I'm not sure about Yahoo.. Anyone stills using it? Nov 1, 2012 at 3:19
  • Or you could modify the plugin to use data-atributes (part of html5) Nov 1, 2012 at 3:20

If those pages are made with a server side language, you can include a verification for a constant or a variable, if that exist, then the file can be loaded, if it doesn't exist, you can redirect to another file or just block it.

It's much better to send a redirection.

You can do that also with the .htaccess file detecting who is requesting the file, for instance, if a bot is requesting the file, and you only want to send it to browsers, then you can block or redirect the bot before it reaches the server side language.


Since you don't have access to robots.txt then .htaccess is also out of the question.

You might consider a HTTP HEADER.

See: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

X-Robots-Tag: noindex

PHP code example:

<?php header("X-Robots-Tag: noindex", true); ?>

Unfortunaly - and this isn't documented by google - it works. I've a client site with a ajax adv that injected a asynchronously into body, and Google deindex that page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.