Should I block an .htm resource in robots.txt

Question

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls.

For example, a script that builds a file every 5 minutes that contains a list of our latest forum threads. It would be intensive to request this information from the database on every page load (as the list is in a site-wide sidebar), so we build a static file on the server that we request instead.

The issue is, the file has been found by Google Webmaster Tools, and it gets marked as either thin content or that it's missing H1 tags etc. So our solution was to add the file to our robots.txt file so Google Webmaster Tools would ignore it. However, now the file is coming up under 'blocked resources'.

What is best practice in this case?

Should the file remain blocked? Or do I unblock it, and have it as thin content?

Answer 1

For this .htm file to be considered a "blocked resource" it must be linked to in the client-side HTML. Perhaps as the source of an iframe? Serving content in an iframe is generally bad for SEO and when Google comes to index this file it will naturally be seen as "thin content".

This HTML snippet needs to be cached server-side and included server-side when the page is delivered to the client. Then, as far as the user-agent is concerned, there is no separate .htm file. If it's linked server-side then it can be stored in a location outside of the public HTML folders.

Answer 2

What I would do is place the file in a password protected directory (or a directory not visible for the users and Google) and use the file just when is needed for the include.

See this article for how to use .htaccess to password protect a directory.