I'm developing a site that's entire function is defined by the documents it provides. The documents are provided to the user both through embedding in the page (using the Google Drive Viewer) as well as through download links.

My question is, if I have the plaintext for each of these documents (ranging from PDFs to MS Office files), is there a good way to make this content visible to the search engine spiders?

I thought of including the entire doc file content in the alt attribute, but given many of these documents are over 100 pages, I would assume that will break something somewhere. Also thought about setting the download and href src to the plaintext then use JS to change the link destination onload, but I would assume search engines will detect and not look fondly on that.

Is there a good, standard practice for handling something like this or am I basically sol for getting this content noticed?

3 Answers 3


Use an XML Sitemap. It allows you to tell search engines about every file you wish for them to index.

  • I knew they supported PDFs, but are you sure about MS Office (including old non-x and new w/ x)? I rarely see them turn up in search results for Google (can't speak for the others), so I'm a bit skeptical...
    – Dan
    Jan 17, 2013 at 2:54
  • 2
    Naturally if a search engine doesn't support a certain format having it in your sitemap won't make a difference. But if they add support in the future you'll be ready without having to make any changes. Additionally, different search engines support different things. Microsoft may make Bing support .docx files before Google does.
    – John Conde
    Jan 17, 2013 at 2:56

If you search filetype:doc and filetype:docx and so on in Google, you'll see most common Office files, and PDFs, are routinely indexed.

They perhaps won't perform as well as a properly optimised web page, but for the kind of searches that people who want that sort of thing do (e.g., people looking for academic papers and so on), I can't see there being an issue there.

Incidentally, definitely don't stuff alt attributes with whole documents worth of text: I'd be concerned about that looking like spam.

+1 for John's Sitemap suggestion, too. That's always good practice.


What you should do:

  1. Ask users about sharing their docs over Internet
  2. Create sitemap.xml with users list
  3. Create sitemap_username1.xml with users data and list of its docs

I think you url structure looks

/ --root

/username/ -- user data

/username/docnameX -- doc

/username/docnameX?plaintext -- doc in plain text

OR even better case use #

but https://stackoverflow.com/questions/2181186/how-to-access-url-hash-fragment-from-a-django-request-object

but whocares https://stackoverflow.com/questions/3847870/php-to-get-value-of-hashtag-from-url

/username/docnameX#plaintext -- doc in plain text

So you create /sitemap.xml

with next

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

I got this example from http://edition.cnn.com/sitemap_index.xml

And yes, you can create gz archive with xml too


<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">



<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

Do not insert links to you text copy of document.

You may have problems with content duplicating. I did'n find right rules what have to do in current situation.

Google anyway find your text copy so just wait and check google cache with request like site:sitename.tld cache:sitename.tld.

If you find a lot of plaintext copies of docs on hight places you may try use this solutions https://stackoverflow.com/questions/677419/how-to-detect-search-engine-bots-with-php , https://stackoverflow.com/questions/10613025/how-can-i-use-serverhttp-referer-to-find-that-user-came-from-google

About big sitemaps https://stackoverflow.com/a/4241045/1346222 and http://dynamical.biz/blog/seo-technical/sitemap-strategy-large-sites-17.html

You can provide multiple Sitemap files, but each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 10MB (10,485,760 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 10MB. If you want to list more than 50,000 URLs, you must create multiple Sitemap files.

Prefer don't spoof SE (search engines) with lastmod changefreq priority - this is recommendation for SE and it will recalculate it anyway independently.

I thought of including the entire doc file content in the alt attribute

Good idea put in title 5-10 words from first paragraph of doc if you put all text SE may mark this as spam.

Good topic about sitemaps from Google http://googlewebmastercentral.blogspot.com/2008/01/sitemaps-faqs.html

  • "OR even better case use #" - If your intention is for the server to generate the plaintext version based on the value of the fragment identifier then this will not work, as you kind-of suggest, but gloss over with a link to an apparent (but invalid) workaround. The fragment id is not passed to the server. You also state, "Do not insert links to you text copy of document", but you appear to have done exactly that in the sitemap?
    – MrWhite
    Jan 17, 2013 at 12:17
  • @w3d just read topic whocares. On php you can get hash, in django too so this work. Delete example of bad sitemap.
    – b1_
    Jan 17, 2013 at 13:26
  • I'm sorry, but you are mistaken. In the SO question you link to, PHP is parsing a string of data which has not originated from the HTTP request. As stated in the accepted answer, "unless you have the url beforehand this really is pointless". And in the Django question, the answer states "This is not sent to the server". The fragment id the OP is seeing is apparently passed with a buggy CURL implementation (a one-off). It does not matter what server-side language you are using, you can't use the fragment id to request a server-side resource.
    – MrWhite
    Jan 17, 2013 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.