Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm developing a site that's entire function is defined by the documents it provides. The documents are provided to the user both through embedding in the page (using the Google Drive Viewer) as well as through download links.

My question is, if I have the plaintext for each of these documents (ranging from PDFs to MS Office files), is there a good way to make this content visible to the search engine spiders?

I thought of including the entire doc file content in the alt attribute, but given many of these documents are over 100 pages, I would assume that will break something somewhere. Also thought about setting the download and href src to the plaintext then use JS to change the link destination onload, but I would assume search engines will detect and not look fondly on that.

Is there a good, standard practice for handling something like this or am I basically sol for getting this content noticed?

share|improve this question

3 Answers 3

What you should do:

  1. Ask users about sharing their docs over Internet
  2. Create sitemap.xml with users list
  3. Create sitemap_username1.xml with users data and list of its docs

I think you url structure looks

/ --root

/username/ -- user data

/username/docnameX -- doc

/username/docnameX?plaintext -- doc in plain text

OR even better case use #

but http://stackoverflow.com/questions/2181186/how-to-access-url-hash-fragment-from-a-django-request-object

but whocares http://stackoverflow.com/questions/3847870/php-to-get-value-of-hashtag-from-url

/username/docnameX#plaintext -- doc in plain text

So you create /sitemap.xml

with next

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap>
        <loc>http://sitename.tld/sitemap_users.xml</loc>
        <lastmod>2007-04-18T12:05:20-04:00</lastmod>
    </sitemap>
    <sitemap>
        <loc>http://sitename.tld/sitemap_else.xml.gz</loc>
        <lastmod>2006-07-28T08:42:17-04:00</lastmod>
    </sitemap>
</sitemapindex>

I got this example from http://edition.cnn.com/sitemap_index.xml

And yes, you can create gz archive with xml too

/sitemap_users.xml

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap>
        <loc>http://sitename.tld/sitemap_username1.xml</loc>
        <lastmod>2007-04-18T12:05:20-04:00</lastmod>
    </sitemap>
    <sitemap>
        <loc>http://sitename.tld/sitemap_username2.xml</loc>
        <lastmod>2006-07-28T08:42:17-04:00</lastmod>
    </sitemap>

</sitemapindex>

/sitemap_username1.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>http://sitename.tld/username1/</loc>
        <lastmod>2012-07-27</lastmod>
        <changefreq>monthly</changefreq>
        <priority>0.8</priority>
    </url>
    <url>
        <loc>http://sitename.tld/username1/docname1</loc>
        <lastmod>2011-09-18</lastmod>
        <changefreq>daily</changefreq>
        <priority>0.4</priority>
    </url>
</urlset>

Do not insert links to you text copy of document.

You may have problems with content duplicating. I did'n find right rules what have to do in current situation.

Google anyway find your text copy so just wait and check google cache with request like site:sitename.tld cache:sitename.tld.

If you find a lot of plaintext copies of docs on hight places you may try use this solutions http://stackoverflow.com/questions/677419/how-to-detect-search-engine-bots-with-php , http://stackoverflow.com/questions/10613025/how-can-i-use-serverhttp-referer-to-find-that-user-came-from-google

About big sitemaps http://stackoverflow.com/a/4241045/1346222 and http://dynamical.biz/blog/seo-technical/sitemap-strategy-large-sites-17.html

You can provide multiple Sitemap files, but each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 10MB (10,485,760 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 10MB. If you want to list more than 50,000 URLs, you must create multiple Sitemap files.

Prefer don't spoof SE (search engines) with lastmod changefreq priority - this is recommendation for SE and it will recalculate it anyway independently.

I thought of including the entire doc file content in the alt attribute

Good idea put in title 5-10 words from first paragraph of doc if you put all text SE may mark this as spam.

Good topic about sitemaps from Google http://googlewebmastercentral.blogspot.com/2008/01/sitemaps-faqs.html

share|improve this answer
    
"OR even better case use #" - If your intention is for the server to generate the plaintext version based on the value of the fragment identifier then this will not work, as you kind-of suggest, but gloss over with a link to an apparent (but invalid) workaround. The fragment id is not passed to the server. You also state, "Do not insert links to you text copy of document", but you appear to have done exactly that in the sitemap? –  w3d Jan 17 '13 at 12:17
    
@w3d just read topic whocares. On php you can get hash, in django too so this work. Delete example of bad sitemap. –  b1- Jan 17 '13 at 13:26
    
I'm sorry, but you are mistaken. In the SO question you link to, PHP is parsing a string of data which has not originated from the HTTP request. As stated in the accepted answer, "unless you have the url beforehand this really is pointless". And in the Django question, the answer states "This is not sent to the server". The fragment id the OP is seeing is apparently passed with a buggy CURL implementation (a one-off). It does not matter what server-side language you are using, you can't use the fragment id to request a server-side resource. –  w3d Jan 17 '13 at 15:51

If you search filetype:doc and filetype:docx and so on in Google, you'll see most common Office files, and PDFs, are routinely indexed.

They perhaps won't perform as well as a properly optimised web page, but for the kind of searches that people who want that sort of thing do (e.g., people looking for academic papers and so on), I can't see there being an issue there.

Incidentally, definitely don't stuff alt attributes with whole documents worth of text: I'd be concerned about that looking like spam.

+1 for John's Sitemap suggestion, too. That's always good practice.

share|improve this answer

Use an XML Sitemap. It allows you to tell search engines about every file you wish for them to index.

share|improve this answer
    
I knew they supported PDFs, but are you sure about MS Office (including old non-x and new w/ x)? I rarely see them turn up in search results for Google (can't speak for the others), so I'm a bit skeptical... –  Dan Jan 17 '13 at 2:54
2  
Naturally if a search engine doesn't support a certain format having it in your sitemap won't make a difference. But if they add support in the future you'll be ready without having to make any changes. Additionally, different search engines support different things. Microsoft may make Bing support .docx files before Google does. –  John Conde Jan 17 '13 at 2:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.