Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a client for whom we have purchased Google Site Search. It works well for HTML pages served by the CMS, but files aren't being reliably indexed.

I wrote a script to generate an XML feed (sitemap) of all the files in the CMS which I've plugged in to Google Webmaster Tools for the site. It says that for that sitemap 923 URLs have been submitted, but only 26 have been indexed.

The client relies heavily on searching within files, which is why we decided to use Google search, so this is a bit of a problem.

Many of the files aren't linked to from any page on the site, as they are old and therefore don't merit having a page of their own. But they still need to be accessible through search for archiving purposes.

The file archive xml can be found at www.sniffer.org.uk/file-archive and the standard xml sitemap (of pages) can be found at www.sniffer.org.uk/sitemap.xml.

Any thought would be much appreciated!

share|improve this question
add comment

1 Answer

I'm not seeing a big problem with indexing.

If I search on google for site:www.sniffer.org.uk it shows 2640 indexed pages and documents. There are lots of PDF and RTF documents indexed when I click through the pages of the SERPs.

I spot checked about five files from the sitemap.xml and ten files from file-archive. In all cases, I opened the document or page, copied a sentence out of it. Put it in quotes, and search for it on Google. Here is an example that I tried from the file-archive: https://www.google.com/search?q=%22redeveloped+housing+requires+retrofit%22

One problem I came across was a page in the sitemap.xml that is 404 on the website: http://www.sniffer.org.uk/?cID=253

When I did this search: https://www.google.com/search?q=%22which+soil+issues+are+addressed+in+other+countries%22 Google appears to only index a copy of the document on some other site. It didn't return http://www.sniffer.org.uk/files/5713/4183/7996/UKLQ01.pdf

share|improve this answer
    
I added some explanation to the OP -- many of the files aren't linked to from any page on the site, as they are effectively archived and don't merit their own page on the site. –  melat0nin Feb 7 '13 at 16:02
    
I checked 5 more of the old files from the file_archive, and found one that wasn't indexed on sniffer.org.uk because Google was choosing to index it on another site where it is duplicated. I updated my answer that info. Still not seeing pervasive indexing problems. –  Stephen Ostermiller Feb 7 '13 at 16:29
    
Thanks for doing those checks. I've updated the script which generates the XML feed of files so it's a static file located at sniffer.org.uk/file-archive.xml. I read something about Google not allowing XML files in a directory to reference files outside that directory (ie /file-archive/ can only reference files beneath within that directory, recursively) so having the static file in the site's root might hopefully solve that. That was on Thursday and 26/944 files have been indexed so far, so might just need to wait :) –  melat0nin Feb 12 '13 at 10:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.