2

I have a client for whom we have purchased Google Site Search. It works well for HTML pages served by the CMS, but files aren't being reliably indexed.

I wrote a script to generate an XML feed (sitemap) of all the files in the CMS which I've plugged in to Google Webmaster Tools for the site. It says that for that sitemap 923 URLs have been submitted, but only 26 have been indexed.

The client relies heavily on searching within files, which is why we decided to use Google search, so this is a bit of a problem.

Many of the files aren't linked to from any page on the site, as they are old and therefore don't merit having a page of their own. But they still need to be accessible through search for archiving purposes.

The file archive XML can be found at www.sniffer.org.uk/file-archive and the standard XML sitemap (of pages) can be found at www.sniffer.org.uk/sitemap.xml.

Improve this question

1 Answer 1

Reset to default
1

I'm not seeing a big problem with indexing.

If I search on google for site:www.sniffer.org.uk it shows 2640 indexed pages and documents. There are lots of PDF and RTF documents indexed when I click through the pages of the SERPs.

I spot checked about five files from the sitemap.xml and ten files from file-archive. In all cases, I opened the document or page, copied a sentence out of it. Put it in quotes, and search for it on Google. Here is an example that I tried from the file-archive: https://www.google.com/search?q=%22redeveloped+housing+requires+retrofit%22

One problem I came across was a page in the sitemap.xml that is 404 on the website: http://www.sniffer.org.uk/?cID=253

When I did this search: https://www.google.com/search?q=%22which+soil+issues+are+addressed+in+other+countries%22 Google appears to only index a copy of the document on some other site. It didn't return http://www.sniffer.org.uk/files/5713/4183/7996/UKLQ01.pdf

Improve this answer
4
  • I added some explanation to the OP -- many of the files aren't linked to from any page on the site, as they are effectively archived and don't merit their own page on the site.
    – melat0nin
    Commented Feb 7, 2013 at 16:02
  • I checked 5 more of the old files from the file_archive, and found one that wasn't indexed on sniffer.org.uk because Google was choosing to index it on another site where it is duplicated. I updated my answer that info. Still not seeing pervasive indexing problems.
    – Stephen Ostermiller ♦
    Commented Feb 7, 2013 at 16:29
  • Thanks for doing those checks. I've updated the script which generates the XML feed of files so it's a static file located at sniffer.org.uk/file-archive.xml. I read something about Google not allowing XML files in a directory to reference files outside that directory (ie /file-archive/ can only reference files beneath within that directory, recursively) so having the static file in the site's root might hopefully solve that. That was on Thursday and 26/944 files have been indexed so far, so might just need to wait :)
    – melat0nin
    Commented Feb 12, 2013 at 10:19
  • file-archive.xml, this file was generated and posted to google with this name or you've changed the name? Because if it was sitemap.xml and you renamed it after you submit it to google you would have to resubmit it in Webmaster tools with the new name.
    – knif3r
    Commented Feb 23, 2016 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.