Crawlers that generate sitemaps only find one page on my site despite being allowed to crawl by robots.txt

Question

Having a real problem here, I've tried lots of services to sort this including http://www.xml-sitemaps.com/ as well as software and a PHP script.

Basically my domain is http://gamingdeluxe.co.uk/

Whenever I run it through a sitemap generator it ONLY finds that 1 page, and doesn't follow any of the links.

However robots.txt is setup as

User-agent: *
Allow: /

so that shouldn't be a problem?

I found a Java crawler called DRKSpiderJava and it crawled my site sucessfuly—shame no export to .xml feature. so they seem to be crawlable.

Any ideas why I cant generate one?

Answer 1

I would start by trying making your site any "w3c norm-compliant".

Use some validation engine (like http://validator.w3.org/check) to ensure your site is easily parsable without errors...

After trying to parse your home page, I see some critical errors like:

• Unable to Determine Parse Mode!,
• No DOCTYPE found, and unknown root element. Aborting validation.

...and a lot of other (16 errors mentioned my w3.org validator)

Maybe that is why no standard external parsers can do the job for you...