Having a real problem here, I've tried lots of services to sort this including http://www.xml-sitemaps.com/ as well as software and a PHP script.

Basically my domain is http://gamingdeluxe.co.uk/

Whenever I run it through a sitemap generator it ONLY finds that 1 page, and doesn't follow any of the links.

However robots.txt is setup as

User-agent: *
Allow: /

so that shouldn't be a problem?

I found a Java crawler called DRKSpiderJava and it crawled my site sucessfuly—shame no export to .xml feature. so they seem to be crawlable.

Any ideas why I cant generate one?

  • worked for me just fine.
    – nix
    Jul 29, 2011 at 22:07
  • I've no idea what happened. I just kept trying and trying and then suddenly one of the sites worked, then all the others did :s very confusing.
    – Owen Melbourne
    Jul 29, 2011 at 22:11

1 Answer 1


I would start by trying making your site any "w3c norm-compliant".

Use some validation engine (like http://validator.w3.org/check) to ensure your site is easily parsable without errors...

After trying to parse your home page, I see some critical errors like:

  • Unable to Determine Parse Mode!,
  • No DOCTYPE found, and unknown root element. Aborting validation.

...and a lot of other (16 errors mentioned my w3.org validator)

Maybe that is why no standard external parsers can do the job for you...

  • again not sure what happened, I just loaded the website through w3 and it came out as "Passed! 1 Warning" - and that was the html5 standards are not 100% reliable at the moment.
    – Owen Melbourne
    Jul 29, 2011 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.