Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Having a real problem here, I've tried lots of services to sort this including http://www.xml-sitemaps.com/ as well as software and a PHP script.

Basically my domain is http://gamingdeluxe.co.uk/

Whenever I run it through a sitemap generator it ONLY finds that 1 page, and doesn't follow any of the links.

However robots.txt is setup as

User-agent: *
Allow: /

so that shouldn't be a problem?

I found a Java crawler called DRKSpiderJava and it crawled my site sucessfuly—shame no export to .xml feature. so they seem to be crawlable.

Any ideas why I cant generate one?

share|improve this question

migrated from stackoverflow.com Jul 31 '11 at 1:17

This question came from our site for professional and enthusiast programmers.

worked for me just fine. – nix Jul 29 '11 at 22:07
I've no idea what happened. I just kept trying and trying and then suddenly one of the sites worked, then all the others did :s very confusing. – Owen Melbourne Jul 29 '11 at 22:11

I would start by trying making your site any "w3c norm-compliant".

Use some validation engine (like http://validator.w3.org/check) to ensure your site is easily parsable without errors...

After trying to parse your home page, I see some critical errors like:

  • Unable to Determine Parse Mode!,
  • No DOCTYPE found, and unknown root element. Aborting validation.

...and a lot of other (16 errors mentioned my w3.org validator)

Maybe that is why no standard external parsers can do the job for you...

share|improve this answer
again not sure what happened, I just loaded the website through w3 and it came out as "Passed! 1 Warning" - and that was the html5 standards are not 100% reliable at the moment. – Owen Melbourne Jul 29 '11 at 22:14

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.