4

Having a real problem here, I've tried lots of services to sort this including http://www.xml-sitemaps.com/ as well as software and a PHP script.

Basically my domain is http://gamingdeluxe.co.uk/

Whenever I run it through a sitemap generator it ONLY finds that 1 page, and doesn't follow any of the links.

However robots.txt is setup as

User-agent: *
Allow: /

so that shouldn't be a problem?

I found a Java crawler called DRKSpiderJava and it crawled my site sucessfuly—shame no export to .xml feature. so they seem to be crawlable.

Any ideas why I cant generate one?

migrated from stackoverflow.com Jul 31 '11 at 1:17

This question came from our site for professional and enthusiast programmers.

  • worked for me just fine. – nix Jul 29 '11 at 22:07
  • I've no idea what happened. I just kept trying and trying and then suddenly one of the sites worked, then all the others did :s very confusing. – Owen Melbourne Jul 29 '11 at 22:11
1

I would start by trying making your site any "w3c norm-compliant".

Use some validation engine (like http://validator.w3.org/check) to ensure your site is easily parsable without errors...

After trying to parse your home page, I see some critical errors like:

  • Unable to Determine Parse Mode!,
  • No DOCTYPE found, and unknown root element. Aborting validation.

...and a lot of other (16 errors mentioned my w3.org validator)

Maybe that is why no standard external parsers can do the job for you...

  • again not sure what happened, I just loaded the website through w3 and it came out as "Passed! 1 Warning" - and that was the html5 standards are not 100% reliable at the moment. – Owen Melbourne Jul 29 '11 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.