I've been running a site for a while now and my AWstats page is starting to present pages with... some sort of endless stream of references to pages. It's a bit bizarre, I'm not sure what's happened.

AWstats page

  • AWStats is based on your server logs, so someone is visiting those URLs. Possibly a spider. Make sure your website is returning a proper 404 error for URL that doesn't actually exist. – GeoffAtkins Dec 4 '19 at 9:13

When you see subdirectories on top of subdirectories that do not exist, it's most likely a miscoded relative URL that a bot is hitting.

I would check your site and update to absolute URLs, as they can remedy this problem AND it can help from your entire site being scraped and duplicated at a different domain.

  • 1
    I would recommend using root relative URLs rather than absolute URLs. All links to other pages, css, js, and images should start with a slash. <a href="/articles.php">Articles</a>. I found the site in question and it looks like they are trying to solve this problem using ../articles.php as the link which may help some, but only for a single directory level. – Stephen Ostermiller Dec 5 '19 at 14:03
  • Hi @StephenOstermiller, curious why you recommend relative URLs over absolute? I know relative has benefits during development but I feel potential negatives can outweigh the positives. – Rob T Dec 5 '19 at 15:15
  • Using absolute URLs in every link can substantially increase document sizes. Root relative links (starting with the slash) have none of the drawbacks of <a href='article.php'> style relative links and only add one extra character. I recommend that they are used in conjunction with rewrite rules to ensure a canonical host name and HTTPS. – Stephen Ostermiller Dec 5 '19 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.