A list of all urls that are accessed at my domain

Question

I have run a website for years. It's mostly a wordpress install but I also scp lots of files up there to host them and regularly do quick-and-dirty projects with static html updated by some sort of script. All these projects get thrown in directory structure somewhere.

It's a mess that has built up over a decade and I'd like to clean it up (and potentially move to another hosting company from Dreamhost) But I don't want to break any of the inbound links without giving them sensible redirects.

So my question is: can I, via dreamhost, get a list of all the pages/files (or even attempted to access) that have been accessed at the domain?

Answer 1

This is what the server's access logs are for. However, depending on your hosting provider (control panel settings / server config) these logs may not be stored for longer than a month.

You would analyse your access log for all 200 OK responses where the HTTP Referer is not your own site (internal link). And probably exclude all static resources (images, CSS, JS files etc.)

You can also check Google Analytics (or other stats software) if you are using that.

If you have registered your site with Google Search Console (GSC) then there is a report of inbound links from other sites. However, this doesn't tell you if users have bookmarked site pages or typed the URL directly.

Answer 2

It looks like DreamHost supports web-based statistics using awstats: https://help.dreamhost.com/hc/en-us/articles/216661708

I'm not sure if it will retroactively load in data from your account's existing access logs if you haven't already got it enabled, but using something like this to view traffic data is a little bit easier to chew than the raw log files.