In Screaming Frog, when I do a basic crawl of one of my website's in the 'Internal' tab it shows a sub-domain at the very top. This sub-domain was one I had previously created, but I removed it quite a while ago (over a half year ago or so). I created and deleted it all in CPanel.

Screaming Frog lists the status code for the sub-domain as 0 and the status is DNS lookup failed which I think sounds good, but I'm little confused why it's showing still. Perhaps more importantly would be to ask if this could cause of any problems or if it's even worth my time to figure a way to remove it?

Another odd detail is that I do have 2 other sub-domains for this site and it's not listing either one of those.

  • 2
    If Screaming Frog is able to find this subdomain I would assume that it is because you are referencing (or linking to) it in your page(s)?
    – MrWhite
    Aug 24 '15 at 19:33
  • I thought I deleted all references to it, but I suppose it could still exist somewhere. Perhaps I should create a 301 redirect for it.
    – jhawes
    Aug 24 '15 at 19:37
  • What happens when you attempt to access the subdomain in any web browser or even with the command-line tool CURL? Aug 24 '15 at 23:28

If screaming frog is finding the sub-domain, there must still be a link to it on one or more of your pages. That's how it discoverers pages, following links that are in the source code of the page.

So if the other 2 sub-domains aren't coming up it means they most likely aren't linked on the site. (although if they are blocked in robots.txt and you do not have 'ignore robots.txt' in the Screaming Frog settings, then they wont show up, even if they are linked)

To find out where the links are, click on the sub-domain in Screaming Frog, then in the bottom window click the 'Inlinks' tab.

This will give you a list of the pages that are linking to the sub-domain. Open these pages in your browser and if you cannot see the link, view the source code of the page and search for the sub-domain - it may not be visible on the front end.

I don't think a redirect is necessary here, I would just simple remove the link.

  • You got it. Thanks for your help - wish I saw the inlinks tab at the bottom earlier... still getting used to the software.
    – jhawes
    Aug 26 '15 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.