0

I run a small business, for discussion let's say I have 4 pages:

  • Page A (Home)
  • Page B
  • Page C
  • Page D.

Important: NONE of these Pages have internal links to each other. Imagine just 4 tabs at the top of your screen. You get around the site by clicking on a tab.

I have a visitor who comes like clockwork once every 30 days or so. About 8 months now. All "Direct" traffic, and mainly (but not always) start on Page A (my Home page). The total visit time is about 25 seconds. But they have looked like this:

  • Month 1: Page A, then Page B
  • Month 2: Page A, then Page B, then Page C.
  • Month 3: Page B, then page D.
  • Month 4: Page A, then Page B.

I am told that Webcrawlers would tend to go only to Page A, and -- finding no internal links -- would stop there. And certainly would not likely start at Page B.

Is this the behavior of a Webcrawler? The regular timing of visits suggests this. Or is this a legitimate visitor?

Improve this question
8
  • 1
    Those tabs are internal links
    – John Conde ♦
    Commented Apr 25, 2022 at 20:58
  • Ahh, that makes sense. Thank you. But I see lots of other Webcrawlers on my site, with the usual suspect locations (Ashburn, Washington), but none of those go any further than my Page A (Home). The visit duration on those is 0:00. This is the only one that seems to dig deeper. Is that still common?
    – HappyFunBall
    Commented Apr 25, 2022 at 21:03
  • Try this site; when you map your site, take a look at the Tree Map. That's probably what the crawlers see. powermapper.com/products/mapper/maps/website-structure
    – Henry Visotski
    Commented Apr 25, 2022 at 21:30
  • Thank you, Henry. That's an interesting tool. So looking at "Page Stack" view, it's my Home page on top, and then another 14 pages straight across, all on the same "stack" or level. What I find odd is that this Webcrawler (alone) goes into the second level, and totally ignores 10 of 14 pages there. I haven't found any articles that describe this. But again -- I'm new.
    – HappyFunBall
    Commented Apr 25, 2022 at 21:38
  • Oh, and another thing I noticed -- this visitor is behind a VPN I think. GA reports the locations as being all over the place, and sometimes "[not set]". But it's definitely a unique configuration of operating system, screen resolution and browers size. So one specific visitor. This shifting location part is again different from the usual Ashburn/San Antonio (etc) culprits.
    – HappyFunBall
    Commented Apr 25, 2022 at 21:41

1 Answer 1

Reset to default
1

A crawler like google go from one page to another. Internal or external links help them to discover new urls. That's the historical google page rank patent.

So if a page receive external links, she can theoricaly be discovered without any internal link. (a page on a website without internal linking is obviously a really strange practice.)

An other way to help crawler to discover new pages not linked is to add them in sitemap.xml and upload the sitemap.xml in the search console (or other console, depending of the search engine)

Finaly Google is supposed to gain data from crawling with chrome and it may help. But obviously, if you want your page to be crawled (and to rank on search engine) add the into your internal linking schema: it the best way for crawler to find them.

Improve this answer
1
  • Thank you for the thorough reply, Patrick. Helpful.
    – HappyFunBall
    Commented Apr 28, 2022 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.