Why don't SEO tools such as Screaming frog crawl some of my links?

Question

I have several internal links in my inner pages, for example Interlaken, Switzerland. I used several SEO tools including Screaming Frog to generate sitemap.

All SEO tools index links under the "Top hotels" section, but none of them index links under the "Destinations you might like" section.

I try to figure out the differences and fix the issue, but the links look similar to me. All links under both sections have:

• Title
• absolute internal URL
• Text content

The indexed list for the above inner page are:

/
/destinations/7/interlaken
/hotels/7/hotel-du-nord
/hotels/13/hotel-seiler-au-lac
/terms

While I expect to see

/destinations/8/heringsdorf-ahlbeck-bansin/8km-long-beach-in-usedom
/destinations/9/lauterbrunnen/village-of-72-waterfalls
/destinations/10/salzburg/the-land-of-music-museums-and-architecture

As well.

What should I fix to get the links under "Destinations you might like" section indexed?

Answer 1

The links under "Destinations you might like" appear to be all generated by JavaScript - they don't actually appear in the initial HTML source of the page.

This will prevent most "SEO tools" from crawling/seeing these links as they simply don't execute JavaScript and consequently don't see the final rendered page that the user sees.

However, Google may still be able to index these pages by "crawling your site" since Google does a good job of processing JavaScript and seeing the rendered page.

You don't necessarily need a sitemap for Google to index your site - if the pages are crawlable. To take that a step further, if an "SEO tool" is able to crawl your site then you certainly don't need a sitemap for Google's benefit, as Google probably does a better job of crawling your site than the "SEO tool"!

All SEO tools index links under the "Top hotels" section

The links under the "Top hotels" section appear to be present in the HTML source when the page is first sent to the client.

Further reading:
The Sitemap Paradox