2

I am developing a React app.

I’ve had poor indexing coverage until now (only the home page was indexed).

I recently implemented server side rendering (SSR) and indexing coverage appears to be significantly better.

That being said, feels like I am playing the SEO game blind. Is there a way to simulate googlebot to see what would be indexed? Would love to see the recursive paths that googlebot sees.

I know about Google Search Console, but I can only inspect one URL at a time.

Improve this question
3
  • Try Sitebulb.com for your paths. I don't know how it works on Apps, but it will give you a tree map of your links.
    – Trebor
    Commented Mar 27, 2020 at 14:11
  • Thanks for the feedback @Trebor. Is there a way of achieving this withing the Google ecosystem to be 100% sure of how googlebot would handle the links?
    – sunknudsen
    Commented Mar 27, 2020 at 15:03
  • I'm not aware of anything like you're looking for. There is the "Links" section, but it is still pretty manual, i.e., one page at a time to see who is linking to it.
    – Trebor
    Commented Mar 27, 2020 at 15:36

1 Answer 1

Reset to default
1

If you are using server side rendering (which doesn't rely on JavaScript), you should be able to use any crawler. The most popular are probably:

  • wget - free open source, command line based, use the recursive (-r) option
  • Screaming Frog - requires payment after 500 pages, Windows GUI

Of course there are hundreds of other options if you search for "web crawler" or "SEO crawler".

If you are sniffing user agent on your site to see if you should use server side rendering, you'll need to make sure you use server side rendering for your chosen crawler in addition to Googlebot.

Alternately, most crawlers allow you to set the User-Agent they report. For example with wget you would use --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Improve this answer
3
  • Thanks for the feedback Stephen. I was hoping Google and/or Bing had put together a tool to help with isomorphic apps/websites. I know about wget... I was curious to learn more about how search engines crawl recurvively. What is consensus these days? Have crawlers adjusted to index JavaScript websites more effectively? I know Google has in theory, but I've had mixed results with non-SSR React projects.
    – sunknudsen
    Commented Mar 28, 2020 at 17:44
  • 1
    Google bot now executes JavaScript that runs when the page loads. If that JavaScript produces links and text googlebot will treat that content pretty much the same as if it had been written into the HTML of the page. In that respect, googlebot is more advanced than wget and other crawlers. If you were using server-side rendering, you shouldn't need to rely on executing JavaScript.
    – Stephen Ostermiller ♦
    Commented Mar 28, 2020 at 18:12
  • 1
    Google bot doesn't execute JavaScript triggered by user actions. Google bot doesn't try to scroll the page or click on anything. If you have page navigation triggered by something other than links, Googlebot won't be able to find it.
    – Stephen Ostermiller ♦
    Commented Mar 28, 2020 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.