3

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding.

So far, I've tried DeepCrawl and Screaming Frog, but I'm unable to extract the data I need. It goes as far as the Home Page and then the crawl stops.

Is there anything I would need to set up on the backend or add to the markup itself so it's crawlable?

Improve this question
2
  • What do you plan to do with the list of URLs?
    – Mike -- No longer here
    May 27, 2016 at 3:34
  • @Mike Part of a UX content audit and initial stages of rebranding the whole thing
    – Ellesa
    May 27, 2016 at 3:37

4 Answers 4

Reset to default
1

You're attempting to SEO a javascript rendered site. The issue is that crawlers don't tend to render the full site, as they're skipping the majority of javascript. In this case, it's indexed static text on the front page.

You can attempt to add pages individually to a sitemap and request a re-index in Search Console. Google has gotten better at this in recent times but honestly it's still not quite there. They still don't handle javascript sites very well, it's very resource intensive. As mentioned in another post, you can use software to make it crawlable. Another example is prerender.io. What you need is to re-present the site to the crawler as a static site.

Improve this answer
0

Not that this is supposed to be the place to recommend software, but I will advise using Xenu Link sleuth found at: http://home.snafu.de/tilman/xenulink.html

It will crawl every single page on your site connected to the URL you specify and there are no restrictions to how large the list can go, plus you can save the list as a tab separated file.

The only way markup can control crawl-ability is if the robots meta tag was used and that would only control search engine's ability to crawl your website.

Improve this answer
2
  • 1
    Does it work with AJAX rendered sites such as those built with AngularJS?
    – Stephen Ostermiller ♦
    Jun 9, 2016 at 13:15
  • Here I thought he wanted to crawl a website where part of it was made with AngularJS and where it contained static links. It's not a very browser-friendly site if all links were only active if javascript is enabled.
    – Mike -- No longer here
    Jun 9, 2016 at 23:33
0

Screaming Frog now has a 'rendering' configuration for crawling JavaScript frameworks such as AngularJS - https://www.screamingfrog.co.uk/seo-spider-6-0/.

Improve this answer
1
  • 3
    Your user picture leads me to believe that you have some affiliation with Screaming Frog. Your response is on-topic here but your answer needs to be improved: 1. Any affiliation must be disclosed. 2. Link only answers are not high quality. A couple paragraphs describing this feature and how to use it, would make this answer much better. 3. We don't appreciate using this site only to promote your product. Please refrain from promoting your product further without becoming an active member in other areas as well.
    – Stephen Ostermiller ♦
    Jul 29, 2016 at 18:56
0

We can crawl Angular that has the _escaped_fragment_ solution implemented - but not without I'm afraid.

There's a cracking article by a client of ours here where he explains how he indexed a site built with Angular - you might find this interesting!

Crawling JS is on the product roadmap; watch this space...

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.