I'm getting a lot of URL's in screaming frog where the URLs are no-follow no-index but, they are still getting crawled in screaming frog. Is it fine or should I change anything in this?


What you are describing is normal. Neither noindex nor nofollow should prevent crawlers such as Screaming Frog or Googlebot from crawling those pages.

When Screaming Frog encounters a nofollow page, it will not add the links on that page to its crawl queue. nofollow doesn't prevent that page itself from being crawled. Screaming Frog supports a configuration option called "Follow internal or external ‘nofollow’" that would cause it to crawl all the links on nofollow pages. However, that configuration option is not enabled by default.

Screaming Frog is does not create a search index, so noindex would not normally apply to it. Screaming frog does support a configuration option called "respect noindex" that causes Screaming Frog not to include noindex URLs in its reports. Note that that it wouldn't prevent Screaming From from crawling the URLs, it just won't report on them. That would be similar to the way that Googlebot can crawl the URLs, just not include them in its search index. You can enable that configuration if you desire.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.