Questions tagged [screaming-frog]
The screaming-frog tag has no usage guidance.
23
questions
2
votes
2
answers
50
views
Regex / Regular Expression for finding duplicate URL directories in Screaming Frog SEO Spider
While analysing a website with the Screaming Frog SEO Sider I've seen a problem of some pages containing duplicate URL directories.
To fix this problem I need to know which URLs this affects.
So I ...
- 21
0
votes
0
answers
35
views
Ecommerce site Screaming Frog (and other tools) crawls return internal cart urls with parameters
Example url: somedomaindotcom/patches-metal-rubber-1-can/?add-to-cart=4183
When I crawl my WooCommerce sites with tools like Screaming Frog/AHREFS/SEMRush they return these one-time usage urls that ...
1
vote
1
answer
1k
views
Screaming Frog reports internal links are unsafe cross-origin
I've just run a Screaming Frog spider against a client's website and it has returned 370 links that are Unsafe Cross-Origin Links. However when I have investigated the links they don't match the ...
- 287
4
votes
1
answer
212
views
Can Screaming Frog check for invalid SSL certificates?
I've been using Screaming Frog to test the home pages of a few hundred sites I manage, to see what status codes they return. I noticed in my browser that one of those sites has an invalid (expired) ...
- 361
0
votes
0
answers
104
views
How Can I Use Screaming Frog to Find Specific Text Using RegEx or XPath?
I'm using Screaming Frog to crawl my site. I created a custom search to look for "myterm". When i crawl the site, the reports shows how many occurrences of "myterm" are shown on ...
- 1
1
vote
0
answers
29
views
Screaming Frog reporting duplication between two canonicalized pages
I have Page-A with canonical pointing towards page B. Page B should be considered the original and should be ranked in SERPS. But the Screaming Frog crawler is showing that page A and B are ...
- 11
3
votes
1
answer
66
views
Why don't SEO tools such as Screaming frog crawl some of my links?
I have several internal links in my inner pages, for example Interlaken, Switzerland. I used several SEO tools including Screaming Frog to generate sitemap.
All SEO tools index links under the "...
- 133
1
vote
1
answer
435
views
Screaming frog crawling, images, JS and CSS files as well
When I plug in my website URL in the free version of Screaming frog, it also lists the images, CSS and JS files. Is it suppose to be like that?
- 23
1
vote
1
answer
120
views
Is it normal that URL's are getting crawled in Screaming frog Still they have a no-follow and no-index tag on them
I'm getting a lot of URL's in screaming frog where the URLs are no-follow no-index but, they are still getting crawled in screaming frog. Is it fine or should I change anything in this?
- 470
1
vote
1
answer
138
views
Why would Google Search Console report many times more indexed URLs than a Screaming Frog crawl finds?
I've just crawled my website in Screaming Frog and I have around 6K Internal URLs and 4K external URLs. Then I looked in GSC and saw that Google indexed 50K pages.
How is that possible?
2
votes
1
answer
169
views
How can find I the place where the Screaming Frog found a link on my site?
I use screaming frog for my site. For some URLs it finds I don't know where it found those links.
This is an example image of my screaming frog:
- 123
3
votes
1
answer
436
views
Screaming Frog reports "Mailto" link create 404 errors
A site I've built gets too many 404 errors with Screaming Frog
All the 404 originate from the "mailto:" links on the site - for some reason the scan identifies non-exist pages which looks like this:
...
- 31
1
vote
2
answers
66
views
Discrepancy in ScreamingFrog Depth Stats and actual list of URLs
I have crawled my website with Screaming Frog, it has roughly 200 pages, most of which are in example.com/category/product, however, the tool shows me that these are actually on 4-5 clicks away from ...
- 199
1
vote
1
answer
246
views
Screaming Frog hreflang missing confirmation links for rel="alternative"
Screaming frog tells me that I have missing confirmation links in my hreflang, but I cannot seem to identify the problem. Here is a screenshot of the code on our Korean page:
Here is a screenshot of ...
- 13
2
votes
1
answer
67
views
Can you run screamingfrog on Google Cloud?
I've seen articles online on how to achieve this.
They were written in 2016. Not too sure how relevant it is now? Has anyone tried?
- 21
2
votes
0
answers
262
views
Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages
I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
- 369
1
vote
4
answers
723
views
Big discrepancy in number of known pages in Google Search Console and ScreamingFrog
In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...
- 199
1
vote
0
answers
95
views
Screaming Frog marking duplicate on noindex pages
Screaming Frog is detecting as duplicate all login pages of my site despite I have the directive robots-noindex.
Do I have a problem on my site, or is just a false positive from Screaming Frog?
It ...
- 358
2
votes
1
answer
241
views
Screaming Frog removes custom request headers
We are working on an Angular SPA that uses token-based authentification (our client app gets a token from the server and saves it to localstorage). Then in each request to our API the client adds a ...
3
votes
1
answer
4k
views
Screaming Frog reports missing confirmation link when the hreflang x-default element is the same in all pages
I am confused about hreflang="x-default".
I have a site in 2 languages. In each page I have a link to the page itself and a link to the same page in the other language. If I understand correctly, I ...
- 819
3
votes
1
answer
1k
views
Checking site for meta refresh redirects
How do you check or crawl site (couple of urls) to explore existing meta refresh redirects?
Screaming Frog doesn't handle them - it indicates a page with meta refresh as 200 (it is partly true), even ...
- 10.1k
1
vote
1
answer
1k
views
Image sitemap. Ok to use the ALT text as <image:title>?
Screaming Frog uses the alt text of an image when i create a sitemap of a crawled site and places it in the image sitemap as .
Is this correct? Or should the actual image title be used as in the ...
- 1,177
0
votes
1
answer
266
views
Screaming Frog displaying deleted subdomain
In Screaming Frog, when I do a basic crawl of one of my website's in the 'Internal' tab it shows a sub-domain at the very top. This sub-domain was one I had previously created, but I removed it quite ...
- 113