All Questions
Tagged with web-crawlers crawl-errors
2 questions with no upvoted or accepted answers
2
votes
0
answers
190
views
When setting up a campaign, Facebook is crawling my website using an Iphone user agent
Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers.
The first one I can identify as a robot and serve the SSR (HTML) version to it.
The user agent is:
...
- 988
0
votes
0
answers
44
views
In my Crawl Stats, 23% of requests are "Unauthorized" - is that bad?
Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good?
68% of my crawl requests return 200 (OK) which is good of course.
Is this ...
- 425