All Questions
Tagged with web-crawlers crawl-errors
19 questions
1
vote
1
answer
49
views
What is the Query to add in robots.txt for blocking URLs has some special characters?
I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below
example.com/...
0
votes
0
answers
44
views
In my Crawl Stats, 23% of requests are "Unauthorized" - is that bad?
Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good?
68% of my crawl requests return 200 (OK) which is good of course.
Is this ...
- 425
2
votes
1
answer
528
views
Remove large amounts of pages from Bing Index Explorer
I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
- 153
2
votes
0
answers
190
views
When setting up a campaign, Facebook is crawling my website using an Iphone user agent
Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers.
The first one I can identify as a robot and serve the SSR (HTML) version to it.
The user agent is:
...
- 988
3
votes
2
answers
520
views
Google Search Console: 404 errors on existing pages
There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
- 173
5
votes
1
answer
194
views
Can Googlebot discover URLs that are not in an anchor?
In my Search Console I had a sudden increase in 404's.
They were linked from a webpage that listed files from the site in a table but none of these URLs were nested inside an <a> tag.
This was ...
- 1,187
4
votes
2
answers
146
views
Increase in 5XX errors and 404 errors on GWT, are we under attack?
A lot of URLs are suddenly giving us 404s and 500s.
On closer inspection, these URLs do not exist on our website and are being linked from random sources.
As more URLs were inspected on GWT, a lot ...
- 689
1
vote
1
answer
673
views
Google Bot Crawler Issue with Codeigniter URLS
I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL.
http://www.example.com/segment1/segment2&sa=U&ved=...
- 111
4
votes
1
answer
2k
views
Website not fetched by Bingbot: Unexpected Error
My website has been online for a couple years. It's on the first result for Google but it doesn't seem to be indexed by Bing. I tried to fetch it with bing in their webmaster tools and it always ...
- 43
2
votes
1
answer
2k
views
How to Keep "Good" Bots That Ignore Robots & Nofollow From Getting Into Honeypots?
We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
- 3,339
1
vote
2
answers
95
views
What to do about crawler errors after removing non-essential parts of my website?
So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors.
I am ...
1
vote
1
answer
650
views
Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors
Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code.
How can I go about diagnosing this issue?
The website is eCommerce business using the Magento platform....
- 381
1
vote
1
answer
146
views
Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?
I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG
several times a day from the Bingbot. This started a couple of days ago.
Anybody know why this is?
- 543
0
votes
1
answer
173
views
Spiders crawling broken links
Tl:Dr
Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin.
The Problem
It seems at least once a week we're getting a big burst ...
- 101
1
vote
3
answers
13k
views
Google Webmaster Tools - Crawl errors with 500 status code
I will try to explain the situation:
In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs.
After a while I changed something on my website, upgrading my pages ...
- 141
3
votes
2
answers
146
views
Yandex frequently replaces page names with ampersands
The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is:
/mypage.aspx?param=value
then it ...
- 543
0
votes
3
answers
251
views
Free standalone crawlers [closed]
Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors?
Something that shows me all 404's, errors etc
- 512
2
votes
2
answers
261
views
Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?
My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
- 745
3
votes
2
answers
152
views
How to set crawl rate in Google Webmaster Tools for seven days or a month?
My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
- 745