Questions tagged [web-crawlers]
A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
905
questions
5
votes
3
answers
1k
views
Will the content of a PDF on our website affect SEO?
I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings?
We want to put some PDFs on our website but don't want to see our results take a ...
- 461
5
votes
1
answer
1k
views
Should we allow HEAD requests?
Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from ...
- 181
5
votes
5
answers
4k
views
Which tools can help limit maximum page view per IP to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
- 199
5
votes
1
answer
2k
views
Can robots.txt redirect to a different domain?
I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer.
example: https://www.mydomain.example/...
- 51
5
votes
1
answer
206
views
Does Google return to visit a page with 404 Error?
If a page indexed in Google shows a 404 Error, does Google return to visit it again?
In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
- 61
5
votes
1
answer
1k
views
What is the default behavior of robots if there is no robots meta tag?
Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in:
<meta name="robots" ...
- 153
5
votes
2
answers
2k
views
Google reports 404 errors on URLs that haven't existed for six months and are no longer linked to on my site
My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found).
But when I look at the source pages that are accused of ...
- 195
5
votes
1
answer
3k
views
Is there any reason to allow Yahoo! Slurp to crawl my site?
I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
- 307
5
votes
2
answers
2k
views
How is crawler seeing unlinked directories / files?
I'm running a crawler on my website to test for broken links and such.
It starts by using a URL like www.domain.com
One curious thing is that it is showing directories with no internal links. For ...
- 171
5
votes
2
answers
1k
views
Does Google index hidden pages based on direct visits on Chrome?
Suppose I have a "hidden" page on a website, but I don't explicitly tell bots to ignore it (no noindex tag, not disallowed in robots.txt, etc). It's not linked to anywhere, doesn't appear on the ...
- 153
5
votes
1
answer
4k
views
Why am I getting bot hits from compute-1.amazonaws.com?
I have a WordPress website with AWS on use, namely the Cloudfront service, to serve CSS, images and JS from the cloud.
Lately, I noticed a lot of hits from bots coming from :
IP: 54.236.71.87 ...
- 153
5
votes
2
answers
201
views
Extremely large spike in traffic on the 1st - 4th of every month from mobile browsers
I've noticed that on the 1st - 4th of the recent months (since January), several sites I maintain are getting thousands of requests from mobile browsers, whereas throughout the rest of the month, the ...
- 191
5
votes
1
answer
239
views
Consequences of blocking Semrush and other bots?
For typical websites, are there any disadvantages to my clients to blocking spiders like Semrush, Maxmind and the plethora of other "non search-engine" bots?
Wouldn't blocking these ...
- 7,939
5
votes
2
answers
7k
views
Can Angular project be indexed in Google without Angular Universal?
I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
- 151
5
votes
2
answers
773
views
Mysterious visits to privileged URLs from anonymous user/bot
I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs:
54.209.60.63 - - [03/Nov/2015:19:09:...
- 153
5
votes
1
answer
105
views
Is it expected behaviour that Google also crawls #! urls?
I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...
- 151
5
votes
2
answers
4k
views
How to crawl a webPage with dynamic content added by javascript
I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
- 161
5
votes
2
answers
751
views
What is the notified-CompliancePage page?
I sometimes have bots (I assume) hitting pages such as:
/notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw==
/verify-CompliancePage?...
- 543
5
votes
1
answer
777
views
Am I covering the major search engine spiders in my anti-crawl protection white list?
I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my ...
- 411
5
votes
1
answer
78
views
Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?
I consider preventing indexation of all of MediaWiki 1.33.0's Special pages;
(In Hebrew, "מיוחד" means "special"):
Disallow: /מיוחד:*
Disallow: /index.php?title=מיוחד:*
Disallow: /index.php%3Ftitle%...
user58733
5
votes
1
answer
1k
views
Dynamic content using .append() or .innerHTML() and SEO
I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO.
On my webpage I have an empty ...
- 153
5
votes
1
answer
1k
views
Block Yandex crawler
Our site has been behaving very strangely for the last few days, lots of time outs etc. Finally think I found the cause, the Yandex bot is crawling around 10,000 pages an hour! I need to stop it ...
- 2,390
5
votes
2
answers
86
views
Should we still care about ajax-crawling specification?
It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site.
Are they still valid and necessary to be followed?
- 159
5
votes
1
answer
343
views
What is User-Agent "AF_ID=<xxxxxx>"
We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like AF_ID=<xxxxxx>.
I've looked around for ...
- 121
5
votes
1
answer
408
views
Does Google crawl and index AJAX on HTTPS sites?
We made transitioned out website to HTTPS. We also now use nearly AJAX to load almost all the content. Therefore all URLs of existing pages have changed. We used 301 redirects as recommended and we ...
5
votes
1
answer
299
views
AdSense crawler access fails because the bot adds a slash to the login URL
I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler.
In this site, ...
- 151
5
votes
4
answers
356
views
Total Indexed 0, 99 not selected in webmaster tools
I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
- 263
5
votes
2
answers
4k
views
How To Track Down and Stop Rogue Bots?
Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:
Unknown robot (identified by 'bot*') consumed 164 GB this month.
By comparison, Googlebot ...
- 6,007
4
votes
2
answers
328
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
4
votes
3
answers
19k
views
Allow all robots robots.txt
In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
Yasmina Saraya
4
votes
5
answers
3k
views
Google Bots Crawling my Site daily is this good or a bad thing?
Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
Ron Nesler
4
votes
3
answers
354
views
Do i need to have a separate page for each article on my website?
I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google?
Is is ...
Arihant
4
votes
1
answer
545
views
How do I "noindex" a text (.txt) file?
I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as ...
- 573
4
votes
2
answers
678
views
Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?
here is my htaccess file......
RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR]
RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$
RewriteRule ^patients/...
- 53
4
votes
3
answers
177
views
Google shows the wrong title for my website - robots.txt issue
Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows
this
in translation from Romanian it says:
"pages_rss_title"
I noticed the ...
4
votes
3
answers
3k
views
Does Google index images declared using 'srcset'?
As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
- 307
4
votes
3
answers
8k
views
Hide a site entirely from search engines (Google, Bing etc.)
My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc.
However, the websites still need to be accessible for our customers, and ...
- 161
4
votes
2
answers
11k
views
Is my whole site is blocked by this Disallow: /
This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site.
User-agent: Baiduspider
Disallow: /
User-agent: *
Disallow: /...
- 41
4
votes
2
answers
1k
views
Why doesn't this web crawler obey requests to stop crawling my pages?
I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers:
X-Robots-Tag: noindex, nofollow
But once in a while I see this in the page logs:
...
- 163
4
votes
4
answers
5k
views
Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]
I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
- 311
4
votes
1
answer
3k
views
How to make sure an individual page from a website is not indexed by search engines?
I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed.
Can it be reliably ...
- 463
4
votes
2
answers
6k
views
Sosospider: what does it actually want?
On one website I have, looking at the logs, I find lots and lots of "Sosospider" hits. Is this the same for everyone, or just me? Now, I have never once been sent any traffic from anything which looks ...
delete
4
votes
1
answer
176
views
Weird bot requests in logs
I restored a website on a new server after it was down (mistakenly deleted) for a while, and it immediately got bombarded with requests to non-existing pages like /tissue-salts-for-weight-loss.html ...
- 43
4
votes
4
answers
565
views
My personal website was cloned in its entirety, is this a security concern?
I was googling my (real) name yesterday (which is the name of my site) on a lark and discovered a website that had copied my personal website and its content. Like a carbon copy including links to ...
- 43
4
votes
3
answers
1k
views
How does Googlebot find a new page on a website?
I was wondering how Google identifies a new article or piece of content on a website.
What is the method it uses to do so for all websites?
- 51
4
votes
1
answer
659
views
Semalt ignores robots.txt, does their own form actually do what they promise?
Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess.
I just found out that they're also have a ...
- 1,570
4
votes
2
answers
164
views
Is there any way I can defend against automated file download bots?
My company offers a very specifically tailored Android application to supplement our other software. Since users must have an account in our other software in order to use the Android application, is ...
- 141
4
votes
1
answer
967
views
Link rel="nofollow" vs robots.txt disallow for "/search" URLs?
There's a /search path on my web app that I don't want robots to crawl or index.
What would be the best approach for this scenario?
1: Block them from accessing by adding a Disallow for /search on ...
- 978
4
votes
3
answers
309
views
How to prevent a competitor from cloning my website
I run a website that provides meta-data from specialized electronics (data like weight, dimensions, applications, voltage, etc). It also compare products using all this data so user can see the ...
- 151
4
votes
1
answer
806
views
How does changing an HTTP referrer header help circumvent crawler blocking
I have been researching different ways a web crawler might be blacklisted/blocked from a web-server and how to potentially circumvent that. One of those ways is to change the referrer header on the ...
- 143