All Questions
Tagged with web-crawlers robots.txt
24 questions
10
votes
1
answer
2k
views
How does "Noindex:" in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
- 99.4k
5
votes
1
answer
263
views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
- 355
5
votes
2
answers
2k
views
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
A lot of websites hide their sitemap.xml file.
What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file.
So my question is ...
- 689
4
votes
1
answer
674
views
Semalt ignores robots.txt, does their own form actually do what they promise?
Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess.
I just found out that they're also have a ...
- 1,600
34
votes
3
answers
10k
views
Is there a difference between an empty robots.txt and no robots.txt at all?
On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.
Do crawlers behave differently in ...
- 453
15
votes
7
answers
26k
views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
- 1,600
9
votes
1
answer
7k
views
How can I stop Google from indexing embedded iframes?
I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
- 245
7
votes
3
answers
38k
views
Do I really have to block MJ12Bot (as the prevailing visitor on my site)?
I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
- 972
6
votes
1
answer
1k
views
What is difference between robots.txt, sitemap, robots meta tag, robots header tag?
So I am trying to learn SEO and I am honestly confused and have following 8 questions.
Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt?
Is it ...
- 185
6
votes
2
answers
6k
views
Serve a different robots.txt file for every site hosted in the same directory
We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
- 381
5
votes
3
answers
2k
views
Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources
I have a robots.txt file:
User-agent:*
Disallow:/path/page
Disallow:/path/
Allow:/
The disallowed path is still getting crawled.
I have searched this problem and what they said, order of precedence ...
- 51
5
votes
1
answer
6k
views
Redirect Google crawler to different robots.txt via .htaccess
I have googled for the answer all day and still couldn't find an answer.
I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
- 61
5
votes
4
answers
8k
views
Does Google ignore robots.txt [duplicate]
I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
- 153
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
4
votes
2
answers
721
views
Controlling robot crawling concurrency using robots.txt and the undocumented setting "host-load"
I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
- 141
3
votes
1
answer
916
views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
3
votes
2
answers
5k
views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
morpheous
3
votes
1
answer
168
views
Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?
This question is a follow up to this one:
Should one hide RTL encoded URLs in robots.txt or not?
Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
user58733
3
votes
1
answer
344
views
Why are bots repeatedly crawling pages of my site that I never created?
In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
- 165
2
votes
2
answers
4k
views
What does crawl-delay: 1 mean if it is interpreted as is?
I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler.
I understood from this answer by Stephen Ostermiller that:
...
user58733
2
votes
2
answers
225
views
Search engines that index your site despite a disallow in robots.txt
Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"
So that leaves me wondering, that rewording must be that some search engines disregard your ...
- 131
1
vote
1
answer
193
views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
- 251
1
vote
2
answers
376
views
crawl-delay: x --------- has someone here managed to understand how the math works there?
From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
user58733
1
vote
3
answers
4k
views
Robot.txt disallow *?s=
Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ?
Disallow: *?s=
- 13