All Questions
6 questions
10
votes
1
answer
2k
views
How does "Noindex:" in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
- 99.4k
5
votes
2
answers
2k
views
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
A lot of websites hide their sitemap.xml file.
What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file.
So my question is ...
- 689
9
votes
1
answer
7k
views
How can I stop Google from indexing embedded iframes?
I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
- 245
5
votes
4
answers
8k
views
Does Google ignore robots.txt [duplicate]
I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
- 153
4
votes
2
answers
721
views
Controlling robot crawling concurrency using robots.txt and the undocumented setting "host-load"
I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
- 141
1
vote
1
answer
193
views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
- 251