All Questions
15 questions
1
vote
0
answers
55
views
Do any search engines exist which disobey rel="nofollow" or "Disallow"? [closed]
Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
- 121
2
votes
1
answer
35
views
Should article preview pages be crawled and indexed by search engines?
I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content".
Since ...
- 217
1
vote
2
answers
555
views
How does robots.txt work with sites in subfolders?
I have a single web host with a number of other parked domains/sites in sub-directories, like this:
example.com is the primary site and root directory of the web hosting.
example.com/www.example2....
- 215
2
votes
0
answers
418
views
Set bot crawl delay 10 seconds EXCEPT Googlebot?
Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?
I like being indexed by all these bots but with Yandex and Baidu it is ...
- 131
3
votes
2
answers
652
views
Optional query string specification with "Disallow" in robots.txt
I'm creating a robots.txt file for my website which now has a lot of pages.
I'm disallowing all the pages that I do not require to crawl by so:
Disallow: /folder/file.aspx
I've quite a lot of ...
- 133
0
votes
0
answers
12
views
How to make sub-directories appear in Google Search Engine? [duplicate]
I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
- 101
2
votes
1
answer
2k
views
How can Iframes be optimized for search engines?
If my site has an Iframe. What is the best way to optimize this for search engines?
We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
4
votes
3
answers
8k
views
Hide a site entirely from search engines (Google, Bing etc.)
My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc.
However, the websites still need to be accessible for our customers, and ...
- 161
2
votes
2
answers
579
views
Can a website require search engine robots to support cookies?
Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
- 165
2
votes
2
answers
225
views
Search engines that index your site despite a disallow in robots.txt
Recently, WordPress changed their options from "Block search engines" to "Discourage search engines"
So that leaves me wondering, that rewording must be that some search engines disregard your ...
- 131
8
votes
5
answers
33k
views
What does "Disallow: /search" mean in robots.txt?
In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section.
User-agent: Mediapartners-Google
Disallow: /search
Allow: /
I know that Disallow will ...
- 2,933
1
vote
1
answer
222
views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from (...
- 141
3
votes
1
answer
3k
views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
- 405
2
votes
2
answers
248
views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
- 671
4
votes
1
answer
557
views
Should I block search engines with robots.txt on my Facebook app?
There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
- 199