Tagged Questions
2
votes
1answer
43 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
1answer
116 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
222 views
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site.
What's the symptom? Google + Bing crawl, but don't index any of the pages.
Let me be clear: I'm ...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
4
votes
1answer
130 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
1
vote
2answers
516 views
SEO: disallowing Google from indexing forms?
From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not?
I mean my forms hardly ever contains keyword/keyphrases.
So I'm wondering what's the point of letting ...
4
votes
1answer
332 views
Should I block search engines with robots.txt on my Facebook app?
There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
