1
vote
1answer
264 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
0
votes
1answer
100 views

News like pages get index, but not showing up in SERP

Well I have some crawlers to scrape news from different news agency websites and auto-publish it to my own website, the thing is my pages get index in Google, but even if I search the exact same ...
4
votes
1answer
275 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
1answer
141 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...