Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Well I have some crawlers to scrape news from different news agency websites and auto-publish it to my own website, the thing is my pages get index in Google, but even if I search the exact same words, it's not showing up.

It only shows the source website, I'd like to know what can I do to make Google to show my pages on SERP too? should I link back to the source website? or is it basically not useful to crawl another website and publish it's content as your own?

share|improve this question
up vote 1 down vote accepted

It is not useful to crawl another website and publish its content as your own. Google refers to that practice as "scraping". Here is an article about it: Google Penalty: Why You Should Not Copy Content which is summarized:

Many people believe that setting up a website loaded with copied content is an easy way to make lots of money via AdSense and advertising. They are mistaken.


Google, the almighty search engine, heavily penalizes websites that scrape content. Other search engines like Bing, Yahoo, Baidu etc. also impose similar penalties on misbehaving websites. Such penalty will push your website down the search results and that will make it difficult (or almost impossible) for users to reach your website.

In your case, it appears that Google is crawling your site, but not indexing your content because it indexes the content elsewhere.

share|improve this answer
Thanks Stephen, but what would be the solution? I still want my users to access the news via my website, what's the best approach that doesn't harm my SEO – Sohail Aug 13 '13 at 11:27
There is no best solution. Google doesn't like duplicate content. What you're doing is exactly what they don't want to see from websites. – John Conde Aug 13 '13 at 11:31
You can use "content syndication" where you publish links to the original source, or hide the content from Googlebot, but Google still won't rank you for that content. – Stephen Ostermiller Aug 13 '13 at 12:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.