2

Last day when i visited google webmaster tools, i found crawler errors for 42 not found pages. all the pages were having "luminous-solutions.net" at last in the URL. Like

http://article-stack.com/2010/08/page/luminous-solutions.net

I had never written any post which is having this word. For the safety, i had manually searched it on my site using internal and external search facility. I also had searched in source code as well.

I had searched in database, newly installed plugins, theme code.... But i found nothing with "luminous-solutions.net"

404 means, some text in contents are pointing to these links. But link doesn't exist. I had manually checked those pages as well.

May this fault occurred from google crawler itself or really some virus infected my site?

|improve this question
4

Don't worry about them.

There are many spammy sites that create links such as that. GoogleBot honours them and tries to follow the URI only to hit a 404. If you do a search for link:http://article-stack.com/2010/08/page/luminous-solutions.net in respective search engines you'll often find the culprit (otherwise they'll have been taken down already).

A search for just link:luminous-solutions.net reveals a set of typical spam.

If you definitely do not want crawlers to follow them then create a deny all by default robots.txt and only allow the URLs you know will result in correct pages being displayed (but that may be considered overkill).

|improve this answer
  • This does not explain how the Google bot found this link. – Ilian Iliev Oct 12 '10 at 12:30
  • That's assuming that the spammy sites actually honors robots.txt. Many don't care. – bpeterson76 Oct 12 '10 at 15:05
  • The spammy site may (and usually do not) not honour robots.txt and neither does it need to. I was referring to the spammy site HAVING that link (there have been loads of these and if you catch them soon enough you'll usually see syndicated content and lots of domains with the same weird paths on the end of their URLs) and then being subsequently crawled by a search engine (so the robots.txt is under their control). By you editing your robots.txt to be less permissive Google's crawler would never even attempt to index that page, so no 404 in Webmaster Tools. – Metalshark Oct 12 '10 at 17:56
2

A while ago I saw quite a few spammers taking advantage of poorly coded site searches. Some searches if they have nothing to return from their site will then show web results (IE Google search results). If they site doesn't keep their search results from being indexed a spammer can easily generate search results they know will return their site and build a few links to make sure the page gets indexed and voila they have links from your site to theirs.

I haven't seen this lately because Google doesn't like to show search results in their results and URLs from site search are pretty easy to pick out since they usually contain parameters like query= q= s= or search= so they do a good job of keeping them out of their index.

This is pure theory but along those lines it seems like it would be possible to exploit custom 404 pages (and more difficult for Google to detect). Many people create custom 404s to try and direct users to where they want to go, I've seen a bunch that appear to auto populate site search results based on URL parameters. I've also seen a bunch of custom 404s get indexed because they return a 200 status rather then true 404. It's logical if you combine the two you'd end up with a regular looking URL with a list of links to said spam site.

Regardless it looks as though your site is handling it properly.

|improve this answer
  • i dint get u clearly. did u mean to say that search on my site always must return result. Otherwise it may attract spammers? – Amit Kumar Gupta Oct 14 '10 at 2:44
  • I'm actually saying the opposite, thinking like a spammer, what is the easiest way to generate a page with links to your site? search "mysite.com" in Google, the problem is those links will never get indexed by Google. Fortunately for spammers, Google has provided the solution, through their APIs, free site search, & CSE their search results are displayed everywhere. If you want to make sure every search on your site returns a result you can do so using Google's results. The spammer the just has to generate a query that is guaranteed to include their site in the results & get it indexed. – Joshak Oct 14 '10 at 23:12
  • As I mentioned above I don't see this tactic too often any more as Google has gotten better at keeping site search results out of their index. – Joshak Oct 14 '10 at 23:14
0

Crawling your website with xenu does not show such links. Maybe the problem is in the submitted site map, can you send us a link to the sitemap?

|improve this answer
  • i missed to check sitemap. Still the path is article-stack.com/sitemap.xml as usual – Amit Kumar Gupta Oct 12 '10 at 10:33
  • Also nothing there, strange... Try to resubmit the site map and wait for the new indexing to be finished and check are there any crawl errors now. – Ilian Iliev Oct 12 '10 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.