Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This might belong on another stack site but idk. Soooo.... A year ago I started two sites that I didn't want indexed by Google or any other search engine since both sites were being developed based on the seo results of other clients.

I was blocking both sites with the "no index,no archive" metatag technique and blocking all with robots.txt

Previously both of the sites would index fine. Recently I removed the metatags from both sites. On a clean index, I submitted new sitemaps to Google Webmaster Tools.

Within a couple of days I watched the crawl rate for both start to climb. One says 6 pages indexed but nothing is showing up when i search Google index (site:http://www.fakewebsite.com). The same is happening for the other.

I don't understand whats happening! I admit it could be a panic attack caused by impatiently waiting for spiders. But because it's happening on both sites the same it's scaring me away from ever using noindex metatags of any kind... ever again. And really I hope someone has experience with this.

Could someone please: 1. Give ideas for a problem and solution? 2. Or just tell me it's normal and I'm impatient?

share|improve this question

migrated from stackoverflow.com May 3 '11 at 9:11

This question came from our site for professional and enthusiast programmers.

    
probably belongs on Webmasters.se –  AShelly May 3 '11 at 5:25
    
How recently did you remove the meta tags? If you could paste your meta tags here (remove anything that would identify what your site is, if you wish to have privacy) and we'll be more than happy to help you. Also, if you choose an answer for your other questions, it will increase the likelihood that people will answer your questions. Welcome to the community! :) –  Melanie May 3 '11 at 6:30
    
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <link rel='alternate' type='application/rss+xml' title='ROR' href='ror.xml'/> <meta name='robots' content='all'/> <title></title> <meta name='description' content=''/> <meta name='keywords' content=''/> <meta name='geo.position' content=''/> <meta name='geo.region' content=''/> <meta name='geo.placename' content=''/> <meta name='copyright' content=''/> <meta name='author' content=''/> <meta name='designer' content=''/> <meta name='' content=''/> <meta name='google-site-verification' content=''/> –  Natrix May 3 '11 at 11:53
    
Sorry for crappy format. I tried to post in in the origianl question via edit but looks like it was stripped out. Thanks. –  Natrix May 3 '11 at 11:55
    
You have to use the code tool on your question since it's code. Your meta looks good, though. How long ago did you make the changes to your meta? –  Melanie May 3 '11 at 12:03
show 3 more comments

2 Answers 2

Oh man... I found it! In wm-tools under removal request! An old removal request (index page!) from a different/previous Google account! How did I miss that! Thank you all for your patients with me. I feel a fool now... But... I hope this saves a headache or two down the road for somebody! :)

share|improve this answer
add comment

It's normal and you're impatient. :-) The fact that it's being crawled is a good sign.

share|improve this answer
    
Thanks. I would assume so too. Just seems a bit odd that Webmaster tools says that there are pages indexed yet their not being included in search results. I'll be trying a little of that patients stuff and taking a look in a couple days. Hoping for the best. –  Natrix May 3 '11 at 11:51
    
Hi @paulmorriss and @Melanie, I just started another site two days ago and the shallow crawl indexed the site home page and became available in search results in under 48 hours. And did so with very minimal crawl rate. Yet the websites I mentioned before link being one of them, has an amazingly high (maybe human induced?) crawl rate, shows 28 pages indexed in web master tools but yet nothing in search results after a week. I don't know whether to be worried about reviewing terms of service or if the noindex tag was ok but noarchive caused side effects. :( –  Natrix May 5 '11 at 21:33
    
I think just because it's in the index doesn't mean they'll decide to display it. Maybe they are wary in case its a content farm or some other site they don't want to encourage. –  paulmorriss May 6 '11 at 7:57
    
Man that would be a disaster. The site is a legit service provider so should clear fine. –  Natrix May 6 '11 at 21:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.