On my website I put noindex on my category pages but not the pages under it. Even though two weeks passed, the only indexed page is my home page which is not under any category.

When I look at yandex webmasters, under indexing>statistics>excluded pages I see both www.example.com/category and www.example.com/category/. I wonder if www.example.com/category/ (slash at the end) means "I am not indexing any pages under it" and www.example.com/category means I am not indexing the category.

  • I've found Yandex and Baidu crawlers and indexers work significantly differently than Google. I don't have experience with how they handle noindex, but given some funky behavior I have seen with robots.txt, I wouldn't be surprised. Aug 8 '16 at 11:55

pages are indexed if:

  • they haven't noindex,
  • they have incoming links (internal and/or external),
  • incoming links to them aren't nofollow.
  • bot has uninterrupted url path from its entry point to page, which should be indexed, like: entry url links to url1 (no nofollow) -> url1 links to url2 (no nofollow) -> url2 links to url should be indexed (no nofollow) -> url, which should be indexed (<meta name="robots" content="index" />)

Bot can't understand which is directory and which is not, due to there are many server side configuration to use trailing slash or not, if webmaster want.

Your category and category/ both will treat same, because when you open category page, it might be redirect to category or vice versa, if not then still your noindex tag will be display on that pages, hence yandex will not going to index it. They just look all the webpages weather you have set redirection properly or not, if they see noindex tag on those pages, they will not going to index it.

Your other pages like category/page.html will be index by yandex, because on that page there is no meta noindex tag.

Meta tags apply to page level only, it will not apply to whole directory subpages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.