4

So to start with, I have Google crawl my (WordPress) website with the search console, now I can see my pages in the search engine. But it shows every page separately. Like my /contact, /services and /about. I really want only to show my homepage. I searched for plugins like all-in-one SEO then I stepped over to Yoast for the automatic sitemap and robots.txt. I also entered disallow for those pages from being crawled in my robots.txt. I have set 'Noindex' to all those pages. That also doesn't work.

I have selected private for my pages to see if this changes anything... doesn't work.

Today I have discovered that I also have unwanted URL's that come from nowhere, like: /category/uncategorized, /author/admin and /2016/03...

And for some reason I have for every page that Google shows me the same meta description.

I really want that Google doesn't show these pages, it's really annoying. What can I do?

  • 1
    So you have a page available from /about but you don’t want Google to give this as a result, ever? So people should never be able to find your About page by searching in Google (and other search engines?)? -- How long did you wait after changing robots.txt, and how long after adding noindex? – unor Apr 13 '16 at 10:28
  • So it is really better to let it be like that then? If I search a company, than I don't see the other pages. I see the main page only. So If you go to my main page then you can always click on about, contact,... I've waited a couple of days (3-4) and than rechecked but nothing really happend after changing robots.txt or noindex. Also if I search on google it changes often, sometime there are 3 urls to see, sometime I can see 5 urls... It's really weird. And than those unwanted url's, I don't want that, everyone can see that.. – Quest_ Apr 13 '16 at 10:47
  • 1
    It is always advisable to allow the search engines to crawl all your main pages. e.g. home, about, services, contact etc. However you can disallow crawling of category pages, e.g. /category/uncategorized or author pages, e.g. /author/admin etc. – zipkundan Apr 13 '16 at 15:38
  • ... I have them disallowed in my robots.txt... but it doesn't change anything, Google still show them, also on those pages contact, services,... I use Noindex from Yoast. Still no result. So what can I do else? Or maybe I do something wrong... – Quest_ Apr 13 '16 at 15:56
  • 1
    Keep in mind that search engines do not turn on a dime. They are not real-time. In fact, search engines are notoriously slow. It can take months for a change you make to show up. Depending upon many factors, it can take less time. Minutes even. Any new site will not be updated quickly. – closetnoc Apr 13 '16 at 16:11
2

I have them disallowed in my robots.txt... but it doesn't change anything, Google still show them, also on those pages contact, services,... I use Noindex from Yoast. Still no result. So what can I do else? Or maybe I do something wrong

If you specifically don't want these pages indexed then you should use the noindex robots meta tag in the head section of the page (or X-Robots-Tag HTTP response header).

DO NOT disallow them in robots.txt. A disallow rule in robots.txt does not necessarily prevent indexing, it prevents that page from being crawled by Googlebot. If the page is linked to it can still appear in Google's index. But in this case, it already is indexed.

It will take some time for Google to update its index before these pages are dropped from the SERPs. You can speed up the process by using the URL removal tool in Google Search Console. However, that does seem a bit drastic for these presumably quite innocent pages.

Although why you would want to do this is debatable. (Particularly for pages like /services - isn't that a primary, content rich page?) Isn't it dependent on what you are searching for that determines whether you see these pages? (Or are you doing a site: search?) The fact that you don't see these pages for companies you search for may simply be that the other pages rank higher for the phrase you are searching for. If I search for about google, I find Google's "About" page, but otherwise it's a page that doesn't ordinarily get returned in the SERPs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.