Hot answers tagged

7

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like: X-Robots-Tag: noindex After implementing the HTTP header, remove the disallow from your robots.txt. ...


4

You are sending Google two conflicting commands: you are telling them to NOINDEX some pages but then you are also preventing them from reading the NOINDEX header by blocking them from crawling these pages (in the robots.txt) so they can't see the NOINDEX command. You should only use the NOINDEX tag and let their bots crawl your website in order to see it; ...


3

Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt. However, that doesn't necessarily prevent the /search pages from appearing as link only results in the SERPS (if other pages link to them) - but this is unlikely. And they are very unlikely to rank higher than other search results anyway. 2:...


3

The most accurate way is to add your site to Google Search Console and check the Coverage report. If pages are indexed they will be in the green Valid Pages section. If they're not they'll be in the grey Excluded section, in a sub-report called "Excluded by 'noindex' tag". These reports take time to update if you've just set up Search Console, so if you ...


2

You'd be correct to use dofollow on inlinks pointing to the pages you want to exclude, but you won't be able to use dofollow on the noindexed pages themselves, if that's what you were asking. Noindex on a page equates to noindex,nofollow long-term. Noindexing isn't the perfect solution to fixing thin/duplicate content though. For one thing, the criteria ...


2

As A Biron mentioned, after you've verified that the noindex directive is properly written, use Search Console's coverage tool to look at the number of "valid" indexed pages. If there are still results, either wait for Google to re-crawl your site on its own, or use Search Console's temporary removals tool to pwn your site's entire search presence at once. ...


1

This issue is fixed without doing anything further. If you are in a similar situation, just be patient. Apparently it takes a month or two for Google to validate.


1

Search engines don't index the same content on two different URLs regardless of whether or not you use canonical tags. Crawlers use the shingle algorithm to compare pages and can see when any two pages on the internet have substantially the same content. When search engines encounter duplicate pages they usually choose to index one of them and ignore the ...


1

The meaning of a canonical is to tell google which one is the original. Google will prefer to index the original one, and not the duplicate. Keep in mind: insert <link rel="canonical" href="example.com/a"/> also on the original page example.com/a (self reference) do not mix robots="noindex/follow" with the rel="...


1

This is totally fine from an SEO standpoint. You're telling the crawlers not to index these "No Index" pages, so that's just what they'll do. Ideally you would configure the server to return a "403 Forbidden" status code so that non-search crawlers get the idea too, but in practice it probably does not really matter.


1

A page that returns a 410 status code will be dropped from Google's index very shortly after Googlebot re-crawls it, even "a little bit faster than a 404" according to Mueller from Google. However, to me it seems like you are using the 410 Gone status code incorrectly. If the pages "still have full content", then they aren't really gone! ...


1

I don't think you're missing anything, unless your jobs pages are undiscoverable (no internal links). Sounds to me like you probably just need to wait. How long has it been? Google can take days to weeks for search results to "settle". Otherwise, Google may have decided that your jobs are not important enough to show. From the Google page you linked: ...


1

Google suggests "pretty URLs" because making them more readable by humans is better for user experience. If you're not indexing those pages, then users will not see them in search results. So you don't need to change their URL structure since you won't be "eligible" for the benefit anyway.


1

If you want to use htaccess, you can use the http header X-Robots-Tag: noindex to accomplish what you need for the specific files you need, or with a pattern. You can also provide other instructions, such as noarchive, nofollow, or even be specific for a particular crawler, for example Googlebot: X-Robots-Tag: googlebot: nofollow X-Robots-Tag: otherbot: ...


1

By default Screaming Frog crawls images, CSS and JS, but there are configuration items to turn off crawling them. See SEO Spider Configuration | Screaming Frog The settings are located in the "Spider Crawl" tab of the configuration.


1

Create a container called $root In that container upload your robots.txt file blocking the folders or files you want Change the file access level to Public Go to http://yourstorage.blob.core.windows.net/robots.txt to check if everything is ok Optional: Register your storage domain in Google Search Console to check all the statistics of indexed pages (Use ...


1

Both. If you add NOINDEX to some articles, they won't help you with SEO because they won't be seen by search engines. If most of your articles are not seen by search engines, then your website is missing opportunities to rank and bring organic traffic. If your question is if adding NOINDEX to some articles will hurt the rankings of the other articles or ...


1

Nowadays, don't worry about it. Although we don't set the x-robot http header noindex for static file css, js, Googlebot know what file should be indexed or not. Except image, because the images can grab organic traffic for our site. Btw, So many people still don't understand what is robots.txt file and noindex tag. Robots text file what should to crawl and ...


Only top voted, non community-wiki answers of a minimum length are eligible