It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you suggest.
The other way is to simply issue 301 Permanent Redirect so that all traffic uses the official URLs. This will also allow you to enforce the use of HTTPS ...
If you are migrating domain names, do not remove your old domain's content from the Google index or you will see your rankings drop. Leave it be in Google's index, and implement proper 301 redirects. Also, use Google Search Console's Change of Address tool as potatomodem mentioned in his answer.
"Content" does not "duplicate" on the new ...
The XML sitemap is usually the best method to pro-actively notify search engines of anything, not just 404 pages. But I fail to understand why you would want to do that? Having 404 pages in Google's index isn't necessarily a bad thing, unless of course these 404 pages are still getting a lot of search hits, in which case you may want to 301 redirect them to ...
those pages/URLs should NOT be indexed
Well, they should be indexed, but not at the other hostnames.
You should certainly implement OPTION#1 and configure a rel="canonical" element. This should also protect you against non-canonical query strings (intentional or malicious).
AND, ideally, you would 301 redirect any requests for non-canonical ...
Obviously, the first thing that comes to my mind is: do you really need to reuse the old high rank URL for a different page? If you can avoid it, it seems that you already know what to do.
But one alternative I've thought of would be:
Re-launch your website and hold on for a while before reusing the old URL.
Set the 301 redirect (maybe try to force a re-...
Interesting question, because we are indeed seeing more Github results seeping into SERP's.
While Github and Gitlab aren't exactly SEO-focused (no amazing SEO tool suite), as they were not originally meant for websites that rank highly, if you write titles and copy that are well-phrased and answer the searchers' intent, you will rank.
For instance, try ...
There is no difference between follow, index and index, follow. These would both permit robots to index content and follow links*1. However, this is the default action for all bots, so the element is entirely superfluous.
(*1 But not because it contains the keywords index and follow, but because it does not contain the keywords noindex and/or nofollow.)
Sitemaps are only valid for pages that appear below them in the site hierarchy.:
A sitemap can be posted anywhere on your site, but a sitemap affects only descendants of the parent directory. Therefore, a sitemap posted at the site root can affect all files on the site, which is where we recommend posting your sitemaps.
By putting your sitemap in a /...
Generally google tries to show geo-specific or location specific results.
Google has several ways of determining target locale, for example:
A target locale specified using Search Console's International
Country-code top-level domain names (ccTLDs). These are tied to a
specific country (for example .de for Germany, .cn for China), and
Yes, 410 pages do deindex a bit faster than 404 pages, once Google sees them.
According to Mueller from Google:
The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so.
That being said, Google needs to crawl them before they will be able to find the 404/...
DuckDuckGo sources its results "most commonly from Bing", so it can be useful to use Bing's removal tool to help get stuff off of DuckDuckGo.
Bing Content Removal Tool
As of 2021, DuckDuckGo has no native removal tools.
Google needs a source for the links of your pages, so it can index them. There are multiple ways than can make Google index more pages:
Add a sitemap with most of the pages.
Include breadcrumbs in your pages.
Submit the urls using Search Console or via the API.
So the simple answer to the question Would adding links between pages help Google index more of ...
To the best of my knowledge, Google excludes urls rather than pages, it's a subtle distinction but potentially a useful one to make in this case.
It's always possible that google has indexed a page that has a url showing up in the excluded list, because they indexed that page at a different url that wasn't excluded.
For example, a very common situation ...
If you are fully deleting your site, it can be a one step process:
Delete all files and the domain from the server.
Googlebot and Bingbot will the pages from the search index the next time they crawl them and get an error. It could take a couple weeks for all the pages of your site to be removed from search engines, but it will happen.
If you want to ...
If you don't own the website, Google's site: operator can give you some rough info. It's not that accurate, though.
If you do own the website, use Google Search Console to check the Index Coverage report. This should give you an accurate count of the number of pages that are indexed.
As suggested in other answers you can handle this issue with help of header status.
If you are still in real hurry and want it removed as fast as possible then you can use remove urls from your Google Webmaster Tools >> Search Console.
Usually it is advisable to allow it to get drop by itself but this is when your 404 page is hurting your rankings or ...
Here's what I did:
Google wouldn't index the sub-page even after several times I asked for re-index (for over a week) because it was a duplicate content (to Google) although it wasn't really duplicate. I did then a content rewrite and requested re-index in Search Console. It got indexed in 1-2 days. However, the keyword's position was lost.
I wanted to make ...
The <Files> directive only applies to filenames, not file-paths, so your <Files> directive will never match and the header will not be set.
To set this header on a specific file (and not all .pdf files - as in the linked question/answers) in the root .htaccess file then you can set an environment variable when this file is requested and ...
That is unexpected. The new site effectively inherits the ranking of the old site since incoming links that predominantly influence search results still point to the same domain. This would normally be the same as updating an existing page, which may impact search results eventually.
Presumably old.example.com did not exist before and so it makes no sense ...
I am fairly sure yoy can just ignore its affect on Google. There is a concept of DA - domain authority and PA - page authority. The page autthority for this page will be very low, but if the page is just uninteresting (as opposed to malicious/unrelated to the site) I would expect it to have no effect on the wider site ranking.
A backlink profile with a small Referring domains to Backlinks ratio will hurt your site in most fields since it doesn't look natural. However, as far as I know in the site building industry, it shouldn't be a bad thing.
There are a number of possibilities here:
You have all those links as dofollow, that's pretty bad and should be changed to nofollow as ...
At first, you should check internal links. The distribution of links between the pages of your site must be done properly. For example, in any site, the most important page is the home page. The number of links made for the home page should be more than other pages of the site.
I think it is better for all 3 next important pages, Cat1, PageA and Blog page ...
Okay, I would focus on making your internal links communicate the "Special Pages" to Google by having your Category, Product and Post pages link to your Special pages.
As mentioned by @mehdi, internal linking is one of the ways to draw attention to what's important in your site. However, in your case, it appears Special Pages are more important ...
This is an example of security through obscurity. This is not a good practice. But if you must, then the noindex tag is respected by major search engines (Google, Bing, etc.).
But the better solution would be to:
Password it, or
Only allow certain IPs (your stakeholders), or
Firewall/Fail2Ban any traffic with "bot" in it
There is a difference between being included in Google's index and included in their SERPs.
You only submitted your change of address a month ago. Google has indexed most of the site since then, but that doesn't mean it will immediately rank the pages for inclusion in the results.
In my experience, it can take some time (even more than a month) between ...
When you click from the search results to the old IP address, you get an error page. Googlebot will see this and remove the page from the search index shortly (usually within a few days). Because it is already returning an error page, there is nothing additional that you need to do. Clearing the cache for your home page in GSC isn't going to help ...
You can get to Crawl Stats by using the nav bar to the left. It's under the "Legacy tools and reports" section, if that's what you're referring to.
As far as manually indexing and crawling on a page-by-page basis, you can no longer access the old style of Google Search Console for those, you have to use the new version.
For more information on how ...
As long as it is about adding (topical relevant) content, you will be fine (very, very low risk). Google loves updated content - as this shows that you care about your readers.
If it is about changing the content (and therefor maybe mix up the topic evaluation/weighting, which google does internally), chances are higher to get some shaking within your ranks.