19

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


18

To know the age of an URL you can follow this link by replacing www.example.com by the URL you want: https://www.google.com/search?tbs=cdr%3A1%2Ccd_min%3A1%2F1%2F2000&q=site%3Ahttp%3A%2F%2Fwww.example.com&safe=active&gws_rd=ssl For example, here's the result from Google for the Meta site of Stack Overflow: Otherwise, the Wayback machine is ...


17

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...


14

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


13

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


12

This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go. There are many proper ways to prevent indexing such as using robots.txt, meta tags or authentication.


10

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file. You could put your JavaScript that shows the password into an external JavaScript file (called showlists.js): $(document).ready(function(){ showLists(); }); Call that JavaScript file in the page head: <...


10

That's a very bad idea and your site will suffer in the organic search rankings. For one, Google does have image recognition abilities and so your assumption that the bot can't "view" the image is wrong. And two, the algorithm does render pages to decide whether they are not mobile friendly and if your pages are delivering different content to users and to ...


10

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you suggest. The other way is to simply issue 301 Permanent Redirect so that all traffic uses the official URLs. This will also allow you to enforce the use of HTTPS ...


10

Googlebot will be able to crawl your site fully because it never sends any cookies. There is, however, something else you need to do. Google expects sites to implement limits on the amount of content that users can view per month. Google has a policy about allowing sites in the search results even though they are limited. They call this policy flexible ...


9

Indirectly, yes: Create a page with links to all the URLs you'd like re-crawled (like a sitemap) and add that to your website.* Submit the URL to that page to Fetch as Google, selecting Desktop as the fetching strategy, as detailed here: Use Fetch as Google. Once Fetch as Google is complete, and within 4 hours time, from the Fetches Table next to the status ...


9

If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers. robots.txt Include a robots.txt at the site's root including: User-agent: * Disallow: / X-Robots-Tag Include the following to your .htaccess to add an X-Robots-Tag ...


9

Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to: Create links to your site from bad places Create doorway pages Spam on your behalf All of that may work initially, but it will come back in the form of a site penalty eventually. SEO for a new site can ...


8

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally. To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...


8

The best course of action is to use canonical URLs. This avoids a situation where you are penalized for duplicate content. When it comes to desktop vs mobile websites, most sites will have something like this on their mobile website: Example for: http://m.mywebsite.com/page.html <link rel="canonical" href="http://mywebsite.com/page.html" /> The ...


8

Zistoloen found a way to have Google display the date when it first indexed the content of the page. I'm adding it to my answer as well because I think I can explain it more clearly. Search Google for something that brings up the page you want as a result Use "Search Tools" Select "Custom Range..." from the "Any time" drop down Put in a large date range ...


7

In my experience, mobile visitors want the same content as your desktop visitors do. I worked for a travel website with lots of information about hotels and restaurants. The site is generally known for hotels, but we thought that mobile users would be much more interested in restaurant content because they we looking for something when they were out. ...


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

You do not have an indexing problem. Google Webmasters Tools is the authoritative source of information about your website with Google. Whatever it says there is true. Operators like site: and link: are known to not show all relevant results. This is on purpose as it prevents others from knowing exactly Google is handling your website inhibits any attempts ...


7

No SEO campaign is perfect. So please throw this notion out of the window. Since search engines change strategies and updates come on the average of several a day, it is impossible to have a perfect SEO score at anytime. However, you can have a highly competitive score regardless of the changes if done well. Without knowing your website name and any terms ...


7

Ideally, no. It is very, very difficult to get penalised (Sandboxing, Deindexing). Having duplicate content may 'de-value' your content and it will have less weight than it would organically but it's Google's job to identify the original content whilst devalueing other pages. Setting a canonical link on your website is your way of telling search engines ...


7

There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody searching for its terms. On the other hand you don't want to publish to Google: Broken links, images, or HTML "Coming soon" or "Under construction" notices Pages ...


7

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with &amp; ... Your sitemap is an XML document. As with any XML document, the data values must be stored XML-entity encoded. The & character is a special character (it ...


7

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. ...


6

Most likely some part of your web site generated links like that, and that is how Google started to crawl the URLs. You should check the links in your web pages to see where these incorrect URLs are, and you should fix them. Also, you could change your Apache configuration so that requests for any other virtualhost than example.com or www.example.com would ...


6

You had posted this question differently twice. This answer was to the question marked as duplicate which I now post here hoping to help you. While the two questions may not be exactly the same, this answer is in response to the question marked as duplicate. Please keep that in mind. You have three things going on here and there is little or no connection ...


6

We just switched from HTTP to HTTPS two weeks ago, and after doing extensive research this is what we did: Added the HTTPS property to our Search console. Redirected HTTP robots.txt to HTTPS robots.txt. Placed a new HTTPS sitemap in the robots.txt, and removed the HTTP sitemap URL from robots.txt. Made sure that the old HTTP sitemap was intact, and made ...


6

Early 2018 update to Google Search Console will now tell you if it's overridden a page with a different canonical: Google chose different canonical than user: This URL is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical. Because we consider this page a duplicate, we did not index it; only the canonical page is ...


6

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic search (or elsewhere). All three services respect the robots.txt file, so you could simply disallow it there. User-agent: AhrefsBot Disallow: / User-agent: Neevabot ...


5

One way I found here, is using an <iframe> and blocking the URL in robots.txt.


Only top voted, non community-wiki answers of a minimum length are eligible