38

There are a number of reasons your content may not appear in search engine results, however, it is important to note that a search engine's index may contain pages that it doesn't display in its results page. How to tell if your content is actually indexed It may actually be difficult to tell if your content is indexed. Search for all the documents from ...


34

To prevent your PDF file (or any non HTML file) from being listed in search results, the only way is to use the HTTP X-Robots-Tag response header, e.g.: X-Robots-Tag: noindex You can do this by adding the following snippet to the site's root .htaccess file or httpd.conf file: <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </...


18

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


17

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...


17

To know the age of an URL you can follow this link by replacing www.example.com by the URL you want: https://www.google.com/search?tbs=cdr%3A1%2Ccd_min%3A1%2F1%2F2000&q=site%3Ahttp%3A%2F%2Fwww.example.com&safe=active&gws_rd=ssl For example, here's the result from Google for the Meta site of Stack Overflow: Otherwise, the Wayback machine is ...


13

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from some registrars. For example, Google found one domain name I registered using GoDaddy, indexed it, and began sending search results within 20 minutes of ...


12

A similar question was asked on Stack Overflow back in January. John Mueller was kind enough to respond with the following: "googleon" and "googleoff" are only supported by the Google Search Appliance (when you host your own search results, usually for your own internal website). They are not supported by Google's web-search at all. As far as I ...


12

MrWhite's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so. (Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?) <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Under nginx ...


12

This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go. There are many proper ways to prevent indexing such as using robots.txt, meta tags or authentication.


10

That's a very bad idea and your site will suffer in the organic search rankings. For one, Google does have image recognition abilities and so your assumption that the bot can't "view" the image is wrong. And two, the algorithm does render pages to decide whether they are not mobile friendly and if your pages are delivering different content to users and to ...


10

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you suggest. The other way is to simply issue 301 Permanent Redirect so that all traffic uses the official URLs. This will also allow you to enforce the use of HTTPS ...


9

They do indeed index the text itself. For proof, check out this site: http://svg.nicubunu.ro/ If you search for the relevant strings, the site shows in the search rankings. I do not know how long it takes to index a new .svg page or element.


9

Indirectly, yes: Create a page with links to all the URLs you'd like re-crawled (like a sitemap) and add that to your website.* Submit the URL to that page to Fetch as Google, selecting Desktop as the fetching strategy, as detailed here: Use Fetch as Google. Once Fetch as Google is complete, and within 4 hours time, from the Fetches Table next to the status ...


9

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file. You could put your JavaScript that shows the password into an external JavaScript file (called showlists.js): $(document).ready(function(){ showLists(); }); Call that JavaScript file in the page head: <...


9

If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers. robots.txt Include a robots.txt at the site's root including: User-agent: * Disallow: / X-Robots-Tag Include the following to your .htaccess to add an X-Robots-Tag ...


9

Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to: Create links to your site from bad places Create doorway pages Spam on your behalf All of that may work initially, but it will come back in the form of a site penalty eventually. SEO for a new site can ...


8

Wordpress is set to ping Google whenever you write a new post. From http://wordpress.org/support/topic/google-blog-pinging Settings->Writing. At the bottom you'll find a list of "Update Services". The default is to ping the pingomatic server, which then pings a dozen or more other places for you, Google included. However, if you want to ping Google ...


8

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally. To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...


8

The best course of action is to use canonical URLs. This avoids a situation where you are penalized for duplicate content. When it comes to desktop vs mobile websites, most sites will have something like this on their mobile website: Example for: http://m.mywebsite.com/page.html <link rel="canonical" href="http://mywebsite.com/page.html" /> The ...


8

Zistoloen found a way to have Google display the date when it first indexed the content of the page. I'm adding it to my answer as well because I think I can explain it more clearly. Search Google for something that brings up the page you want as a result Use "Search Tools" Select "Custom Range..." from the "Any time" drop down Put in a large date range ...


7

Robots.txt just tells search engines not to crawl your pages. It does not tell them to not index your pages. So if your pages have links from other websites the search engines will know they exist. And because off-page factors affect rank, sometimes greatly, your pages can rank well for long-tail search terms without ever being crawled. To actually prevent ...


7

In my experience, mobile visitors want the same content as your desktop visitors do. I worked for a travel website with lots of information about hotels and restaurants. The site is generally known for hotels, but we thought that mobile users would be much more interested in restaurant content because they we looking for something when they were out. ...


7

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the tag ...


7

You do not have an indexing problem. Google Webmasters Tools is the authoritative source of information about your website with Google. Whatever it says there is true. Operators like site: and link: are known to not show all relevant results. This is on purpose as it prevents others from knowing exactly Google is handling your website inhibits any attempts ...


7

No SEO campaign is perfect. So please throw this notion out of the window. Since search engines change strategies and updates come on the average of several a day, it is impossible to have a perfect SEO score at anytime. However, you can have a highly competitive score regardless of the changes if done well. Without knowing your website name and any terms ...


7

Ideally, no. It is very, very difficult to get penalised (Sandboxing, Deindexing). Having duplicate content may 'de-value' your content and it will have less weight than it would organically but it's Google's job to identify the original content whilst devalueing other pages. Setting a canonical link on your website is your way of telling search engines ...


7

There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody searching for its terms. On the other hand you don't want to publish to Google: Broken links, images, or HTML "Coming soon" or "Under construction" notices Pages ...


7

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with &amp; ... Your sitemap is an XML document. As with any XML document, the data values must be stored XML-entity encoded. The & character is a special character (it ...


7

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. ...


6

This is not a good idea and no, you cannot trust these IP ranges. The IP addresses used by Google are not public. But some/most search engine crawlers can be identified by doing a reverse DNS lookup on the IP address. A googlebot example: 66.249.64.0 has a PTR record to crawl-66-249-64-0.googlebot.com, and any IP with a PTR record to a subdomain on ...


Only top voted, non community-wiki answers of a minimum length are eligible