47 votes
Accepted

Why does my IP address become Google's IP address when using Chrome on the mobile?

You have Data Saver turned on in the settings of Google Chrome. It proxies all your HTTP (not HTTPS or incognito) traffic through an optimisation server at Google to make the pages smaller.
wb9688's user avatar
  • 676
18 votes
Accepted

What is .well-known/traffic-advice directory?

Googlebot with name "Chrome Privacy Preserving Prefetch Proxy" tries to find instructions if it can preload your website for the user surfing on Chrome (Chrome thinks, for example, the link ...
Jasom Dotnet's user avatar
12 votes
Accepted

Should we drop AJAX crawling scheme?

Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site ...
Ilmari Karonen's user avatar
10 votes
Accepted

Googlebot crawls my page too often

Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others. URLs that are more popular ...
Emirodgar's user avatar
  • 3,807
10 votes
Accepted

How to avoid robots from indexing pages of my app through alternate URLs?

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
Itai's user avatar
  • 6,007
9 votes

Are indexed Wordpress admin pages a security threat?

When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on ...
CristianTM's user avatar
9 votes
Accepted

Is it worthwhile to pay for boosting SEO for a new wesite?

Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to: Create links to your site from bad places ...
Stephen Ostermiller's user avatar
9 votes
Accepted

Can Googlebot discover URLs that are not in an anchor?

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/ ...
Milo Tischler's user avatar
9 votes

Why is preventing Googlebot from crawling some pages on my site good for SEO?

Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several ...
Stephen Ostermiller's user avatar
8 votes

Are indexed Wordpress admin pages a security threat?

Security issue? Doubt it. It just makes your site more visible to people who likes to attack wordpress sites. Other than the visibility everything is the same, security through obscurity is not ...
cdvv7788's user avatar
  • 189
8 votes

Does googlebot crawl the variable name of a php

Except in the case of misconfiguration, the PHP code is executed on the server and what gets transmitted to the client is only the result of the execution which may be a mix of static HTML + dynamic ...
Patrick Mevzek's user avatar
7 votes
Accepted

Is it bad for SEO to submit a partially finished site to Google?

There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody ...
Stephen Ostermiller's user avatar
7 votes

Are indexed Wordpress admin pages a security threat?

Google doesn't want to index non-content pages including admin pages and pages that ask you to login. Putting wp-admin in robots.txt is better for SEO because it will usually prevent the page from ...
Stephen Ostermiller's user avatar
7 votes

How disqus comments (inside cross domain iframe) are indexed by google?

This is because what you had read is not fully true, as Google says: Google tries to associate framed content with the page containing the frames, but we don't guarantee that we will. Or it was ...
marcanuy's user avatar
  • 2,644
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with & ... ...
MrWhite's user avatar
  • 42.7k
6 votes

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it. Remember that you can submit your sitemap to most search engines ...
Raul Reyes's user avatar
  • 2,486
6 votes

Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?

Googlebot now renders pages and views the page as a user sees it when it loads, including applying CSS and running JavaScript. Google will detect text that is hidden using either CSS or JavaScript. ...
Stephen Ostermiller's user avatar
6 votes
Accepted

How to block only Yandex bot using robots.txt

There are many "Yandex bots". If you want to block all the Yandex bots then: User-agent: Yandex Disallow: / However, if you only want to block the main indexing "YandexBot" then: User-agent: ...
MrWhite's user avatar
  • 42.7k
6 votes
Accepted

Does changing from HTTPS to HTTP kill my backlinks?

Google will switch between https and http (and vice versa) without any noticeable link juice loss - as long as the links (apart from prefix) are identical. However if you want to make sure that your ...
garth's user avatar
  • 1,079
6 votes
Accepted

How does Google bot know that my page content has changed?

Google uses an algorithm to index your page at calculated intervals. The more frequent your updates are - more often the bot will index your page. The bot only needs to compare the html which is on ...
Dmitriy Kravchuk's user avatar
6 votes
Accepted

Weird bot requests in logs

So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (...
Mike Ciffone's user avatar
  • 6,221
6 votes

What is .well-known/traffic-advice directory?

As @Maximillian Laumeister mentions the only reference to that .well-known uri is a proposal for prefetch control. I have also come across an obscure reference to it as a middleware layer able to be ...
Chris Rutherfurd's user avatar
5 votes
Accepted

Are indexed Wordpress admin pages a security threat?

If I was running wordpress, then I will not block anything with robots.txt, because it prevents Google for crawling, not for indexing. Just search on Google "Google analytics Web", and you will see ...
Goyllo's user avatar
  • 5,507
5 votes
Accepted

How does Googlebot find a new page on a website?

Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML ...
closetnoc's user avatar
  • 32.8k
5 votes

Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?

A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit. A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist ...
MrWhite's user avatar
  • 42.7k
5 votes
Accepted

How to block Googlebot from accessing one specific page

Since this does appear to be the real Googlebot, the recommended way to block access/crawling is to use /robots.txt: User-agent: googlebot Disallow: /blocked.php However, if you still want to block ...
MrWhite's user avatar
  • 42.7k
5 votes

How do I stop Google from indexing AJAX calls within Javascript?

I would put the Ajax function into robots.txt: Disallow: /somepage/AjaxFunction.aspx That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only ...
Stephen Ostermiller's user avatar
5 votes
Accepted

Does Google return to visit a page with 404 Error?

Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through: Google Search ...
Emirodgar's user avatar
  • 3,807
5 votes
Accepted

Is it good for SEO to change an article's slug when changing its title?

Check out the URL of this very post on Stack Exchange, it's a great example of SEO-friendly slug+id addressing: /questions/119694/is-it-good-for-seo-to-change-an-articles-slug-when-changing-its-title ...
Maximillian Laumeister's user avatar
5 votes
Accepted

GoogleBot doing POST requests

Yes, GoogleBot is capable and does send POST requests (if it is safe - according to Google) - details here. In the same article, it talks about blocking the URL via robots.txt - which is a fair ...
Georg Keferböck's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible