Skip to main content
Share Your Experience: Take the 2024 Developer Survey
47 votes
Accepted

Why does my IP address become Google's IP address when using Chrome on the mobile?

You have Data Saver turned on in the settings of Google Chrome. It proxies all your HTTP (not HTTPS or incognito) traffic through an optimisation server at Google to make the pages smaller.
wb9688's user avatar
  • 676
21 votes
Accepted

What is .well-known/traffic-advice directory?

Googlebot with name "Chrome Privacy Preserving Prefetch Proxy" tries to find instructions if it can preload your website for the user surfing on Chrome (Chrome thinks, for example, the link ...
Jasom Dotnet's user avatar
10 votes
Accepted

Googlebot crawls my page too often

Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others. URLs that are more popular ...
Emirodgar's user avatar
  • 3,817
10 votes
Accepted

How to avoid robots from indexing pages of my app through alternate URLs?

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
Itai's user avatar
  • 5,997
9 votes
Accepted

Can Googlebot discover URLs that are not in an anchor?

Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/ ...
Milo Tischler's user avatar
9 votes

Why is preventing Googlebot from crawling some pages on my site good for SEO?

Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several ...
Stephen Ostermiller's user avatar
8 votes

Does googlebot crawl the variable name of a php

Except in the case of misconfiguration, the PHP code is executed on the server and what gets transmitted to the client is only the result of the execution which may be a mix of static HTML + dynamic ...
Patrick Mevzek's user avatar
7 votes

How disqus comments (inside cross domain iframe) are indexed by google?

This is because what you had read is not fully true, as Google says: Google tries to associate framed content with the page containing the frames, but we don't guarantee that we will. Or it was ...
marcanuy's user avatar
  • 2,644
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with & ... ...
MrWhite's user avatar
  • 42.8k
6 votes

Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?

Googlebot now renders pages and views the page as a user sees it when it loads, including applying CSS and running JavaScript. Google will detect text that is hidden using either CSS or JavaScript. ...
Stephen Ostermiller's user avatar
6 votes
Accepted

How to block only Yandex bot using robots.txt

There are many "Yandex bots". If you want to block all the Yandex bots then: User-agent: Yandex Disallow: / However, if you only want to block the main indexing "YandexBot" then: User-agent: ...
MrWhite's user avatar
  • 42.8k
6 votes
Accepted

Does changing from HTTPS to HTTP kill my backlinks?

Google will switch between https and http (and vice versa) without any noticeable link juice loss - as long as the links (apart from prefix) are identical. However if you want to make sure that your ...
garth's user avatar
  • 1,079
6 votes
Accepted

How does Google bot know that my page content has changed?

Google uses an algorithm to index your page at calculated intervals. The more frequent your updates are - more often the bot will index your page. The bot only needs to compare the html which is on ...
Dmitriy Kravchuk's user avatar
6 votes
Accepted

Weird bot requests in logs

So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (...
Mike Ciffone's user avatar
  • 6,390
6 votes

What is .well-known/traffic-advice directory?

As @Maximillian Laumeister mentions the only reference to that .well-known uri is a proposal for prefetch control. I have also come across an obscure reference to it as a middleware layer able to be ...
Chris Rutherfurd's user avatar
5 votes
Accepted

How does Googlebot find a new page on a website?

Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML ...
closetnoc's user avatar
  • 32.9k
5 votes

Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?

A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit. A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist ...
MrWhite's user avatar
  • 42.8k
5 votes
Accepted

How to block Googlebot from accessing one specific page

Since this does appear to be the real Googlebot, the recommended way to block access/crawling is to use /robots.txt: User-agent: googlebot Disallow: /blocked.php However, if you still want to block ...
MrWhite's user avatar
  • 42.8k
5 votes

How do I stop Google from indexing AJAX calls within Javascript?

I would put the Ajax function into robots.txt: Disallow: /somepage/AjaxFunction.aspx That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only ...
Stephen Ostermiller's user avatar
5 votes
Accepted

How often do search bots typically visit a website?

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do. Search bots almost ...
Stephen Ostermiller's user avatar
5 votes
Accepted

Does Google return to visit a page with 404 Error?

Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through: Google Search ...
Emirodgar's user avatar
  • 3,817
5 votes
Accepted

Is it good for SEO to change an article's slug when changing its title?

Check out the URL of this very post on Stack Exchange, it's a great example of SEO-friendly slug+id addressing: /questions/119694/is-it-good-for-seo-to-change-an-articles-slug-when-changing-its-title ...
Maximillian Laumeister's user avatar
5 votes
Accepted

GoogleBot doing POST requests

Yes, GoogleBot is capable and does send POST requests (if it is safe - according to Google) - details here. In the same article, it talks about blocking the URL via robots.txt - which is a fair ...
Georg Keferböck's user avatar
5 votes
Accepted

Can I hide ads for Googlebot so it loads my site faster?

What you're describing is definitely cloaking. "Cloaking refers to the practice of presenting different content or URLs to human users and search engines," according to Google. That means you'd be ...
Henry Visotski's user avatar
5 votes
Accepted

What headers used in request by google bot?

if ($_SERVER['HTTP_CACHE_CONTROL'] == 'no-cache') { Also bear in mind that the Cache-Control HTTP request header (as with most request headers) are entirely optional, so may not be present as part of ...
MrWhite's user avatar
  • 42.8k
5 votes
Accepted

Why does Googlebot keep coming back to recrawl pages that have been returning 410 for a long time?

Once Googlebot finds a working URL, it will never forget it. It will always come back to crawl it periodically. It doesn't matter what status it returns. It doesn't matter if it still has any links ...
Stephen Ostermiller's user avatar
5 votes

I have 2.1 million pages, how to get crawled by googlebot more frequently?

You need two things: Time External links Googlebot will eventually increase its crawl rate and crawl your entire site. Getting 2 million pages crawled could take a year or two. Even then Google ...
Stephen Ostermiller's user avatar
5 votes
Accepted

Link rel="nofollow" vs robots.txt disallow for "/search" URLs?

Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt. However, that doesn't necessarily prevent the /search pages from appearing as ...
MrWhite's user avatar
  • 42.8k
5 votes
Accepted

How to allow Googlebot to crawl paywall content?

Google says in their flexible sampling documentation: Enclose paywalled content with structured data in order to help Google differentiate paywalled content from the practice of cloaking, where the ...
Stephen Ostermiller's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible