47
votes
Accepted
Why does my IP address become Google's IP address when using Chrome on the mobile?
You have Data Saver turned on in the settings of Google Chrome. It proxies all your HTTP (not HTTPS or incognito) traffic through an optimisation server at Google to make the pages smaller.
- 676
18
votes
Accepted
What is .well-known/traffic-advice directory?
Googlebot with name "Chrome Privacy Preserving Prefetch Proxy" tries to find instructions if it can preload your website for the user surfing on Chrome (Chrome thinks, for example, the link ...
- 342
12
votes
Accepted
Should we drop AJAX crawling scheme?
Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site ...
- 9,112
10
votes
Accepted
Googlebot crawls my page too often
Nothing wrong with that, as Google confirmed, they crawl based on URLs' popularity so, in your case, that article must have more internal/external links than the others.
URLs that are more popular ...
- 3,807
10
votes
Accepted
How to avoid robots from indexing pages of my app through alternate URLs?
It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
- 6,007
9
votes
Are indexed Wordpress admin pages a security threat?
When a new bug is found, in this case in wordpress, the first thing hackers will do is try to find vulnerable sites that use wordpress. A good way of doing it is trying to find wp-admin pages on ...
- 189
9
votes
Accepted
Is it worthwhile to pay for boosting SEO for a new wesite?
Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to:
Create links to your site from bad places
...
- 98.4k
9
votes
Accepted
Can Googlebot discover URLs that are not in an anchor?
Yes they do crawl everything that looks like a URL. I have made a test (in German, but red="not crawled" and green="crawled"): http://www.sirpauls.com/welchen-links-folgt-google-ein-experiment/
...
- 503
9
votes
Why is preventing Googlebot from crawling some pages on my site good for SEO?
Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several ...
- 98.4k
8
votes
Are indexed Wordpress admin pages a security threat?
Security issue? Doubt it. It just makes your site more visible to people who likes to attack wordpress sites. Other than the visibility everything is the same, security through obscurity is not ...
- 189
8
votes
Does googlebot crawl the variable name of a php
Except in the case of misconfiguration, the PHP code is executed on the server and what gets transmitted to the client is only the result of the execution which may be a mix of static HTML + dynamic ...
- 8,325
7
votes
Accepted
Is it bad for SEO to submit a partially finished site to Google?
There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody ...
- 98.4k
7
votes
Are indexed Wordpress admin pages a security threat?
Google doesn't want to index non-content pages including admin pages and pages that ask you to login. Putting wp-admin in robots.txt is better for SEO because it will usually prevent the page from ...
- 98.4k
7
votes
How disqus comments (inside cross domain iframe) are indexed by google?
This is because what you had read is not fully true, as Google says:
Google tries to associate framed content with the page containing the
frames, but we don't guarantee that we will.
Or it was ...
- 2,644
7
votes
Accepted
Ampersand (&) in actual URL and sitemap
if this difference of ampersand in URL and sitemap will cause any issue.
tl;dr No issue, because the URLs are the same.
Since in sitemap & has to be escaped I replaced & with & ...
...
- 42.7k
6
votes
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
This will not affect your SEO. Any web crawler will be able to find a website's HTML sitemap as long as there is a link pointing to it.
Remember that you can submit your sitemap to most search engines ...
- 2,486
6
votes
Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?
Googlebot now renders pages and views the page as a user sees it when it loads, including applying CSS and running JavaScript. Google will detect text that is hidden using either CSS or JavaScript.
...
- 98.4k
6
votes
Accepted
How to block only Yandex bot using robots.txt
There are many "Yandex bots". If you want to block all the Yandex bots then:
User-agent: Yandex
Disallow: /
However, if you only want to block the main indexing "YandexBot" then:
User-agent: ...
- 42.7k
6
votes
Accepted
Does changing from HTTPS to HTTP kill my backlinks?
Google will switch between https and http (and vice versa) without any noticeable link juice loss - as long as the links (apart from prefix) are identical.
However if you want to make sure that your ...
- 1,079
6
votes
Accepted
How does Google bot know that my page content has changed?
Google uses an algorithm to index your page at calculated intervals. The more frequent your updates are - more often the bot will index your page.
The bot only needs to compare the html which is on ...
- 176
6
votes
Accepted
Weird bot requests in logs
So I dealt with something similar a while back and it was a real pain. I was constantly getting requests for random pdfs and Mexican food recipes in my logs. The IPs were Googlebot, Bingbot, etc (...
- 6,221
6
votes
What is .well-known/traffic-advice directory?
As @Maximillian Laumeister mentions the only reference to that .well-known uri is a proposal for prefetch control. I have also come across an obscure reference to it as a middleware layer able to be ...
- 10.2k
5
votes
Accepted
Are indexed Wordpress admin pages a security threat?
If I was running wordpress, then I will not block anything with robots.txt, because it prevents Google for crawling, not for indexing. Just search on Google "Google analytics Web", and you will see ...
- 5,507
5
votes
Accepted
How does Googlebot find a new page on a website?
Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML ...
- 32.8k
5
votes
Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?
A 302 to the home page is likely to be seen as a soft-404 anyway, so that's unlikely to be of any benefit.
A 404 can be perceived as temporary (a 410 is more permanent) and if the page doesn't exist ...
- 42.7k
5
votes
Accepted
How to block Googlebot from accessing one specific page
Since this does appear to be the real Googlebot, the recommended way to block access/crawling is to use /robots.txt:
User-agent: googlebot
Disallow: /blocked.php
However, if you still want to block ...
- 42.7k
5
votes
How do I stop Google from indexing AJAX calls within Javascript?
I would put the Ajax function into robots.txt:
Disallow: /somepage/AjaxFunction.aspx
That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only ...
- 98.4k
5
votes
Accepted
Does Google return to visit a page with 404 Error?
Nice question. The short answer is yes. As long as the error 404 URL has internal/external links pointing to them, Google will eventually visit the page. We can check this through:
Google Search ...
- 3,807
5
votes
Accepted
Is it good for SEO to change an article's slug when changing its title?
Check out the URL of this very post on Stack Exchange, it's a great example of SEO-friendly slug+id addressing:
/questions/119694/is-it-good-for-seo-to-change-an-articles-slug-when-changing-its-title
...
- 15.8k
5
votes
Accepted
GoogleBot doing POST requests
Yes, GoogleBot is capable and does send POST requests (if it is safe - according to Google) - details here.
In the same article, it talks about blocking the URL via robots.txt - which is a fair ...
- 584
Only top scored, non community-wiki answers of a minimum length are eligible