Skip to main content

Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Indexed in Bing and Yahoo Search but not in Google although Sitemap is accessed by Google Bot [duplicate]

My website is indexed in Yahoo and Bing. If I search for website, I could see both Yahoo and Bing show it as first result. In Google Search I do not see my website indexed at all. I have tried the ...
Du-Lacoste's user avatar
0 votes
0 answers
21 views

Issues with Googlebot crawling and negative number pages

I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
Julien's user avatar
  • 101
1 vote
0 answers
10 views

Is mismatching trailing slash in hreflang a problem for indexing spiders like Googlebot?

I got a Joomla 4 website. there's a .htaccess rule redirecting URLs ending with / to version without slash, also www. is removed I'm using Aimy Canonical plugin for rel="canonical" URLs. Can'...
Destroy666's user avatar
1 vote
2 answers
55 views

How to handle subtagline on landing page, above the fold, for mobile, without getting penalized by Google?

So I have a pretty good above the fold page for desktop, and it's structured like this: The issue is, I am changing my website to make it more mobile friendly. On mobile, when I use the same landing ...
Programmer Joe's user avatar
1 vote
2 answers
62 views

Best practice for delivering multi-language website

I'm about to turn my website into one that's for multiple languages. I read these two points from Google Search Central which I find rather conflicting. The first point states: Use different URLs for ...
mike_s's user avatar
  • 21
0 votes
0 answers
24 views

Page with Google's reCAPTCHA gets massive traffic peaks from Googlebot, why?

In a web application there is a contact form that is protected by Google's reCAPTCHA. The page as such is set to NOINDEX, FOLLOW: <meta content="noindex, follow" name="robots"&...
basZero's user avatar
  • 177
0 votes
1 answer
75 views

How to decrease the response time of Crawl request

I'm searching for documents related to crawl and indexing. I don't know how to decrease the response time of Google crawl And what impacts on it? Hope you can help me!
Bryant's user avatar
  • 1
0 votes
0 answers
7 views

Sitemap Successfully Crawled and Discovered all 90K pages, yet Google Search results empty [duplicate]

I have my website running on Spring Boot and have created sitemap.xml for google indexing. When logged into the google console I could see all 90K pages are discovered and success as the status. It is ...
Du-Lacoste's user avatar
1 vote
0 answers
12 views

301 Old content to New Content but now different language (negative SEO effect?)

I have a question. I have a Dutch website and used to receive backlinks from good and high-quality websites. Now, I want to 301 redirect the Dutch page to an English page. Will this affect my website'...
Main Elite's user avatar
1 vote
1 answer
107 views

Bizarre URLs showing as index errors on website (Massive uptick in "non indexed URLs"

My site is having issues with Googlebot queries (ridiculous numbers of them that are bogging down the server); I'm not sure if that is related to this... My Google Search Console shows a MASSIVE ...
skrimpy's user avatar
  • 11
0 votes
0 answers
112 views

google search console crawl stats average response time jump after temporary network change

I have one windows IIS server to serve my website images and it works good enough . for a part of time, starting from 27 Dec 2023 to 16 Jan 2024, I changed public or valid ip of IIS web server and set ...
sadegh mohebbi's user avatar
0 votes
0 answers
21 views

Exceed crawl limits for a website [duplicate]

In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
Kenix's user avatar
  • 23
0 votes
1 answer
43 views

Crawl budget with external links

I want to index 1+ millions pages and I have a lot special pages for each items. Items link to multiple external websites. Should I use nofollow for these external links to reduce spending crawl ...
Kenix's user avatar
  • 23
1 vote
1 answer
46 views

Bots trying to crawl non-existant movie related pages which appear to be from IMDB on my project management website

In the Visitors log in cPanel I can see that googlebot, semrushbot, yandex and others are trying to access non-existant urls. The urls are related to movies & actors and appear to be on IMDB. They ...
projectmonkey's user avatar
0 votes
1 answer
62 views

Risk-benefit of cleaning up thousands of "Alternate page with proper canonical tag"?

The canonical URL to our products is example.com/product-alias in the sitemap, inside link rel="canonical" etc., and Google picks it up. Yet from the category pages, we link to our products ...
ᴍᴇʜᴏᴠ's user avatar
0 votes
0 answers
34 views

Indexing speed of 500k+ pages site

I am reaching out for some guidance regarding the Google Search Console (GSC) indexing of my website. I launched this site about a month ago, and I'm encountering some challenges with Google's ...
Nik's user avatar
  • 1
1 vote
0 answers
38 views

Getting indexed on Google from mysql database [closed]

I have a website that sells 100,000 items. The item data is stored in a mysql database. So, there aren't 100,000 html files on my site. Therefore, google can't see them. I read that the solution to ...
CheeseFlavored's user avatar
0 votes
0 answers
22 views

Block Googlebots from fetching robots.txt on a microservice subdomain

We have some microservices and as an example, Language Service's URL is the following; http://prod-lang-service.example.com/ We have been getting errors (NotFoundException) on Rollbar that Google ...
Shalafister's's user avatar
1 vote
0 answers
90 views

How to allow indexing images for Favicon only?

Generally I like to disallow search engines to index images on my site. There is one exception to this: I like the Google Favicon Bot to index images in order to show the icon in search results. I am ...
Fabian Barney's user avatar
3 votes
1 answer
32 views

Why site:example.com doesn't give the same amount of results in incognito mode

Did you ever tried to search on Google site:domain.example on incognito mode? I am never getting the same amount of results than when I am doing it in normal mode. Sometimes more, sometimes less, do ...
Charly berthet's user avatar
2 votes
1 answer
79 views

Validation of canonical links not working in Google Search Console

I internationalized my site's canonical links using the guide from here: https://yoast.com/hreflang-ultimate-guide/#hreflang-canonical Which has an output like this: <link rel="canonical" ...
James L.'s user avatar
  • 217
1 vote
1 answer
29 views

How to properly block indexing my admin subdomain?

I have a website hosted on https://admin.example.com. However, the public website displayed is on this sub-directory : https://www.example.com/website/ For technical reasons, we cannot install it or ...
B4b4j1's user avatar
  • 131
0 votes
1 answer
49 views

How to overcome Google indexing issue? [duplicate]

It's almost been more than 6 months i'm facing Google indexing issue my site can not be indexed. The issues are described below: 1- "Excluded by ‘noindex’ tag" although I have searched all ...
FARIS ISMAIL's user avatar
2 votes
1 answer
267 views

Google Search console: Blocked due to access forbidden (403) when blocking all country except (India, Srilanka)

I am a very old user of cloudflare for the last 4 Years, I have 14+ websites of different categories which has have added to cloudflare last year. My website was accessible and ranked on google which ...
vicky's user avatar
  • 21
2 votes
0 answers
867 views

How good is Google in indexing and crawling SPA webpages?

I am about to relaunch our website, but I am uncertain whether I should go with a Vue.js SPA webpage or a Nuxt.js SSR webpage. The big advantage of an SPA webpage is that it has an app-like feeling ...
Jan's user avatar
  • 121
1 vote
1 answer
201 views

Googlebot Spam / DDoS?

I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
4m1r's user avatar
  • 111
1 vote
1 answer
143 views

Large increase in 'Discovered - currently not indexed' pages in Google Webmasters

For last few months, we noticed a large increase in 'Discovered - currently not indexed' pages in Webmaster tools. By last month, we had almost half of our site pages were classified as discovered but ...
JM John's user avatar
  • 43
1 vote
0 answers
18 views

Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]

Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
John Paul's user avatar
2 votes
1 answer
644 views

How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
Clipso's user avatar
  • 21
4 votes
3 answers
148 views

Use query parameters or hashes when implementing referral codes?

We're implementing a feature where users can generate unique referral links that they can share and receive rewards when people sign up. The pages themselves have identical content. There are two ...
dodov's user avatar
  • 171
1 vote
1 answer
20 views

How can a react/vue apps like these rank just by using client side rendering?

By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about. here is this site: ...
AbHi's user avatar
  • 11
1 vote
1 answer
220 views

0 discovered links in Google Search Console and only my homepage is indexed [duplicate]

Last month I launched my website. It is using SSR, fixed almost all high/medium priority SEO issues existing in screaming frog, all pages are indexable and SEO friendly and from my index page I have a ...
Dragos Romaniuc's user avatar
1 vote
1 answer
114 views

If Both Article Page and PDF File Have the Same Content, Will It Cause SEO Problems?

I have a blog page and I publish articles on this blog page. You can download these articles as PDFs by clicking the download button on that article. PDFs are the same content found in the article. ...
ardacarofficial's user avatar
0 votes
1 answer
292 views

Does blocking api URL hurt rendering by googlebot

In my GSC crawl stats most of the 404s and URLs using page resource load are: API URLs for a JavaScript part of our content Cloudflare's managed challenge If I add an entry to robots.txt for 1, will ...
ooxbal's user avatar
  • 1
2 votes
0 answers
450 views

Why is the Chrome-Lighthouse user agent hitting my server every day?

Every day, I find lines like this one in my access.log file: 74.125.210.42 - - [08/Mar/2023:08:11:13 +0100] "GET /fr/ HTTP/2.0" 200 24160 "-" "Mozilla/5.0 (Macintosh; Intel ...
zindigi hachim no ippo's user avatar
2 votes
1 answer
200 views

Google Spidering ajax Call URL as Stand Alone Content

My page has an ajax call that loads fresh html into the mother page. Google has now spidered these ajax urls and runs it's search console algorythm over them as if it were a stand alone page. In the ...
M Corby's user avatar
  • 21
0 votes
0 answers
19 views

Does pinging a sitemap under a certain path affect crawling of parent directories/sitemaps?

I couldn't find much information about this topic in the official docs, but under Google's best practices: "You can host your sitemaps anywhere on your site, but a sitemap affects only ...
CPHPython's user avatar
  • 101
1 vote
1 answer
256 views

How did Google crawler view my password-protected pages?

I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped. I am trying to figure out how this could have happened. It is a large site ...
James L.'s user avatar
  • 217
1 vote
1 answer
120 views

Site search form getting spammed, but by Google IP

I have the search form on a section of a site that is getting bombarded with requests like: search query = baccarat philippe starck furniture, ➡️g188.ph⬅️, play free slots with bonus gcash logo cash, ...
rjbathgate's user avatar
2 votes
1 answer
48 views

Will search engines deindex the entire resource when returning HTTP 410 for a parameterized URL?

Will the HTTP 410 response code make search engines deindex the underlying page resource? Or will Googlebot and Bingbot only deindex the URL with query it's parameters as whole? The topic Redirection ...
davidgo's user avatar
  • 8,104
2 votes
1 answer
4k views

What does User-agent: Mediapartners-Google mean?

I was going through the google support website and there they said: "To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
Muhammad Ahmad's user avatar
1 vote
0 answers
761 views

Removing __NEXT_DATA__ script tag for google bots

Our e commerce website is built on next js. We wanted to optimise the bit crawl bandwidth by reducing html size shipped to google bots. We want to treat user and bot differently. As we know bots don’t ...
Ayyappa Gollu's user avatar
0 votes
0 answers
159 views

Google Bots Crawling Invalid URL

Google Bots crawl URLs by appending trailing 200 and false for all the article pages. I am unsure what the issue is here and why that is getting appended on every article page. I have tried scanning ...
Srinivas Ramakrishna's user avatar
3 votes
1 answer
126 views

What is the actual HTML rendered to bots?

We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered. 1. Scenario1: Google rich results Now, we have observed ...
Ayyappa Gollu's user avatar
2 votes
1 answer
82 views

How to handle additional content on paginated pages?

we have category pages with a few hundred products per category. The category pages are properly paginated according to Googles guidelines for pagination. One page shows 16 products. In addition to ...
Luke's user avatar
  • 141
1 vote
1 answer
76 views

Is blocking indexing and crawling at the same time bad?

have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
Supersporadic's user avatar
1 vote
0 answers
64 views

Google Crawls mostly Javascript files of my HTML website [closed]

I have an issue with my HTML static page website, and the issue in my HTML(DOM) crawls only 11% and javascript crawls 70%
Yasin Sunni's user avatar
2 votes
1 answer
52 views

Is disallowing all useragents sufficient to keep a site from being crawled by search engines?

We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
Ian Pringle's user avatar
2 votes
0 answers
42 views

One day after putting my site online and registering it with Google I got spam mail. How?

I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
Frank Conijn - Support Ukraine's user avatar
2 votes
0 answers
30 views

Googlebot increased the number of requests for a certain URL even though it has no links

As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
Lalikman's user avatar

1
2 3 4 5
19