Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
0 votes
1 answer
31 views

How to decrease the response time of Crawl request

I'm searching for documents related to crawl and indexing. I don't know how to decrease the response time of Google crawl And what impacts on it? Hope you can help me!
Bryant's user avatar
  • 1
0 votes
0 answers
7 views

Sitemap Successfully Crawled and Discovered all 90K pages, yet Google Search results empty [duplicate]

I have my website running on Spring Boot and have created sitemap.xml for google indexing. When logged into the google console I could see all 90K pages are discovered and success as the status. It is ...
Du-Lacoste's user avatar
1 vote
0 answers
9 views

301 Old content to New Content but now different language (negative SEO effect?)

I have a question. I have a Dutch website and used to receive backlinks from good and high-quality websites. Now, I want to 301 redirect the Dutch page to an English page. Will this affect my website'...
Main Elite's user avatar
0 votes
1 answer
42 views

Bizarre URLs showing as index errors on website (Massive uptick in "non indexed URLs"

My site is having issues with Googlebot queries (ridiculous numbers of them that are bogging down the server); I'm not sure if that is related to this... My Google Search Console shows a MASSIVE ...
skrimpy's user avatar
0 votes
0 answers
70 views

google search console crawl stats average response time jump after temporary network change

I have one windows IIS server to serve my website images and it works good enough . for a part of time, starting from 27 Dec 2023 to 16 Jan 2024, I changed public or valid ip of IIS web server and set ...
sadegh mohebbi's user avatar
0 votes
0 answers
21 views

Exceed crawl limits for a website [duplicate]

In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
Kenix's user avatar
  • 23
0 votes
1 answer
32 views

Crawl budget with external links

I want to index 1+ millions pages and I have a lot special pages for each items. Items link to multiple external websites. Should I use nofollow for these external links to reduce spending crawl ...
Kenix's user avatar
  • 23
1 vote
1 answer
42 views

Bots trying to crawl non-existant movie related pages which appear to be from IMDB on my project management website

In the Visitors log in cPanel I can see that googlebot, semrushbot, yandex and others are trying to access non-existant urls. The urls are related to movies & actors and appear to be on IMDB. They ...
projectmonkey's user avatar
0 votes
1 answer
53 views

Risk-benefit of cleaning up thousands of "Alternate page with proper canonical tag"?

The canonical URL to our products is example.com/product-alias in the sitemap, inside link rel="canonical" etc., and Google picks it up. Yet from the category pages, we link to our products ...
ᴍᴇʜᴏᴠ's user avatar
0 votes
0 answers
32 views

Indexing speed of 500k+ pages site

I am reaching out for some guidance regarding the Google Search Console (GSC) indexing of my website. I launched this site about a month ago, and I'm encountering some challenges with Google's ...
Nik's user avatar
  • 1
1 vote
0 answers
34 views

Getting indexed on Google from mysql database [closed]

I have a website that sells 100,000 items. The item data is stored in a mysql database. So, there aren't 100,000 html files on my site. Therefore, google can't see them. I read that the solution to ...
CheeseFlavored's user avatar
0 votes
0 answers
18 views

Block Googlebots from fetching robots.txt on a microservice subdomain

We have some microservices and as an example, Language Service's URL is the following; http://prod-lang-service.example.com/ We have been getting errors (NotFoundException) on Rollbar that Google ...
Shalafister's's user avatar
1 vote
0 answers
75 views

How to allow indexing images for Favicon only?

Generally I like to disallow search engines to index images on my site. There is one exception to this: I like the Google Favicon Bot to index images in order to show the icon in search results. I am ...
Fabian Barney's user avatar
3 votes
1 answer
27 views

Why site:example.com doesn't give the same amount of results in incognito mode

Did you ever tried to search on Google site:domain.example on incognito mode? I am never getting the same amount of results than when I am doing it in normal mode. Sometimes more, sometimes less, do ...
Charly berthet's user avatar
2 votes
1 answer
67 views

Validation of canonical links not working in Google Search Console

I internationalized my site's canonical links using the guide from here: https://yoast.com/hreflang-ultimate-guide/#hreflang-canonical Which has an output like this: <link rel="canonical" ...
James L.'s user avatar
  • 217
1 vote
1 answer
28 views

How to properly block indexing my admin subdomain?

I have a website hosted on https://admin.example.com. However, the public website displayed is on this sub-directory : https://www.example.com/website/ For technical reasons, we cannot install it or ...
B4b4j1's user avatar
  • 131
0 votes
1 answer
49 views

How to overcome Google indexing issue? [duplicate]

It's almost been more than 6 months i'm facing Google indexing issue my site can not be indexed. The issues are described below: 1- "Excluded by ‘noindex’ tag" although I have searched all ...
FARIS ISMAIL's user avatar
2 votes
1 answer
231 views

Google Search console: Blocked due to access forbidden (403) when blocking all country except (India, Srilanka)

I am a very old user of cloudflare for the last 4 Years, I have 14+ websites of different categories which has have added to cloudflare last year. My website was accessible and ranked on google which ...
vicky's user avatar
  • 21
2 votes
0 answers
538 views

How good is Google in indexing and crawling SPA webpages?

I am about to relaunch our website, but I am uncertain whether I should go with a Vue.js SPA webpage or a Nuxt.js SSR webpage. The big advantage of an SPA webpage is that it has an app-like feeling ...
Jan's user avatar
  • 121
1 vote
1 answer
146 views

Googlebot Spam / DDoS?

I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
4m1r's user avatar
  • 111
1 vote
1 answer
133 views

Large increase in 'Discovered - currently not indexed' pages in Google Webmasters

For last few months, we noticed a large increase in 'Discovered - currently not indexed' pages in Webmaster tools. By last month, we had almost half of our site pages were classified as discovered but ...
JM John's user avatar
  • 43
1 vote
0 answers
18 views

Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]

Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
John Paul's user avatar
2 votes
1 answer
486 views

How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
Clipso's user avatar
  • 21
4 votes
3 answers
116 views

Use query parameters or hashes when implementing referral codes?

We're implementing a feature where users can generate unique referral links that they can share and receive rewards when people sign up. The pages themselves have identical content. There are two ...
dodov's user avatar
  • 171
1 vote
1 answer
19 views

How can a react/vue apps like these rank just by using client side rendering?

By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about. here is this site: ...
AbHi's user avatar
  • 11
1 vote
1 answer
175 views

0 discovered links in Google Search Console and only my homepage is indexed [duplicate]

Last month I launched my website. It is using SSR, fixed almost all high/medium priority SEO issues existing in screaming frog, all pages are indexable and SEO friendly and from my index page I have a ...
Dragos Romaniuc's user avatar
1 vote
1 answer
96 views

If Both Article Page and PDF File Have the Same Content, Will It Cause SEO Problems?

I have a blog page and I publish articles on this blog page. You can download these articles as PDFs by clicking the download button on that article. PDFs are the same content found in the article. ...
ardacarofficial's user avatar
0 votes
1 answer
258 views

Does blocking api URL hurt rendering by googlebot

In my GSC crawl stats most of the 404s and URLs using page resource load are: API URLs for a JavaScript part of our content Cloudflare's managed challenge If I add an entry to robots.txt for 1, will ...
ooxbal's user avatar
  • 1
1 vote
0 answers
381 views

Why is the Chrome-Lighthouse user agent hitting my server every day?

Every day, I find lines like this one in my access.log file: 74.125.210.42 - - [08/Mar/2023:08:11:13 +0100] "GET /fr/ HTTP/2.0" 200 24160 "-" "Mozilla/5.0 (Macintosh; Intel ...
zindigi hachim no ippo's user avatar
2 votes
1 answer
175 views

Google Spidering ajax Call URL as Stand Alone Content

My page has an ajax call that loads fresh html into the mother page. Google has now spidered these ajax urls and runs it's search console algorythm over them as if it were a stand alone page. In the ...
M Corby's user avatar
  • 21
0 votes
0 answers
18 views

Does pinging a sitemap under a certain path affect crawling of parent directories/sitemaps?

I couldn't find much information about this topic in the official docs, but under Google's best practices: "You can host your sitemaps anywhere on your site, but a sitemap affects only ...
CPHPython's user avatar
  • 101
1 vote
1 answer
193 views

How did Google crawler view my password-protected pages?

I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped. I am trying to figure out how this could have happened. It is a large site ...
James L.'s user avatar
  • 217
1 vote
1 answer
92 views

Site search form getting spammed, but by Google IP

I have the search form on a section of a site that is getting bombarded with requests like: search query = baccarat philippe starck furniture, ➡️g188.ph⬅️, play free slots with bonus gcash logo cash, ...
rjbathgate's user avatar
2 votes
1 answer
45 views

Will search engines deindex the entire resource when returning HTTP 410 for a parameterized URL?

Will the HTTP 410 response code make search engines deindex the underlying page resource? Or will Googlebot and Bingbot only deindex the URL with query it's parameters as whole? The topic Redirection ...
davidgo's user avatar
  • 7,904
2 votes
1 answer
3k views

What does User-agent: Mediapartners-Google mean?

I was going through the google support website and there they said: "To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
Muhammad Ahmad's user avatar
1 vote
0 answers
613 views

Removing __NEXT_DATA__ script tag for google bots

Our e commerce website is built on next js. We wanted to optimise the bit crawl bandwidth by reducing html size shipped to google bots. We want to treat user and bot differently. As we know bots don’t ...
Ayyappa Gollu's user avatar
0 votes
0 answers
142 views

Google Bots Crawling Invalid URL

Google Bots crawl URLs by appending trailing 200 and false for all the article pages. I am unsure what the issue is here and why that is getting appended on every article page. I have tried scanning ...
Srinivas Ramakrishna's user avatar
3 votes
1 answer
120 views

What is the actual HTML rendered to bots?

We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered. 1. Scenario1: Google rich results Now, we have observed ...
Ayyappa Gollu's user avatar
2 votes
1 answer
72 views

How to handle additional content on paginated pages?

we have category pages with a few hundred products per category. The category pages are properly paginated according to Googles guidelines for pagination. One page shows 16 products. In addition to ...
Luke's user avatar
  • 141
1 vote
1 answer
62 views

Is blocking indexing and crawling at the same time bad?

have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
Supersporadic's user avatar
1 vote
0 answers
61 views

Google Crawls mostly Javascript files of my HTML website [closed]

I have an issue with my HTML static page website, and the issue in my HTML(DOM) crawls only 11% and javascript crawls 70%
Yasin Sunni's user avatar
2 votes
1 answer
51 views

Is disallowing all useragents sufficient to keep a site from being crawled by search engines?

We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
Ian Pringle's user avatar
2 votes
0 answers
42 views

One day after putting my site online and registering it with Google I got spam mail. How?

I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
Frank Conijn - Support Ukraine's user avatar
2 votes
0 answers
30 views

Googlebot increased the number of requests for a certain URL even though it has no links

As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
Lalikman's user avatar
0 votes
0 answers
19 views

MODx issue, cannot be crawled [duplicate]

I have a page https://zabery365.ru/removal/bath for my client. Please look at the code. I have a big trouble - for a 3 weeks it has not been crawled, pagespeed also said that there are no data. GSC ...
Rohh Thone's user avatar
1 vote
1 answer
61 views

Alternative to AWS lambda redirect that doesn't affect Googlebot

We have the following geographic subdirectories of our website: www.example.com (for USA and worldwide visitors) www.example.com/en-au/ (for Australian visitors) www.example.com/en-gb/ (for UK ...
Steve's user avatar
  • 1,456
3 votes
0 answers
52 views

Can I use canonical tags in place of 301 redirect that remove a parameter that had been used by a meta refresh?

For one of my sites, I had used a the meta tag refresh: ​​<meta http-equiv="refresh" content="...;url=..."> A few months ago, I removed this tag and instead implemented 301 ...
enzo's user avatar
  • 131
3 votes
0 answers
92 views

Does Google crawl domains with dynamic IP addresses?

Does Google crawl web sites with dynamic IP addresses and dynamic DNS entries?
Ralph Trenkler's user avatar
2 votes
0 answers
308 views

What is correct way to stop query indexing using Robots.txt?

How to stop indexing search result using Robots.txt? hello. Recently, I'm having an issue with indexed pages in Google Search Console growing 10x. These pages are mostly search query (/?s=) and ...
cheonmu's user avatar
  • 21
2 votes
2 answers
735 views

Will Googlebot click on tab navigation / buttons and index links?

I have the following UI: There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
Lars Flieger's user avatar

1
2 3 4 5
19