Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

How does google read Quora's QA schema, it doesn have RDFa, Json LD, or Microdata? [closed]

I'm making a scraper to read question / answer data for students that supports RDFa, Json LD, and Microdata, but quora confuses me. In a google search, I see a QA block, but if I go to the URL https://...
1
vote
1answer
23 views

Googlebot returning 200 and 300 for the same URLs

Is there a reason why a URL on our website would sometimes return 200 and sometimes return a 301? This appears to happen on a number of our URLs even something basic like example.com/robots.txt When I ...
2
votes
2answers
230 views

Does submitting sitemap.xml once change the way the site's new pages will be crawled/discovered in the future?

My static site doesn't generate sitemap.xml automatically. Hundreds of pages have been modified recently (meta description, etc.). I would like Google to recrawl the whole website. Submitting the URLs ...
0
votes
0answers
5 views

What Is The Reasons That GSC Shows "Page Resources Couldn't Be Loaded" In Live Test? [duplicate]

I Edit My robots.txt File And Put This Code In It: User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml But The Issue Is Not Fixed. Sometimes My CSS File, Sometimes JS File And Sometimes ...
2
votes
1answer
45 views

Does ARIA help SEO? Have you experienced any benefits for organic search?

I've seen mention of ARIA benefitting SEO, but the author cites no evidence — none of the articles cited mention this. I have also searched and haven't found any reliable resources to suggest that ...
3
votes
1answer
64 views

Is there a way to check how Googlebot would render non-public pages?

I have a page which uses react, and from what I understood docs, Google bot, before indexing, is executing JS (see here: JavaScript SEO Basics). First thing which I want to ask, is if my thinking is ...
3
votes
2answers
45 views

Linking relevant ecommerce links with nofollow

I have a price comparison website which lists different products. On the product pages it lists 10-20 webshops per product where the item can be purchased. At the moment I am linking these sites with ...
3
votes
1answer
32 views

Does google index content that's inserted using setTimeout after the page is loaded?

Let's say we have this code <body> content-1 </body> <script> setTimeout(changeBodyContentTo("content-2"), 5000); </script> In this case will google crawl and ...
0
votes
1answer
22 views

<title> tag and favicon aren't correctly showed by Google Search [duplicate]

I'll go straight to the point. For a few days I've been working on the SEO on my website and I noticed that the favicon still isn't showed by Google Search and that the <title> tag is showed ...
4
votes
1answer
47 views

How to specify paywall contents for multiple pages

I am reading this guide to apply paywall contents to my website so that Google knows I am not cloaking. Unless I have misunderstood something, the example in that guide only deals with 1 particular ...
4
votes
1answer
138 views

Does google follow links discovered in JSON-LD?

Lets say I have a site with a page of 20 products which are fed through an API. The products are loaded using infinity scroll so the initial page render has 5 products tops. Upon scrolling it adds 5 ...
1
vote
1answer
28 views

Why does not googlebot crawl my website? [duplicate]

I create some websites and ping them to google using pingomatic. Some websites are soon indexed by google, but others are not indexed at all(site: does not return any result). Note that this is not ...
7
votes
4answers
3k views

Why is preventing Googlebot from crawling some pages on my site good for SEO?

Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site. https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://...
4
votes
1answer
63 views

Weird bot requests in logs

I restored a website on a new server after it was down (mistakenly deleted) for a while, and it immediately got bombarded with requests to non-existing pages like /tissue-salts-for-weight-loss.html ...
3
votes
0answers
59 views

Why are pages of this sub-domain indexed by google? [closed]

If you site:news.cnblogs.com in google's search box, you will see a lot of web pages are indexed by google. But if you click on any of the search results, you will be redirected to account.cnblogs.com ...
4
votes
1answer
67 views

How to allow Googlebot to crawl paywall content?

My website has pages for around 1000 companies. Without an account, I only allow users to view 5 company pages a month (I track them by cookie). And Googlebot can crawl my site freely since it does ...
1
vote
1answer
35 views

If noindex pages don't have H1, meta tags, or other desirable elements for SEO, will it hurt the SEO of indexed pages?

I have some pages with noindex. These pages have some SEO errors. SEO errors include: not using H1 Not using meta tags Not adding alt="" tags to images But there no such SEO problems on my ...
1
vote
1answer
25 views

Can Googlebot crawl content from JS fetch?

In my website for many pages, on page load I would call fetch in JS to get the data from the server and then render it. I wonder whether Googlebot can crawl content from fetch? I have tried to search ...
1
vote
0answers
52 views

Can Google bot index pages built with websocket if I implement dynamic rendering?

This is the first time I build a website so I did not know anything about SEO when I started. I built many pages of my website to be rendered by React + Redux with the data received from websocket (...
3
votes
2answers
173 views

Does Googlebot simulate page scroll to load additional content?

The content of the home page of the project I'm building is divided into 6 parts where the user scrolls and as soon as the first part "section" reaches the end another part is added via AJAX,...
2
votes
1answer
59 views

Google Search Console and sitemap-index file gzip

I googled for a long time and tried to find something at https://www.sitemaps.org/protocol.html#index about gzip compression for sitemap INDEX files, but I couldn't find anything. I have a very large ...
2
votes
2answers
79 views

What impact do text-based accordion elements have on SEO?

I'm looking for some consensus on what the current school of though is about accordion elements. Here is a page that uses them as an example: https://www.goaskeve.com/venues/goldsmiths-centre/ Will ...
0
votes
0answers
11 views

Does separator (?) in URl affetcs seo? [duplicate]

I'm currently trying to build a websites that list all the states in the country. When a user click on a specific text(state) it redirected to a new page and where all the state in the country are ...
2
votes
1answer
25 views

How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt Could someone elaborate on the difference between these two: http://example.com/page.htm allow: /page ...
1
vote
0answers
24 views

Refresh website cache and indexing [duplicate]

I made changes to my website 2 weeks ago, ever since Google is not indexing and updating the cache. Last it is showing a cache of 16Jun on the landing page. All other search engines like DuckDuckGo, ...
0
votes
1answer
32 views

Google taking my logo as title and description some text on the page [duplicate]

I made a website using nuxt.js and added title and meta description on each page. I also made a sitemap.xml and robots.txt for the site and submitted in Google search console.The issue is Google and ...
0
votes
0answers
15 views

Google Jobs. Issue when clicking on the preview in Google Job Rich result test

Recently I have submitted a sitemap with about 30K URLs in it for jobs. Followed the documentation google is providing and now I can see that the sitemap is successfully submitted and in google search ...
0
votes
0answers
22 views

Google Search Console says it crawled the website but the access logs show no google Bots access at all

Google Search Console tells me that it crawled the website last today (I requested a fresh crawling, as the last was some months ago.) The crawling report tells me now it crawled it, last, today. ...
1
vote
0answers
41 views

Does Googlebot respect the cache-control header?

Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
2
votes
1answer
74 views

Does Googlebot crawl JS or JSON inside an HTML page?

My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data. A) In one ...
0
votes
0answers
10 views

Multiple images on page, but google to only crawl certain ones [duplicate]

Say I have a page with a main image of a product at the top and several related product images at bottom. When I google the main product, the google search results also show up the main product image ...
1
vote
2answers
34 views

Is it better for SEO to use generic meta data that is the same for multiple pages or to omit the meta data?

Our company has developed software to create and publish posts automatically on a WordPress site. Since the posts are published automatically and the images are also saved automatically, is it good ...
1
vote
1answer
46 views

How does authentication-required content get indexed/crawled?

Assume we have a VOD platform to watch online videos and it requires authentication to show that video or any kind of content. User has to be signed up and after that, they can access the videos etc. ...
0
votes
0answers
26 views

Logfiles indicate Googlebot trying to access suspicious URL's

I am seeing suspicious URLs in my log files that look malicious leading to me being concerned about their intent. Reverse IP checks out to be Googlebot so I cannot block these requests if which I am ...
0
votes
1answer
72 views

Google crawls links in Gmail

We faced a strange issue - within a few-hour window, lots of links in the emails were crawled by Google (Google Bot IP addresses were used). When looking for information on the internet, there is ...
1
vote
1answer
28 views

Schema data for Google & Bing bots crawling partial hydrated pages

We are using static html, combined with partial page hydration via javascript. Visitors download a full html page when they first arrive at the website, from there javascript intercepts any internal ...
1
vote
1answer
61 views

Why Googlebot is crawling only one page?

I am trying to understand why Google bot is crawling only one page only on my website. At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
2
votes
2answers
58 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
0
votes
0answers
13 views

Googlebot crawls but doesn't fetch pages [duplicate]

I've been seeing a surge in the number of "Crawled - currently not indexed" pages in search console recently. Screenshot below: When I look in detail to URLs one by one, I see that some of ...
1
vote
0answers
22 views

Mobile usability issues reported by Google at Search Console [duplicate]

Google search console reported that some pages have a Mobile Usability issues: Clickable elements too close together Content wider than screen Text too small to read Some details: this is not about ...
0
votes
0answers
30 views

In my Crawl Stats, 23% of requests are "Unauthorized" - is that bad?

Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good? 68% of my crawl requests return 200 (OK) which is good of course. Is this ...
0
votes
0answers
16 views

Should API urls be disallowed in robots.text on SPAs

I have a single page app that I want google to index. However, I can see from the search console that Google also crawls /api/* urls. Search console does not tell wether google bot calls them directly ...
2
votes
1answer
43 views

Does Google consider a website that publishes data reports (same words, different numbers) as duplicate content?

I am working on a website that generate reports from a template. This template will be using the exact same keywords but different numbers to generate webpages that contains reports. The generated ...
1
vote
1answer
81 views

Can search engine fetch/index website that using ipv6 only?

I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact. Can google and other search engine crawl and index it? If yes, will it ...
1
vote
1answer
135 views

Why Google cant see and index my root page from Sitemap?

I will firstly ask a question and below the question you will see an explanation. Can you tell me where is the actual issue in indexing, is my Sitemap correctly set up, and how do I solve this problem ...
2
votes
1answer
26 views

Could a page with little meaningful content hurt the SEO and rankings of other more important pages of my website?

I have a blog and affiliate links website. I have lots of posts with niche content and some posts with product reviews. And that's my main source of traffic: search engine results that are shown to ...
5
votes
2answers
79 views

Googlebot doesn't see links in Angular SPA because routerLink is rendered with an href starting with a hash

In my Angular SPA (Single Page App), I have links like: <a routerLink="/en/my-page" ... When I check it in Google Search Console, I'm finding that Googlebot sees the link as: <a href=&...
1
vote
0answers
13 views

SEO rawtext backlink [duplicate]

If I add an my http address in a text box of another website and as raw text such that it is not a hyperlink, would this count by googlesbot and for SEO purposes as a backlink? I note a question ...
2
votes
1answer
40 views

What is the longest a page can be in the Google Search cache before it is re-crawled?

The more popular a page is, the more often Googlebot crawls it. For less popular pages, Googlebot can go many months between crawls. If the page were taken down in between crawls, the page could ...
2
votes
2answers
158 views

When I redirect a page with a 301, should I update my Sitemap immediately with the new URL?

Those pages are all indexed and rank well, so I want to be extra careful. This is my current sitemap.xml: /OLD_FOLDER/some-slug-1 /OLD_FOLDER/some-slug-2 /OLD_FOLDER/some-slug-3 I'm setting up 301 ...

1
2 3 4 5
18